Major gcc 4.1.1 and up security issue

[Felix von Leitner <felix-fulldisclosure () fefe de>]

So, in my gnupg diff, I used code like this:

  assert(a+100 > a);

with a being an int.  Here, have this example code:

  #include <assert.h>
  #include <stdio.h>

  int foo(int a) {
    assert((int)(a+100) > 0);
    printf("%d %d\n",a+100,a);
    return a;
  }

  int main() {
    foo(100);
    foo(0x7fffffff);
  }

(Also available as http://ptrace.fefe.de/int.c)

Now, if you compile this on a system where int is 32-bit (i.e. almost
anywhere these days), you might expect the assert to trigger in the
second call to foo.  Not so:

  200 100
  -2147483549 2147483647

I opened a gcc bug for this.  They told me that the C standard says
integer overflow for signed integers in undefined and therefore gcc is
right in doing this.

Now you might think that it's just assert, we use if and we are safe.
No, assert() is just a macro that turns into if.  The whole assert code
gets removed here, you won't see a trace of the whole overflow check in
the disassembly.

I found the same issue with gcc regarding pointers.  I found it with gcc
4.1, they told me the same story and fixed it for gcc 4.1.1.  At least
with pointers there is a workaround: you could cast the pointer to a
ptrint_t, add the 100, and then check if it became smaller.  But with
signed it, the portable workaround would be a monstrosity like:

  assert((((unsigned int)a)<<1)+100 > (((unsigned int)a)<<1));

or am I overlooking something?

I'm not saying that the gcc people are wrong with their legalese answer.
I'm saying this will break tons of security checks in existing
applications and will get people to get 0wned.  Please help make the gcc
people fix this!

Felix

PS: I think this is the same base problem and thus starts with gcc 4.1,
but I can only say for sure it happens with gcc 4.1.1.

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/