Re: Compaq Alpha Bounds Checking

[solar () FALSE COM (Solar Designer)]

> In this post below to the Linux security-audit mailing list, Solar was kind
> enough to fulfill my request for performance data on the Compaq ccc compiler
> for Linux/Alpha using bounds checking.  Astonishingly, Solar's tests showed
> virtually no performance overhead for bounds checking.  I found this to be
> both amazing and depressing for StackGuard, and went away to sulk :-)

Sorry for not answering your last post for that long -- it's still in
my mailbox, and was going to be answered once I have the time.

> Today, I got my own access to an Linux/Alpha box with ccc, and to a Tru64
> box.  Both support the "-check_bounds" switch.  I did my own testing, and
> discovered that as far as I can tell, "-check_bounds" does NOTHING AT ALL.
> Am I missing something?

Yes, I guess so -- see below.

>      foo() {
>              char x[50];
>
>              gets(x);
>      }

I would _not_ expect this case to be covered by the compiler's bounds
checking.  This is in fact the reason I didn't use a strcpy() when
demonstrating the bounds checking to you in my first post about ccc.
Their bounds checking only applies to explicit array subscripts:

       -[no]check_bounds
           Generates  runtime  code  to check the values of array
           subscripts (and equivalent pointer arithmetic  involv-
           ing pointers produced by converting an array name to a
           pointer) to verify that  the  resulting  address  lies
           within  the  range  for  which the C standard requires
           well-defined behavior.

This was so obvious for me that I forgot to mention this on the list,
sorry.  Now I realize that when saying "bounds checking" people often
mean "complete protection", or close to that (with DoS in mind).

Speaking of the usage of gets() and such, even if the compiler was
able to pass bounds checking information down to functions (which ccc
doesn't do), it would at least require that you also recompile those
functions themselves.

> Thus I conclude that Solar's amazing performance results that show no
> overhead are because the compiler is lying about implementing bounds
> checking.  There is no overhead because there is no protection.

Well, they could be more verbose in their description, yes.  As for
the "no protection" -- this wasn't meant as a security feature, but
there's _some_ protection, it's just far from being complete.

Finally, as this also goes to BugTraq this time, here's a piece of my
first post on the subject that shows a case where bounds checking can
work (and does indeed work) --

[[ghost@alice tests]$ cat bounds.c
#include <stdlib.h>

int f(int n, int m)
{
        char buffer[100];
        int i;

        for (i = 0; i < n; i++)
                buffer[i] = 'x';

        return buffer[m];
}

int main(int argc, char **argv)
{
        return f(atoi(argv[1]), atoi(argv[2]));
}
[[ghost@alice tests]$ gcc bounds.c -o bounds -O -s
[[ghost@alice tests]$ ./bounds 1111 33
Segmentation fault
[[ghost@alice tests]$ ./bounds 99 33
[[ghost@alice tests]$ ccc bounds.c -o bounds -O
[[ghost@alice tests]$ ./bounds 1111 33
Segmentation fault
[[ghost@alice tests]$ ccc bounds.c -o bounds -O -check_bounds
[[ghost@alice tests]$ ./bounds 1111 33
Trace/breakpoint trap
[[ghost@alice tests]$ ./bounds 99 3333
Trace/breakpoint trap
[[ghost@alice tests]$ ./bounds
Segmentation fault
[[ghost@alice tests]$ ./bounds 99 33
[[ghost@alice tests]$

The first two compiles are with gcc and ccc w/o bounds checking.  We
get segfaults.  Then the program is recompiled with bounds checking,
and we're now getting those traps (just like the man page says).  The
last two tests are to show that the traps are only generated from
bounds checking and not other errors, and that the program is still
working.  BTW, here's what the checks look like:

        mov     $3, $5
        cmpule  $3, 99, $16
        bne     $16, L$10
        mov     -18, $16
        call_pal 0xAA # gentrap
L$10:
[ ... some code skipped: the loop got unrolled and large ... ]
        addq    $sp, $5, $8
        ldq_u   $16, ($8)

I wouldn't say that the option did "nothing at all" to SSH -- it must
have added quite a few checks, which made the binary 5 KB larger.

Signed,
Solar Designer