Compaq Alpha Bounds Checking

[crispin () CSE OGI EDU (Crispin Cowan)]

In this post below to the Linux security-audit mailing list, Solar was kind
enough to fulfill my request for performance data on the Compaq ccc compiler
for Linux/Alpha using bounds checking.  Astonishingly, Solar's tests showed
virtually no performance overhead for bounds checking.  I found this to be
both amazing and depressing for StackGuard, and went away to sulk :-)

Today, I got my own access to an Linux/Alpha box with ccc, and to a Tru64
box.  Both support the "-check_bounds" switch.  I did my own testing, and
discovered that as far as I can tell, "-check_bounds" does NOTHING AT ALL.
Am I missing something?

Consider this program, which I use for trivial StackGuard testing:

     foo() {
             char x[50];

             gets(x);
     }

     main() {
             foo();
             printf("Hey, kenny lives!\n");
     }

Compiled thusly:  cc -check_bounds test.c -o test

Now what are the results of testing this program?

   * When compiled with ccc -check_bounds on Linux/Alpha and given a large
     input, this program prints out "Hey ..." and then seg faults.
   * When compiled with cc -check_bounds on Tru64/Alpha and given a large
     input, the program also prints out "Hey ..." and seg faults.
   * When compiled with StackGuard on Linux/x86 and given a large input, the
     program prints a StackGuard intruder alert and exits.

Ok, so maybe -check_bounds doesn't work for autos, and only works for static
arrays (admittedly, StackGuard has no impact on static array overflows
either). Consider this program.

#include <stdio.h>

char a[25];
char b[12];

main() {
        gets(a);
        strcpy(b,a);
        printf("a=>%s<, b=>%s<\n", a, b);
}

This time, we get:

   * Linux/Alpha:  prints both strings correctly and exits normally.  It
     should not have.
   * Tru64:  prints nothing, just seg fault.  Expected behavior for a
     non-bounds checking compiler.

So, am I missing something here?  At no time could I get either of these
programs to die on a SIGTRAP, which is what the Compaq man pages say should
happen when you over flow an array bounds with -check_bounds turned on.

Thus I conclude that Solar's amazing performance results that show no
overhead are because the compiler is lying about implementing bounds
checking.  There is no overhead because there is no protection.

Caveat:  this is from 1 hour of playing on the Alphas.  If someone knows more
about these machines and can tell me what I did wrong, feel free to speak up.

Crispin
-----
Crispin Cowan, CTO, WireX Communications, Inc.    http://wirex.com
Free Hardened Linux Distribution:                 http://immunix.org

Solar Designer wrote:

> > > Sorry for not answering the questions, but I thought I'd let people
> > > know about Compaq C for Linux/Alpha having a bounds checking feature,
> > > which some of us might find useful --
> >
> > Excellent!  I've been wanting to compare StackGuard against a bounds
> > checking compiler for years.
>
> Well, they are quite different:
>
> 1. Some existing programs might break when compiled with bounds
> checking.  In particular, the following code will likely result in a
> trap with ccc:
>
>         int x[10];
>         int *p;
>
>         p = x - 5;      // trap here
>         while (...) {
>                 p += 5;
>                 [...]
>         }
>
> 2. Bounds checking as implemented in ccc is somewhat limited; I got a
> few warnings (not many) when compiling ssh:
>
> cc: Warning: readpass.c, line 299: In this statement, pointer
> arithmetic was performed more than once in computing an array
> element.  The bounds checking code output by the compiler will only
> verify the "buf+strlen(...)" expression. (badboundchk)
>       p = buf + strlen(buf) - 1;
> ----------^
>
> 3. Bounds checking, unlike StackGuard, can be useful when auditing.
>
> > If someone has access to an Alpha and the Compaq Linux/Alpha compiler,
> > could you do before/after testing on the bounds checking feature and try
> > to determine the %overhead?  The StackGuard overhead data is here:
> > http://immunix.org/StackGuard/performance.html
>
> OK, I did some testing with SSH, replacing sshd only, not the client,
> as there's some problem in ssh client when compiled with ccc that I
> didn't bother to investigate (it's not related to the bounds checking).
>
> Here're the performance numbers (localhost to localhost scp transfer,
> IDEA encryption, no compression):
>
> RH 5.2, default ssh compile             1300 KB/s
> egcs 1.1.2, -O2, EV56                   1450 KB/s
> ccc, -O4 -arch host                     1680 KB/s (6.35 ... 6.38 secs)
> ccc, -O4 -arch host -check_bounds       1680 KB/s (6.35 ... 6.38 secs)
>
> The results with Blowfish are very similar (just about 30% faster in
> all tests).
>
> So, no noticable performance impact.  The binary got about 5 KB larger
> (both were stripped), when compiled with bounds checking.  My guess
> is that ccc does most of its checking at compile time.
>
> In all these tests, the ssh client remained the same (compiled right
> after installing this RH, with egcs 1.0.3 and non-optimal options).
>
> Of course, I've also tried compiling John with ccc.  Bounds checking
> didn't affect the performance of some of the hashes at all, but had a
> significant impact on some others.  I haven't checked the code, yet,
> but this should depend on whether the compiler was able to do the
> checking at compile time or not, whether there was enough parallelism
> in the code already (so extra instructions would need extra cycles,
> which is often not the case), whether there are some spare registers
> for the bounds checking.
>
> Signed,
> Solar Designer