Re: Chasing bugs / vulnerabilties

[Theo de Raadt <deraadt () CVS OPENBSD ORG>]

> Try the UNIX Fuzz experiment, first conducted at the University of Wisconsin
> on multiple UNIX operating systems and when tried again several years later
> revealed only slightly better results (the Fuzz experiment throws garbage
> input on the command line into a program and tests the response).   We (check out
> http://www.cerias.purdue.edu/coast/ms_penetration_testing/v11.html) tried
> the same experiment on WinNT with 'interesting' results.

After Michael mentioned this, I decided to download this and see if it
found any bugs in OpenBSD.  I was pretty sure I'd find at least a few.
I hacked it up a bit, and then ran it against every single binary in a
"default install" openbsd system.

I told Michael what I had found, and he suggested that I tell bugtraq.

I ended up finding the following:

        nc (netcat)
                - lack of buffer termination in code where it fakes
                  argument parsing from stdin, causing strchr to run
                  off the end
                - fixed
        mg
                - did not like stdin being redirected, and crashed
                  in terminal initialization
                - fixed
        ctags
                - about 18 buffer overflows in token parsing
                - fixed
        lex
                - looks like 2 buffer overflows based on strcpy in parsing
                  routines.
                - looks difficult to fix: not yet fixed.
        as
                - indexing into an array using signed char's: fixed
                - other bugs exist which trigger some assert()s,
                  but since we are moving to a new version soon, we'll
                  skip fixing those for now.
        makeinfo
                - part of texinfo.  buffer overflow.  still being
                  looked at.
        sort
                - infinite loop when fed a single (and long) non-NL
                  terminated line
                - not fixed yet
        indent
                - infinite loop if it encountered EOF while waiting for
                  a '}' nesting terminator
                - fixed
        rdist
                - leaves /tmp files created using mkstemp() around
                - still trying to fix this
                - note that our rdist has not been setuid or setgid for
                  roughly half of eternity
        cvsbug
                - leaves /tmp files lying around when terminating due to
                  no action possible
                - fixed
        sendbug
                - leaves /tmp files lying around when terminating due to
                  no action possible
                - fixed

So far, nothing was security sensitive by itself.  While it fixed
things, and that always helps us understand the problems in all
classes of code better, I don't think this is directly related to
'security' work.

However, as I said, it helps us understand the mistakes people make
when they write programs.  It helps us understand that many of those
mistakes are pretty basic C errors.  It helps prove the case for
getting a second or third eye to look at the source code.  Too often,
I look at code from 1985 and discover bugs which lead me to believe I
am the second, third, or perhaps fourth eye.

I would love to find a simple tool which could test a variety of
command line arguments for functions.  Or perhaps something written to
test every single command line option on every base unix command.  I
suspect some vendor probably has something like this, but probably
locked up in their back room; lest operating system quality as a whole
improve and force them to improve theirs too.

So, when we finish fixing the last few of these problems, it'll
probably be about 20 hours of effort by 3 people.  The way I look at
it, the interesting thing is that we actually do this stuff.  Thus,
the important question is probably: "Is anyone else doing this?"

I still consider fuzz to be somewhat of a crutch.  For about half of
these fixes, inspection found other things we could improve.  Now that
we've run this, we will probably discard the tool (mostly since it
causes havoc, running wall(1) and logger(1) against these hugs nasty
input files causes some grief).  We've gotten 99% of the benefit it
can find, and perhaps we can re-run it in 2-3 years.  It's just not a
typical useage tool.  And most other checkers like this are not
either.

However, I think that once a person learns to audit code by just
reading it, that skill is transferable and reusable on a daily basis.

That was my main argument against *printf checkers; whether they be
external or built into a C compiler.  They still miss things.  You
still have to go read the code.

Thanks for the entertainment over the last week from various people
suggesting we can change the C language, redo how varargs/stdarg work,
or showing a complete misunderstanding of the *fundamental weakness*
of static analysis.  A suggestion for those people: Learn how things
work and why they work that way before suggesting how they should be
changed.  Read the stadards, read some implimentations.  Apparently
some people feel that writing diatribes is a replacement for reading.