Re: Fuzzing project brainstorming

[Michal Zalewski <lcamtuf () coredump cx>]

There are many complexities around the general idea - as Kurt
mentions, sometimes, fixing is a lot harder than bumping into a SEGV;
other times, the software may not even have a proper maintainer
anymore. It's doubly tricky if the discoverer doesn't put effort into
evaluating the security impact and prioritizing the bug. Yet another
problem is that it's hard to figure out how fuzzing efforts compare to
each other - if I set up my job poorly, it doesn't matter that I
fuzzed something for 10 cumulative CPU years, and if you mark the
package as clean, you may end up misleading potential users.

But I don't think that these are arguments against trying :-)

I think that the world would benefit in several ways from having a
good, published list of security-critical software that holds up
during intensive fuzz testing; today, this is a bit of "arcane
knowledge" that the select few security experts will have, but others
will have no idea how one parsing library compares to another, etc.
Few days ago, there was a front-page thread on HN steering people
toward a new PNG parsing library, lodepng. Plug it into any fuzzer or
look at the source code and see it yourself...

It would be equally good to maintain a list of high-risk /
high-exposure software that probably hasn't gotten enough fuzzing
cycles - say, Open Office / Libre Office, various document converters,
things like tcpdump, etc. Perhaps a prioritized list of high-risk
software is a good starting point.

Network services have gotten a bit less relevant, but I wouldn't put
them completely out of scope. Fuzzing DHCP, NTP, SMTP, TCP/IP network
stacks, and browser & server HTTP / SSL implementations, SSH, etc,
seems like a good thing, and doesn't require a lot of work.

The somewhat harder things to fuzz are physical link protocols (USB,
eth, etc), firmware, and kernel drivers, so that's probably a topic
for another time.

/mz


On Thu, Nov 20, 2014 at 4:34 AM, Hanno Böck <hanno () hboeck de> wrote:
> Hi,
>
> Following the discussions here I feel this whole fuzzing thing could
> need a project to coordinate efforts and I will probably start
> something within the following days.
>
> I wanted to lay out my rough plans / brainstorming and welcome any
> feedback and especially if people have worries about such a project.
>
> * The core of the project will be a list of free software projects that
>   in one way or another parse fileformats (I'll leave fuzzing network
>   and other input out for now). It should have rough categories (ok =
>   fuzzed and no unfixed issues in latest release, wip: fuzzed and issues
>   are being worked on or already fixed in source repo, stale: fuzzed and
>   issues don't seem to be worked on, unavalable = project with no
>   developers to contact, wontfix = developers don't feel memory access
>   issues and crashes need to be fixed / declare their product
>   unsuitable for untrusted input, unknown = no known fuzzing efforts)
>   I feel that it's important to make the limitation of this info
>   transparent (e.g. about to change rapidly, always further /different
>   fuzzing strategies that might turn up more issues, fuzzing is not a
>   good indicator for overall security etc.).
> * A sharing place for stuff that might be useful for fuzzing, I think
>   especially about patches that disable sanity checks (CRCs etc.) that
>   make fuzzing harder. And maybe file collections with small example
>   files for various file formats.
> * Some introduction tutorials that should give people with no fuzzing
>   experience a starter. Preferrably so easy that everyone with some
>   basic linux/unix knowledge can follow them. Explain zzuf, asan, afl.
> * All kinds of pointers/links to further information.
>
> Data and things like file archive should preferrably be public domain
> / cc0 to make data sharing as easy as possible.
>
> I welcome your feedback. I also welcome all reports (preferrably with
> links to public sources where this is documented - bugtrackers, mailing
> list archives etc.) of fuzzing. The good ("I fuzzed this for so long
> and it seems just nothing turned up") and the bad ("I found 10.000
> unique crashers and reported them all upstream, nobody cares").
>
> cu,
> --
> Hanno Böck
> http://hboeck.de/
>
> mail/jabber: hanno () hboeck de
> GPG: BBB51E42