Re: your mail -Reply

[tqbf () pobox com]

> to test firewalls, I am more interested in what the right tools are.  Who
> makes these tools and what do they test?  If they don't exist currently,
> and have to be home-grown, what specific tests should they perform? 

There isn't necessarilly a solution to every problem. Just because one
solution obviously is not valid (such as running CCS or ISS against a
firewall to validate it) does not mean that there is a valid solution
waiting in the wings. Firewall testing is not a simple matter. It is hard,
not in the "costs money and time" sense, but in the "pushing the envelope
of computer security as a discipline in order to solve it" sense. 

You must keep in mind that "firewall" is a fairly abstract term, which
really simply refers to network access control in general. People call
packet filters, proxies, and protocol translation "firewalls", as well as
arbitrary combinations of these techniques. This makes it hard to come up
with one approach that gives a valid metric of the security of all
platforms. 

Individual firewall components, such as packet filters (as a simplistic
example), are easier to test and validate. Tools exist to do this,
although deploying them to do meaningful testing is harder than writing
them in the first place. For example, the ability to generate an arbitrary
IP packet is useful for filter testing, but coming up with a way to
launch the right packets and guage their effects on the filter is
difficult. Nobody has done it yet.

We are gradually obtaining knowledge of problems with past implementations
of firewall technology. From this information, we can craft a test suite
to check for old holes in new technology. This is not as useful as it
seems, though, because the problem with firewall testing is that the goal
of such testing is to reduce the potential for vulnerability to new holes,
not just to close the old ones. If all you want to do is close the
problems enumerated in old CERT advisories, go ahead and run a scanner.

Hopefully, people will come up with new, broadly applicable techniques for
evaluating the security of stateful filter, proxy, translation, and IP
stack implementations. As this work gets done, more work will be done on
testing firewalls. Right now, I'd suggest that we have nothing resembling
the tools we need to ascertain the important characteristics of most
firewalls, and thus would be very suspicious of any organization that
attempted to claim it had certified or reviewed the security of firewall
platforms.

-----------------------------------------------------------------------------
Thomas H. Ptacek                           SNI Labs, Network Associates, Inc.
-----------------------------------------------------------------------------
http://www.pobox.com/~tqbf       "If you're so special, why aren't you dead?"