Re: How automate firewall tests

["Marcus J. Ranum" <mjr () ranum com>]

Strabla Ruggero wrote:
> What I need is someone that could tell me which type of tests you do on
> your firewalls and that you like too see automated

You've chosen a fairly interesting problem. What do you intend to
measure about a firewall?   It turns out that pretty much the only
aspect of firewalls that the industry has figured out how to measure
is performance - most notably thoughput and total concurrent
streams. Of course, since a firewall is a _security_ device one
would want to measure something about its security but it turns
out that security is a rather elusive property.

Testing a firewall with crafted packets will measure - something - but
it may measure very wrong. After all, unless your packets are crafted
to be indistinguishable from live application traffic, I'd argue that a
firewall was not very good from a security standpoint if it let any of
the packets through. Indeed, if all you're measuring is performance,
the same applies - firewalls that do layer-7 processing (How can
you call something that doesn't do layer-7 processing a "firewall"?
But that's another question) will have different performance properties
depending on the application mix and the layer-7 data going through,
let alone whether the data is correct or not.

There's a paper or two that might help you. One (search for
"Ranum Kostic Molitor") is quite ancient, but the problem remains
the same. Email me privately if you want a copy; I can see
if I can find it. Another is a paper I did back in the NFR days
on how to cheat on IDS benchmarks. It's highly relevant.
http://www.mail-archive.com/firewalls () lists gnac net/msg22759.html
is a repeat thread of this topic from 2002. See also:
http://www.snort.org/docs/Benchmarking-IDS-NFR.pdf 

Good luck; you've bitten off a huge problem. There have been
any number of attempts at testing firewalls (and IDS) poorly;
I've yet to see a test that's worth a pinch of sand.

mjr. 

_______________________________________________
firewall-wizards mailing list
firewall-wizards () listserv icsalabs com
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards