Re: How automate firewall tests

[Shahin Ansari <zohal52 () yahoo com>]

If you would, please send me a copy of the paper you mentioned. I do have a comment, please look below:

"Marcus J. Ranum" <mjr () ranum com> wrote:    Strabla Ruggero wrote:
> What I need is someone that could tell me which type of tests you do on
> your firewalls and that you like too see automated

You've chosen a fairly interesting problem. What do you intend to
measure about a firewall? It turns out that pretty much the only
aspect of firewalls that the industry has figured out how to measure
is performance - most notably thoughput and total concurrent
streams. Of course, since a firewall is a _security_ device one
would want to measure something about its security but it turns
out that security is a rather elusive property.

Testing a firewall with crafted packets will measure - something - but
it may measure very wrong. After all, unless your packets are crafted
to be indistinguishable from live application traffic, I'd argue that a
firewall was not very good from a security standpoint if it let any of
the packets through. Indeed, if all you're measuring is performance,
the same applies - firewalls that do layer-7 processing (How can
you call something that doesn't do layer-7 processing a "firewall"?
But that's another question) will have different performance properties
depending on the application mix and the layer-7 data going through,
let alone whether the data is correct or not.

There's a paper or two that might help you. One (search for
"Ranum Kostic Molitor") is quite ancient, but the problem remains
the same. Email me privately if you want a copy; I can see
if I can find it. Another is a paper I did back in the NFR days
on how to cheat on IDS benchmarks. It's highly relevant.
http://www.mail-archive.com/firewalls () lists gnac net/msg22759.html
is a repeat thread of this topic from 2002. See also:
http://www.snort.org/docs/Benchmarking-IDS-NFR.pdf 
   
  I am curious how above material is affected now that Vendors like Cisco implemented packet statful inspection.  All 
the items regarding UDP, ICMP, and few others change.  The doco above says no good firewall should allowe ICMP, but now 
Cisco claims they keep track of what ICMP requests went out and will only allow 1 reply.  So this would be a valid test 
now ha? 
   
  I would also add some tests regarding how well and fast the firewall handles VoIP traffic.  What VoIP protocols they 
support.  What is the throughput for such packets.

Good luck; you've bitten off a huge problem. There have been
any number of attempts at testing firewalls (and IDS) poorly;
I've yet to see a test that's worth a pinch of sand.

mjr. 

_______________________________________________
firewall-wizards mailing list
firewall-wizards () listserv icsalabs com
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards





                
---------------------------------
Stay in the know. Pulse on the new Yahoo.com.  Check it out.

_______________________________________________
firewall-wizards mailing list
firewall-wizards () listserv icsalabs com
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards