Re: Practical Firewall Metrics (rant)

[tqbf () secnet com]

> packet, and things like that.  For static filtering, it's fairly easily 
> done, but in the case of filters that don't always exhibit the same 
> behaviour, testing them sufficiently is increasingly complex and requires 
> some sort of inside/outside testing mechanism that is cooperative.  

Well, inside/outside testing is necessary even for static filters (unless
you rely on actual legitimate servers as an "inside" testing service, like
waiting for SYN+ACK responses as a sign that a packet made it through a
filter). SNI's firewall testing has always relied on a pretty simple
model, which has been 

        - Some sort of packet injection program on the "outside" of the
          firewall, which sends packets that are specially "signed" so 
          that they can be recognized apart from other traffic

        - Some sort of sniffer program running on the "inside" of the 
          network that can pick out signed packets and report what they
          are

It's not too hard to come up with the actual test code to do this; it's
much harder to figure out what things to test. I certainly don't think
having to have cooperative inside/outside testing is a real obstacle to 
testing firewalls (but I don't know if you're saying this either). 

> I agree that there is value in having such tools, I'm just not sure it's 
> likely that they'll test things in such a way that creates useful results 
> without being very complex beasts if we take into account dynamic filters.

Well, I think one possible antidote to complex testing tools is a simple
overall methodology (ie, signing packets and sniffing them out of traffic
streams) and extremely flexible tools (this was the motivation for CASL,
our programming language).

I agree that testing "dynamic filters" (I assume we're both referring to
stateful filtering firewalls here) is more complex than testing static
filters, but really, the implementation of tools isn't the hard part. The
hard part, as I see it, is coming up with potential vulnerabilities to
test for; what characteristics of stateful filtering implementations do we
really want to determine?

Without giving away any secrets, I can already think of a few. What about
Fyodor's nmap tool? Do wacky connection-soliciting SYN packets confuse a
stateful filter? I think (and can't give specific citations) that we've
already discovered that this causes problems for certain vendors. 

If people are really interested in coming up with a standard set of tests
for firewalls, I'd say the next step is really to think about what it is
that needs to be tested, which is my point (thus far) in this discussion.

-----------------------------------------------------------------------------
Thomas H. Ptacek                                        Secure Networks, Inc.
-----------------------------------------------------------------------------
http://www.enteract.com/~tqbf                           "mmm... sacrilicious"