ICSA: Feet to the fire (wall) again. [long]

[Jon McCown <jmccown () icsa net>]

-----BEGIN PGP SIGNED MESSAGE-----

[It's a rough call... should one contradict the moderator?]

I was about to respond on another issue, and then Phil beat me to it.

A line from the original post "certification means different things to 
different people"
is what struck me as being key to the issue under debate.

I get clanged on a lot about "what ICSA certifies"  and in truth it is a very 
specific zone in the
firewall space [1]:   We set it up in a standard way (a half-dozen required 
services) and we test it 
with the key commercial vendors' tools (currently 3+ commercial tools, some 
of them on 2 OSes) 
and then we follow up with stuff that we've built and obtained "from the 
usual places." 

The key phrase from the (ever bloating) criteria document is "No protocol or 
data content other than 
that specified in the certification security policy shall traverse the 
firewall." [2]  Whether it is telnet,
Ballista, a hacked up version of IPsend, Netsonar, netcat, ISS, or the secret-
twisted-protocols-of-death 
attack that shoves something through-- the product fails.   No, we don't then 
have a beer and then
post it to BUGTRAQ [3]... we ... talk with the vendor about decertification 
[4] (muhahaha). 

Then the vendor's marketing people get really upset and yell at their 
(unfortunate) engineers,
and then the engineers call us and tell us we're idiots.   And then they fix 
things. 
Then we retest. <loop>

What _is_ scary is the wierd stuff we wind up nailing people for.... we've 
gotten to explain
that SOCKS can backflush if you aren't careful; that letting ports above 1024 
through the
firewall is unacceptable (and why, sheesh!); and one vendor (as recently as a 
week ago) learned 
that FTP bounce works a number of interesting ways (not just how some tools 
test for it....).
Er... so what is wrong with a product that blue-screens when you port scan 
it? (anyone?)

Commercial firewall products... wouldn't you think that these issues should 
be in the QA cycle?  [5] 
They are in Chapman/Zwicky, Cheswick/Bellovin, and certainly  
http://www.clark.net/pub/mjr .
If it _were_ as easy as running <insert tool> and then affixing a sticker, 
our overhead would be
much lower, and the list of products certified would be much longer, and I'd 
still have hair. [6]


- - Jon 

[1]  The current criteria (ignore the marketing stuff around it) is at:       
        
http://www.icsa.net/services/consortia/firewalls/certified_products.shtml

[2]  Related to to page 8 of Cheswick and Bellovin....

[3]  Ok... it would be more FUN to have a beer and post it to BUGTRAQ.

[4]   Products which have _never_ been certified remain quietly (but 
grumpily) uncertified.  
       Products which are currently certified must toe the line or get 
publicly "dropped".

[5]  mjr was kind enough to send me some of his network violation tickets 
which are useful in this way http://www.nfr.net/news/giveaways/tickets.html , 
and at the current rate
I'll be out of them all too soon.  Guess I'd better make them last. 

[6]  Some firewalls pass certification testing without problems.  It still 
takes a lot of work in testing though.

[7]  No more footnotes.  Really.  I
-----BEGIN PGP SIGNATURE-----
Version: PGP for Personal Privacy 5.0
Charset: noconv

iQCVAwUBNhlFyqN04bWY62GPAQEfAQP+I03wEZTbDs4eQ+HIXqX/Ua4kS2tx/R/C
ZsJal4n+WsyfXy88i4NFXjTI87IzJHjqAAidts30k8PO4I6QTzPM8SiwAeoXpa1E
rcREGRFC60DH5xQS6Mlpsuuu4CE+pLUphcBUZw3x4bql/cmhBBP2N7WGt8RclCNS
Slbv02KQebw=
=y5GT
-----END PGP SIGNATURE-----