RE: Belaboring the point of FPs

["Bob Walder" <bwalder () spamcop net>]

It is very difficult (err.... read "impossible") as a test lab to test
the effectiveness of every signature and protocol decode in an IDS/IPS
product - that is why most labs don't bother. 

Whilst we spend a lot of time testing performance, resistance to false
positives, susceptibility to false negatives, resistance to common
evasion techniques, ability to handle millions of connections,
stability, ease of use, ease of management, ease of deployment, etc,
etc, etc, we do also have a section in our methodology which addresses
signature coverage.

The best we can do here is select an arbitrary bunch of exploits (over
100 in our current test at www.nss.co.uk/ids) and evaluate how good each
IDS is at a) detecting the exploit "out of the box", b) detecting the
exploit with a custom signature update (i.e. how quickly can the vendor
respond to a "new" exploit they do not currently cover?), and c) how
accurate is the alert? 

We also look at how easy it is to a) create your own signatures and b)
modify the vendor's built-in signatures (if that is possible at all!)

Not perfect, but at least we are making a stab at addressing this area.

Regards,

Bob Walder
Director
The NSS Group

> > Anyways, this is the one thing that cracks me up about 
> > industry analysis and testing. There's tons of focus on 
> > performance, stability, look and feel, protocol decoding, 
> > etc. in IDS tests being published. But what about the reason 
> > you buy an IDS in the first place??? Who's *really* testing 
> > how well an IDS can detect the attacks it says it will. Not 
> > the decoders and the normalizers, I mean the signatures. 
> > Surprisingly VERY few, and everyone else just blindly 
> > accepts that. My hat's off to Blade Software for raising 
> > this issue in the first place. 



---------------------------------------------------------------------------
Attend Black Hat Briefings & Training Federal, September 29-30 (Training), October 1-2 (Briefings) in Tysons Corner, 
VA; the world’s premier 
technical IT security event.  Modeled after the famous Black Hat event in 
Las Vegas! 6 tracks, 12 training sessions, top speakers and sponsors.  
Symanetc is the Diamond sponsor.  Early-bird registration ends September 6 Visit: www.blackhat.com
---------------------------------------------------------------------------