RE: IDS evaluations procedures

[THolman () toplayer com]

Hi Dave,

That's always an interesting question - how do you evaluate detection
systems when all they respond to is known exploits?
Yes, it's easy to launch all known exploits against a detection system using
one of the many vulnerability scanners out there, but how do you test
whether or not the detection system will respond (and hopefully block)
zero-day attacks?  Especially where the attack (and associated
vulnerability) is not known yet?  Just look at the devastating impact of
recent worms that effectively bypassed all detection systems as they were
not programmed to look for the exploit that they didn't know about...
Whilst you're moving in the right direction by detecting anomalies, you have
to be very careful as to how accurate this is.  What technologies are being
used to detect the anomalies?  How many false positives do such detection
methods invoke?  What IS an anomaly?  What it such an anomaly is in fact a
surge in an increase in valid traffic due to heightened end-of-day activity
or spikes in web usage?
To specifically answer your question, look at current attack weather reports
- you'll see approximately 15-20% of perimeter traffic is in fact worms
trying to propagate.  Any evaluation should be designed with this in mind.
..but more importantly, make sure you're evaluating something that will do
the job in hand and doesn't lead you up the garden path with inaccurate
marketing collateral!  :)

Regards,

Tim


-----Original Message-----
From: david.sames () sparta com [mailto:david.sames () sparta com] 
Sent: 12 July 2005 03:40
To: focus-ids () securityfocus com
Subject: IDS evaluations procedures

I'm in the process of developing test procedures for evaluating an internal
anomaly-based detection system. I'd like to construct a test set of nominal
data peppered with attack data. What is a reasonable ratio of attack data to
"normal" traffic that is representative of "real" systems.

Thanks,

Dave

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------