IDS mailing list archives
Re: Crossover Error Rate (WAS "Intrusion Prevention")
From: Bennett Todd <bet () rahul net>
Date: Thu, 12 Dec 2002 16:52:46 -0500
Thanks for bring this up. I hadn't heard of Crossover Error Rate; it's a nice abstraction. I'll certainly keep it in mind. IDSes face a different problem. With sufficient [manual] tuning, against any given sample of traffic an IDS can be adjusted to have perfect behavior --- no false positives, nothing missed. The problem is that the tuning needs to be done again for the next sample of traffic. Fortunately, successive samples of traffic taken at the same point are similar to each other; while there is change over time, it's not fast enough that manual tuning can't keep up with it. You can automate incorporating frequent updates from an external source (e.g. I pull mine automatically from www.snort.org), and occasionally an update introduces a new false positive you have to manually tune out. Unfortunately, traffic taken at different locations doesn't share this property. Each site will in general have to do some more tuning. Worse still, two different organizations will disagree about the classification of a given packet or correlation. For instance, there are some admins who think port scans are always attacks, some who never regard them as attacks, and some who apply different standards in different settings. One man's false positive is another man's incident. And for purposes of trying to have reproduceable documented measures of performance, there's the additional problem that any such measure is only meaningful and reproduceable against a single packet flow. While a packet flow can be captured (e.g. tcpdump) and replayed (e.g. tcpreplay), it will rapidly become less and less meaningful as a measure of real world performance, since attacks (and the signatures required to catch them) change and evolve over time. I don't think we're in a position to make use of the Crossover Error Rate, pretty concept though it is, because the uncontrolled variables in the IDS world have so far defied efforts to produce satisfying, reproduceable concrete measures of performance. The only performance metric I've seen that I do like is "capture a nice long slice of real traffic from the place I want to deploy the IDS. Set up a captive LAN, use tcpreplay to send packets ripping across the bow of the IDS. How fast can you go before you start seeing significant packet losses?" For snort on current generation PCI-bus machines, with quick CPUs and plenty of RAM and good drivers for good interfaces under good OSes, the answer seems to be in the neighborhood of 50Mbps untuned, up to as high as 200-300Mbps with sufficiently careful tuning. Snort 2.0 is supposed to make things better, OS developers continue to tune their stuff, and of course hardware vendors are hard at work as well; I wouldn't be surprised if the equivalent statement a year from now weren't 1Gbps untuned. But an untuned IDS is typically only useful for reporting and stats and incident investigation, if you want to try to act on alarms in [near]-realtime you need to do some tuning. The tuning can be minimized by careful positioning of the IDS, it's awfully easy if you place it inside a good tight firewall, but it's still necessary, that one man's bread problem. -Bennett
Attachment:
_bin
Description:
Current thread:
- Crossover Error Rate (WAS "Intrusion Prevention") Rob Shein (Dec 11)
- Re: Crossover Error Rate (WAS "Intrusion Prevention") Raistlin (Dec 11)
- RE: Crossover Error Rate (WAS "Intrusion Prevention") Rob Shein (Dec 12)
- Re: Crossover Error Rate (WAS "Intrusion Prevention") Bennett Todd (Dec 12)
- Re: Crossover Error Rate (WAS "Intrusion Prevention") Raistlin (Dec 11)