IDS mailing list archives
RE: Network hardware IPS
From: Dave Killion <Dkillion () netscreen com>
Date: Mon, 6 Oct 2003 13:03:26 -0700
Stefano, I hate Marketing spin as much as the next engineer, but with respect, I disagree here entirely. False Positive reduction has nothing to do with Detection Rate. Reducing False Positives has everything to do with accuracy and context. Hypothetical Example: A hostile attack looks like "LeetAttack 1.0" - this is the actual, valid attack string. But say this string is only hostile if sent as the User Agent in an HTTP connection. Maybe it's a backdoor coded by the webserver author, etc whatever. IDS System A has a signature to detect this attack. They look for "1.0" anywhere in an HTTP stream. Do they detect the attack? Yes. How many false positives - that is, triggers on this signature that are not valid attacks - you think they'll get? I'd say quite a bit. So, Detection = 100%, FP ~ 60-99%. IDS System B also has a signature to detect this attack. They look for "1.0", but they are advanced and have a context matching system that allows them to look only at certain fields within the HTTP stream, one of which is the User Agent. So they put "1.0" in the User Agent context. Do they detect the attack? Yes. How's their false positive rate? Lower than System A, I'd wager, but there's still some there. Do they detect the attack any less than System A? No = both systems would always detect every attack. Detection = 100%, FP ~ 30-50% - No decrease in detection, but half the FP's. IDS System C also has a signature to detect this attack. They have the User Agent context as well, and they put "LeetAttack 1.0" as the detection string. Do they detect the attack? You bet - 100%. Do they have False Positives? No - unless someone was stupid enough to make a valid web browser with that string as the User Agent. And you'd have to wonder at their motivations if they did. Detection = 100%, FP = 0% - No decrease in detection, but infinitely less FP's. Obviously, the real world isn't as cut and dry as this example, but the principles are the same - find something unique to the attack, go for root cause, and get the context as specific as possible. You will maximize detection while minimizing false positives. I hope this information is helpful, Dave Killion Senior Security Engineer Security Group, NetScreen Technologies, Inc. This email contains material that is confidential. The content of this email is for the sole use of the intended recipient(s). Any review or distribution by persons other than the intended recipient(s) without the express permission of NetScreen Technologies, Inc. is strictly prohibited. If you are not the intended recipient, please contact the sender and delete/destroy all copies of this email and any related attachments. NetScreen does not guarantee the accuracy or completeness of third party materials or information. -----Original Message----- From: Stefano Zanero [mailto:zanero () elet polimi it] Sent: Friday, October 03, 2003 3:15 AM To: focus-ids () securityfocus com Subject: Re: Network hardware IPS
They claim a "92% reduction in false positives".
Sometimes this kind of bragging makes me wonder: these people actually think they are speaking to clueless folks ? Or the average audience is actually inclined to hear "92%" and then run to buy a copy of whatever they are selling ? False Positive rate and Detection Rate are inversely proportional, in any detection system. It is true for radars, it is true for medical screening systems, it is true for anything. Check out the ROC, receiver operating curves concepts. I cannot draw in an e-mail, but you can pick up a sheet of paper yourself. Draw a graph: on x-axis, put FP rate. On y-axis, put DR. Now think of a clueless, totally clueless, intrusion detection system. It generates totally random answers. You CAN obtain a 100% detection rate with it - if you accept a 100% false positive rate. The diagram on your chart is a line, bisecting the quadrant. If you want a 50% detection rate, you need to accept a 50% false positive rate, and so on. Better intrusion detection systems would have a different graph, which stands "above" the diagonal line. Draw it - it's just any curve you may think of, which (hopefully !) is monotonically increasing, starting from (0,0) and ending up in (100,100). Do you notice something ? You _CAN_ reduce by any factor (92%, 95%, 99.9999%) the FP rate - but you WILL, always, without doubt, pay a price in detection rate terms. You can do it for the "idiot" IDS described above, you can do it for the best IDS you may think of: but it has always got a price! The curve gives you a suggestion: the best "working point" is the one where the rate of FP increase vs. DR increase is at its top. Of course, determining it in reality is not as simple as on our simple equation ! But this model explains clearly (even clear enough for a salesperson maybe) that "decrease in false positive" or "increase in detection rate" mean nothing at all, by themselves. Stefano Zanero ------------------------------------------------------------------------ --- Captus Networks IPS 4000 Intrusion Prevention and Traffic Shaping Technology to: - Instantly Stop DoS/DDoS Attacks, Worms & Port Scans - Automatically Control P2P, IM and Spam Traffic - Precisely Define and Implement Network Security & Performance Policies FREE Vulnerability Assessment Toolkit - WhitePapers - Live Demo http://www.securityfocus.com/sponsor/CaptusNetworks_focus-ids_000101 ------------------------------------------------------------------------ ---
Attachment:
smime.p7s
Description:
Current thread:
- Re: Network hardware IPS, (continued)
- Re: Network hardware IPS Stefano Zanero (Oct 06)
- Re: Network hardware IPS Darren Bolding (Oct 02)
- Re: Network hardware IPS Alvin Wong (Oct 02)
- Re: Network hardware IPS Ravi Kumar (Oct 02)
- Re: Network hardware IPS Alvin Wong (Oct 02)
- Re: Network hardware IPS Ravi Kumar (Oct 06)
- RE: Network hardware IPS Ron Gula (Oct 02)
- Re: Network hardware IPS Gary Flynn (Oct 06)
- Re: Network hardware IPS david maynor (Oct 07)
- Re: Network hardware IPS Gary Flynn (Oct 08)
- Re: Network hardware IPS Gary Flynn (Oct 06)
- RE: Network hardware IPS Dave Killion (Oct 07)
- Re: Network hardware IPS Stefano Zanero (Oct 07)
- RE: Network hardware IPS david maynor (Oct 08)
- RE: Network hardware IPS Dave Killion (Oct 07)
- Re: Network hardware IPS Stefano Zanero (Oct 07)
- Re: Network hardware IPS George W. Capehart (Oct 08)
- RE: Network hardware IPS Dave Killion (Oct 08)
- RE: Network hardware IPS Frank Knobbe (Oct 09)
- RE: Network hardware IPS Kohlenberg, Toby (Oct 09)
- RE: Network hardware IPS Dave Killion (Oct 09)
- Re: Network hardware IPS Stefano Zanero (Oct 14)