Re: False positives, negatives and don't cares

[Paul Schmehl <pauls () utdallas edu>]

--On Monday, August 11, 2003 00:12:46 -0400 Martin Roesch 
<roesch () sourcefire com> wrote:
> This is one of the things that frustrates me when (some) people say that
> Snort generates "tons of false positives".  If you know how Snort's
> detection engine works, you know that Snort merely presents a framework
> run through a set of analysis nodes arranged in certain ways against a
> network traffic stream (or single packet).  Snort doesn't have a
> hard-wired detection engine, it is defined and built at run-time based on
> the user-defined rules that are loaded into it.  In this model it is very
> difficult to have false positives except in the case there are insertion
> attacks, Snort only detects what you tell it to.

Herein lies IDSes' true weakness.  They're only as good as the rules that 
are written for them.  And BTW, it's my considered opinion that rules are 
going to get harder and harder to maintain as the "body" of rules grows 
ever larger.  (For an analogous explanation, see my two articles - "Past 
Its Prime: Is Anti-Virus Scanning Obsolete?[0] and "Life After AV: If 
Anti-Virus is Obsolete, What Comes Next?[1])

I'll give you a real world example from snort:

sid 1002.
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-IIS 
cmd.exe access"; flow:to_server,established; content:"cmd.exe"; nocase; 
classtype:web-application-attack; sid:1002;  rev:5;)

(I see that in rev 5 someone has finally limited the target to 
$HTTP_SERVERS rather than any, which should help greatly in reducing the 
FPs, but will result in FNs when someone installs an IIS server that you 
don't know about.  Previously the rule read $EXTERNAL_NET any -> any 
$HTTP_PORTS.)

Previously this rule looked for "cmd.exe" in the content of any packet, 
regardless of the purpose of the computer to which the packet was 
traveling.  The problem is that "cmd.exe" shows up in a *lot* of packets 
that are coming from the windowsupdate site, because MS is polling for that 
information.  So, you get flooded with FPs and you begin to ignore the rule.

When I pointed this out on the rules list and suggested that the content 
section should look for "cmd.exe?" instead, I was told that "this might 
miss some legitimate infections".  I'm not sure how, because I have yet to 
see an exploit that doesn't supply the "?" immediately after the "cmd.exe". 
It's pretty much necessary.  (I suppose you could put padding between the 
"cmd.exe" and the "?", just to bypass this rule, but no one has yet.)

Rather than swim upstream, I created a local rule that looks for *outgoing* 
traffic only (because I only care about infections coming *from* my 
network) and the content is "cmd.exe?", not "cmd.exe".  I still get some 
FPs, but not nearly as many.  I've even asked if anyone on the list can 
explain the FPs, because they appear to be just random crap stuck in 
memory, but I've never gotten an answer.

I'm not complaining, mind you, I think snort is great.  It's the IDS we 
use.  The idea that there are no FPs is simply not accurate, however. 
There are.

Now, I think your idea of vulnerability correlation holds great promise to 
make IDS much more useful than it already is.  Pouring through 1000's of 
alerts trying to figure out if the boxes they're hitting are vulnerable to 
the attack detected is not my idea of fun, nor is a productive use of my 
always limited time.

Paul Schmehl (pauls () utdallas edu)
Adjunct Information Security Officer
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu

[0]<http://www.securityfocus.com/infocus/1562>
[1]<http://www.securityfocus.com/infocus/1604>

---------------------------------------------------------------------------
Captus Networks - Integrated Intrusion Prevention and Traffic Shaping  
- Instantly Stop DoS/DDoS Attacks, Worms & Port Scans
- Automatically Control P2P, IM and Spam Traffic
- Ensure Reliable Performance of Mission Critical Applications
Precisely Define and Implement Network Security and Performance Policies
**FREE Vulnerability Assessment Toolkit - WhitePapers - Live Demo
Visit us at: http://www.captusnetworks.com/ads/31.htm
---------------------------------------------------------------------------