Re: Anyones doomsday machine running low on IDS analyst tears?

[Will Metcalf <william.metcalf () gmail com>]

Steve,

Thanks for the additional info.  It was more of a heads up for folks
switching to a VRT snort/2.9.0 conf to review their existing
rule-sets. If people have custom rules updating to this version of the
conf can have a significantly negative performance impact for some
sigs.  Of course if you don't update to include other portions of the
VRT/snort-2.9.0 config like extended_response_inspection you don't
have access to file_data so VRT sigs that rely on this won't fire.

Regards,

Will


On Wed, Oct 6, 2010 at 10:01 PM, Steven Sturges
<steve.sturges () sourcefire com> wrote:
> The max_pattern_len is a memory option that is used in conjunction
> with ac-split (introduced in 2.8.6).  In terms of the pattern matcher
> itself, it gets near AC-FULL performance at the AC-BNFA memory.
> Pretty significant when there are 10k+ rules in your configuration.
> Performance-wise, reducing the size of the pattern matcher is two-fold.
>
> -- Use of the faster AC algorithm
> -- Fewer CPU cache misses during pattern matching phase
>
> The whole idea of the fast pattern matcher is to use the most unique
> patterns to select a set of "most likely" rules for further evaluation.
>
> One could argue that of the pattern "User-Agent|3a| Mozilla/3.0
> (compatible|3b| Indy Library)", the real unique part starts with
> "Mozilla", so the pattern itself could be shortened.  Every HTTP
> request has a User-Agent header these days, so that really isn't buying
> you all that much in terms of uniqueness of the pattern itself.  And
> when it comes to evaluation of the content, that's an extra 12
> characters that need to be compared via boyer-moore.
>
> In the specific rule example, if "User-Agent:" is a required part
> of the pattern, using an offset of 13, length 20 would get you
> "Mozilla/3.0 (compati" -- eg: content:"User-Agent|3a| Mozilla/3.0
> (compatible|3b| Indy Library)"; fast_pattern:13,20;
> Pretty unique & short.  Therefore less memory in pattern matcher --
> that can now be used for Stream reassembly, gzip decompression, etc.
>
> Will's solution in the follow-on post of fast_pattern:30,20 also works
> and gives you the tail of the above pattern.
>
> Or, split the two contents:
>
> content:"User-Agent|3A| "; http_header; content:"Mozilla/3.0
> (compatible|3b| Indy Library)"; distance:0; fast_pattern;
>
> Or, increase the max pattern length in the default config.
>
> Rule writers know the most unique part of the content patterns
> they are using, especially as they relate to patterns in other
> rules, and the parameters to fast_pattern give them the tools
> that are needed.
>
> Snort is highly configurable and users have so many options to work
> with here to suit their specific ruleset, sensor, available memory,
> network speed, etc.
>
> Or, you can simply revert to using the slower AC-BNFA configuration
> of yore.
>
> Cheers.
> -steve
>
> > From: *Will Metcalf* <william.metcalf () gmail com>
> > Date: Wed, Oct 6, 2010 at 3:10 PM
> > To: Emerging Threats Signatures <emerging-sigs () emergingthreats net>
> > Cc: Snort Users <snort-users () lists sourceforge net>
> >
> >
> > I kid... I kid.. But seriously ran into something sort of interesting
> > playing snort conf options.  It seems that the default pm in snort has
> > changed the the default pattern matcher to be ac-split via...
> >
> > # Configure the detection engine  See the Snort Manual, Configuring
> > Snort - Includes - Config
> > config detection: search-method ac-split search-optimize max-pattern-len 20
> >
> > If you are using your own custom rule sets/ET rules this means that if
> > you previously had an override for fast_pattern's default cut-off of
> > 20 via offset and length options via fast_pattern:<offset>,<length>;
> > or you expected the entire unique pattern specified via fast_pattern,
> > You are now limited to only 20 bytes for fast_pattern even if you set
> > offset/length. So for example let us consider the following ET
> > signature optimized for 2.8.6 using the default pm in 2.8.6 of
> > ac-bnfa.
> >
> > alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET TROJAN
> > Delf Checkin via HTTP (8)"; flow:established,to_server;
> > content:"POST"; http_method; content:".php"; http_uri; nocase;
> > content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)";
> > http_header; fast_pattern; content:"name="; http_client_body; depth:5;
> > classtype:trojan-activity;
> > reference:url,doc.emergingthreats.net/2008268;
> > reference:url,
> > www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Delf;
> > sid:2008268; rev:5;)
> >
> >  Fast pattern matcher: HTTP Header content
> >  Fast pattern set: yes
> >  Fast pattern only: no
> >  Negated: no
> >  Pattern offset,length: none
> >  Pattern truncated: no
> >  Original pattern
> >    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> >  Final pattern
> >    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> >
> > timestamp: 1286391049
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5         18         4         4
> >         55        3.1        3.8          2.9
> >
> > timestamp: 1286391054
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5         18         4         4
> >          52        2.9        3.8          2.7
> >
> > timestamp: 1286391058
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5         18         4         4
> >        54        3.0        3.9          2.8
> >
> >
> > Ok this is what I expect. I have a long match that should be fairly unique.
> >
> > Lets modify the pm to be the one included in the VRT version of the
> > default snort.conf and/or the 2.9.0 snort.conf.  We now get
> > "User-Agent:|20|Mozilla/" as the match added to fast_pattern.  Uh
> > oh... This isn't a very unique pattern is it, this will cause all
> > packets that contain a firefox UA to be at least partially evaluated.
> >
> > 1:2008268
> >  Fast pattern matcher: HTTP Header content
> >  Fast pattern set: yes
> >  Fast pattern only: no
> >  Negated: no
> >  Pattern offset,length: none
> >  Pattern truncated: 50 to 20 bytes
> >  Original pattern
> >    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> >  Final pattern
> >    "User-Agent:|20|Mozilla/"
> >
> > timestamp: 1286390771
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5      17381         4         4
> > 10212        0.6        4.7          0.6
> >
> > timestamp: 1286390775
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5      17381         4         4
> > 9516        0.5        4.7          0.5
> >
> > timestamp: 1286390779
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5      17381         4         4
> > 9347        0.5        4.5          0.5
> >
> > timestamp: 1286390784
> > Rule Profile Statistics (all rules)
> > ==========================================================
> >   Num      SID GID Rev     Checks   Matches    Alerts
> > Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
> >   ===      === === ===     ======   =======    ======
> > =========  =========  ========= ============
> >     1  2008268   1   5      17381         4         4
> > 9266        0.5        4.1          0.5
> >
> > Evaluating the same pcap we now spend 20x more ticks evaluating the
> > same rule.  I doubt that specifying a fast pattern offset and length
> > will allow us to override the max-pattern-len 20 but lets give it a
> > shot by specifying 0,50 as the offset and length.
> >
> > snaplen = 65535
> > 1:2008268
> >  Fast pattern matcher: HTTP Header content
> >  Fast pattern set: yes
> >  Fast pattern only: no
> >  Negated: no
> >  Pattern <offset,length>: 0,50
> >    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> >  Pattern truncated: 50 to 20 bytes
> >  Original pattern
> >    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> >  Final pattern
> >    "User-Agent:|20|Mozilla/"
> >
> > No dice.. So I guess the take away here is that if you are moving to a
> > VRT snort.conf or a 2.9.0 ruleset and you are running custom rules I
> > would pay real close attention to debug-print-fast-pattern output.  We
> > are going through the poor performers now and making modifications
> > where appropriate for ET rules, just thought folks might want to know
> > ;-)...
> >
> > Regards,
> >
> > Will

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users