Re: FAQ submission: optimizing performance of rules with PCRE

[Joel Esler <joel.esler () sourcefire com>]

All,

I suggest a review of the documentation on the Rule Optimizing engine, 
as well as the High-performance multi-rule pattern matching engine at 
http://www.snort.org/docs/#devel  Read the whitepapers there.

To quote a wise sage:

"Snort uses a two-stage process in the Snort detection engine these 
days.  In it's standard configuration all the rules that are loaded in 
at runtime have their longest pattern matching option 
(content/uricontent) loaded into a fast set-wise pattern matching 
engine.  (Set-wise pattern matchers match all patterns in the set 
simultaneously.)  Once the engine is up and running, traffic is run thru 
the set-wise pattern matcher to pre-qualify rules that *may* fire.  
These rules are chained together and tested after the prequalification 
stage, greatly reducing the number of rules that have to be analyzed for 
any given data set.  For the sake of building the prequalification 
set-wise matching data, the PCRE rule options are ignored and only 
tested when the full rules themselves are tested after prequalification.

There are three basic pattern matching algorithms that we use in Snort 
today, Wu-Manber, Aho-Corasick and Boyer-Moore.  PCRE uses its own 
DFA/NFA mechanisms behind the scenes."

Or in English:  Put content before pcre, because content can be matched 
faster than the pcre engine can (and content is done first).  IOW, 
always put a content before a pcre.

Joel

David J. Bianco wrote:
> The easiest way to understand this is to know that snort processes
> the rule keywords in the order they are given and stops processing
> as soon as one of the matches fails.  Also know that the PCRE engine
> is usually slower than the "content" match engine.
>
> By putting a simple "content" match before the PCRE, you save CPU
> cycles.  Without it, you'd be using the slower engine on every
> packet the rule processes.  With "content", you use the fast engine
> and can throw away packets that don't have a hope of matching the
> regular expression anyway.  Then you only pay the big performance
> penalty on (hopefully) a small percentage of the total traffic.
>
> You can see an example of this in my "EZ Snort Rules" presentation
> (http://www.vorant.com/files/EZ_Snort_Rules.pdf) if it helps
> make things more clear.
>
>         David
>
> James Affeld wrote:
>   
> > "How can I get the best performance with rules that
> > include PCRE content checks?"
> >
> > I understand from Nigel on the VRT and Matt Jonkman
> > that putting in a content: check ahead of the PCRE:
> > helps performance even if the content check doesn't
> > otherwise do anything the pcre doesn't handle. 
> > Something about the content check forcing the program
> > exectution to the optimal pattern patch code, which
> > won't happen without "bare" pcre.  
> >
> > In other words, a rule like 
> >
> > alert tcp $EXTERNAL_NET any -> any any (msg:"Faster
> > PCRE rule"; content: "content checked"; nocase; pcre:
> > "/content checked/i";)
> >
> > will perform better than
> >
> > alert tcp $EXTERNAL_NET any -> any any (msg:"Slower
> > PCRE rule"; pcre: "/content NOT checked/i";)
> >
> > because the redundant content: check causes the more
> > efficient pattern matching code to be invoked.  
> >
> > Disclaimer: I'm pretty confident I understand what to
> > do, but the explanation for why it works this way is
> > all 2nd hand and filtered through my imperfect brain. 
> >
> >
> > Disclaimer 2: I should not be credited - merely
> > echoing the lore of others.
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam protection around 
> > http://mail.yahoo.com 
> >
> >
> > -------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > Snort-users mailing list
> > Snort-users () lists sourceforge net
> > Go to this URL to change user options or unsubscribe:
> > https://lists.sourceforge.net/lists/listinfo/snort-users
> > Snort-users list archive:
> > http://www.geocrawler.com/redir-sf.php3?list=snort-users
> >     
>
>
>
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users
>
>   


-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users