Re: Pattern Matcher Performance (config detection)

[Martin Holste <mcholste () gmail com>]

Got it, that all agrees with my experiences as well.  So, now I'm
interested in your report that you got a 30% CPU savings with ac-nq.
What is your exact config statement?

On Thu, Feb 24, 2011 at 3:04 PM, Mike Lococo <mikelococo () gmail com> wrote:
> On 02/24/2011 03:30 PM, Martin Holste wrote:
> > > * I run a large ruleset of over 7000 rules from VRT and ET on a link
> > >  that peaks at about 1.8gigabits per second each day.
> > > * Running snort compiled with --enable-perfprofiling shows
> > >  that the pattern-matcher accounts for about 80% of snort's
> > >  CPU time using ac-split.
> > > * Switching from ac-split to ac-bnfa increased by CPU usage by
> > >  about 20%, but decreased ram usage by a few hundred megs per process.
> > > * Switching from ac-split to ac-nq decreased CPU usage by about 30%,
> > >  but increased RAM usage by some unknown amount.
> >
> > So are you inferring that you are running 7000 rules on a 1.8 gig link
> > on a single snort instance and aren't always at 100% CPU?  If that's
> > the case, then either you have very little HTTP traffic in your 1.8
> > gig link, or you're not monitoring what you think you're monitoring
> > (BPF filtering, etc.).  Any Snort instance with more than 1000 rules
> > will be overwhelmed at 200-300 Mb/sec of HTTP traffic no matter which
> > pattern matcher you're using.  You can up your Mb/sec a bit with an
> > Endace card, PF_RING, and a few other tricks, but you can't even run
> > 1.8 gig through the preprocessors without hitting 100% CPU.  I would
> > love to be wrong about this, but it's going to take a lot to convince
> > me that you're achieving anywhere near that throughput on a single
> > instance.
>
> I probably should have mentioned that I'm using 4-instances with Endace
> hardware and ~300MB/process buffers to get this throughput.  The CPU's
> are pretty-well maxed out during the daily-peak, and I drop some packets
> but well under a percent when averaged-out.  I also manually tuned out
> many of the worst-performing rules/rule-classes which may explain why
> I'm a little higher than your 200-300mbit ceiling (which is pretty
> accurate in my experience but I've pushed this box a bit further with
> manual tuning).
>
> My traffic and protocol distribution is common for higher-ed if you're
> familiar with that space.  It looks more or less like ISP traffic with a
> extra few high-speed transfers.  HTTP is either first or second protocol
> in terms of bits/sec, so these rates don't come from a weird protocol
> distribution that results in lots and lots of packets essentially not
> getting inspected.
>
> Cheers,
> Mike Lococo

------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in 
Real-Time with Splunk. Collect, index and harness all the fast moving IT data 
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business 
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users