Re: Unusual Snort performance stats

[Matt Watchinski <mwatchinski () sourcefire com>]

I'll clarify :)

 On Mon, Feb 22, 2010 at 3:59 PM, Jason Haar <Jason.Haar () trimble co nz>wrote:

> On 02/23/2010 05:29 AM, Matt Watchinski wrote:
> > 1. Outstanding means that packets never got out of the ethernet card
> > before they got dropped.  IE pcap didn't get to them before they
> > disappeared.
>
> Well that does my mind in. Can you explain to the uninitiated how snort
> can know a packet was received by an Ethernet card, but then dropped
> before it got out of the card?
>
> Does that mean there are two ways to drop packets? Am I correct in
> saying that "dropped packets" implies the OS (ie pcap) received the
> packet but dropped it due to snort/userspace being too busy to extract
> all the buffer within some time period, but "outstanding" is just as
> bad? I've only ever noticed the "Dropped" field before :-(
LogMessage("Outstanding: " FMTu64("12") " (%.3f%%)\n",
                        pkts_recv - pkts_drop - pc.total_from_pcap

void PcapProcessPacket(char *user, struct pcap_pkthdr * pkthdr, const u_char
* pkt)
{
....
    pc.total_from_pcap++;

-

libpcap docs
2.4.x
=====
ps_rec    Number of packets that were accepted by the pcap filter
ps_drops  Number of packets that had passed filtering but were not
          passed on to pcap due to things like buffer shortage, etc.
             This is useful because these are packets you are interested in
             but won't be reported by, for example, tcpdump output.

so

pkts_recv = ps_rec
pkts_drop = ps_drops

--------

So that is what makes up the "OutStanding" packet counter.  So it contains
things that pcap knew about but couldn't pass on, this is normally the
buffer on the ethernet card couldn't hold the packet long enough for it to
get read off the card.

As for dropped packets your statement below is true, pcap handed it off to
snort and snort for some reason couldn't accept it.


> This stats means that some percentage of your traffic contains
> > protocols that snort doesn't do anything with.  Tracking these down
> > and add BPF's to ignore them could improve performance.
>
> That's good advise we could all use I'm sure!
>
> > 3. Are you using CPU affinity to lock the snort process to a specific
> > CPU?  If not this is something to try.  If snort bounces to another
> > CPU then the cache line is reset and performance can suffer.
>
> Are you saying that there's real value in ensuring snort remains on the
> same CPU - even over restarts? Why would the cache matter? I mean,
> restarting snort means your IDS is deactivated until it's fully
> operational again - does keeping it on the same CPU simply minimize that
> outage, or do you mean something else.
Assume the following, snort box with 2 CPU's, not cores two physical CPU's.
Running snort, mysql and base.

If snort and mysql end up on the same cpu, they are sharing time, that might
not be a big deal, but lets say you run a big query in base.  This could
force the nice linux scheduler to relocate snort onto the other CPU, so
mysql can do its job better.  This moves snort over the other cpu, all the
instructions waiting for snort / cache all get moved along with it.  This
isn't good for performance.

Now I haven't looked at the latest intel chip sets for how this works with
CPU's with multiple cores, but the last time I looked they all shared the
same cache line.  So all four cores had to pull from one cache if all those
cores where on the same physical CPU, this is essentially a bottleneck.  So
even if you have all these cores you'll still want to lock snort to one
physical cpu, and all your other stuff to another one.

Hope that clarifies it a bit,
-matt



> Lots of yummy stuff in your message to chew on :-)
>
> Jason
>
>
> --
> Cheers
>
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +64 3 9635 377 Fax: +64 3 9635 417
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users<https://lists.sourceforge.net/lists/listinfo/snort-users%0ASnort-users>list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users



-- 
Matthew Watchinski
Sr. Director Vulnerability Research Team (VRT)
Sourcefire, Inc.
Office: 410-423-1928
http://vrt-sourcefire.blogspot.com && http://www.snort.org/vrt/

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users