Re: BPF/libpcap performance, was Re: Seg Fault

[Phil Wood <cpw () lanl gov>]

Ok, I'm running linux with a MMAP libpcap based on the Alexey Kuznetsov
MMAP kernel option.
I've been running and tuning and adding features for all too long.
I'm hoping to get it into the tcpdump.org arena.  However, the maintainer
is very busy with things (probably his day job).  I have high hopes.  One thing
you can get is low level interface statistics related to the particular pcap
session on a specifiable millisecond interval.  It will show you exactly,
in what time frame your application started losing packets.  For example
this Sunday I lost some packets.  About 2%.  That's not good.  Nobody wants
to lose packets.  And, maybe if I stopped keeping stats, and trying to debug
core dumps and running in gdb-ized snort, I might go a little faster.  But,
I can look at the stats (I generate an ascii record to stderr with a 10000
ms interval) to see when it gets bad and then ponder.

I'll append an abbreviated log output.  The "S:" record generated contains
the following fields after the "S:".

 0 pcap time
 1 pkts processed
 2 pkts dropped
 3 pkts that passed the bpf filter (some may still not have been processed)
 4 pkts that were ignored by the linux pcap ring subroutine
 5 pkts seen by the device (total in and out)
 6 bytes seen by the device (total in and out)
 7 bytes received by snort
 8 number of times the ring subroutine had to wait for more pkts
 9 the index of the ring buffer (where the next packet is going to show up)
10 the maximum number of pkts sucked off the ring before make a poll call
11 specious returns from the poll call (don't seem to see those anymore)

==============================================================================
< here we go...
< I first export my pcap environment:
<
< export PCAP_STATS=0xfff PCAP_FRAMES=max PCAP_TO_MS=10000 PCAP_VERBOSE=1
<
< There is no change to snort source to use this mechanism.

Log directory = /tmp/log/green

Initializing Network Interface eth2
WARNING: OpenPcap() device eth2 network lookup: 
        eth2: no IPv4 address assigned
Kernel filter, protocol 0003, MMAP mode (32768 frames, snapshot 1514), socket type: Raw

        --== Initializing Snort ==--
Rule application order changed to Pass->Alert->Log
Checking PID path...
PATH_VARRUN is set to /var/run/ on this operating system
PID stat checked out ok, PID set to /var/run/
Writing PID file to "/var/run/snort_eth2-bg.pid"
Decoding Ethernet on interface eth2
Parsing Rules file /tmp/scripts/bg.conf
No arguments to frag2 directive, setting defaults to:
    Fragment timeout: 60 seconds
    Fragment memory cap: 4194304 bytes
Stream4 config:
    Stateful inspection: ACTIVE
    Session statistics: INACTIVE
    Session timeout: 30 seconds
    Session memory cap: 8388608 bytes
    State alerts: INACTIVE
    Scan alerts: INACTIVE
    Log Flushed Streams: INACTIVE
Stream4_reassemble config:
    Server reassembly: ACTIVE
    Client reassembly: ACTIVE
    Reassembler alerts: INACTIVE
    Ports:
21 23 25 53 80 110 111 143 513 

        --== Initialization Complete ==--

-*> Snort! <*-
Version 1.8.4-beta1 (Build 91-cpw) <- has the fix to stream4
By Martin Roesch (roesch () sourcefire com, www.snort.org)

<I started this run at midnight, Mon Feb 25 00:00:41 MST 2002>
<Not much going on ... until about 14 minutes after midnight>

S:1014620441.430771 1363 0 1363 0 8314 5578470 940118 251 2644 23 0
...
S:1014621255.022916 139312 0 154992 0 163235 162075725 140157838 67 19300 123763 0
<                          I got 123763 consec pkts before making a poll   ^
<                          I'd say that is rapid fire and I'm running a complete
<                          rules set.
S:1014621265.416852 142875 5524 139683 0 147533 156107421 152480899 34 31103 95637 0
<                          ^ first bunch of lost packets
S:1014621275.817468 92241 0 85281 0 91856 97717880 100403174 46 25040 14158 0
...
S:1014626574.741987 158648 0 160582 0 166374 168058576 160948837 25 2865 140178 0
S:1014626585.125144 104143 12940 99966 0 105523 113214521 115009893 6 8704 82468 0
<                          ^ next bunch
S:1014626595.125235 40423 0 40416 0 46746 48843045 45224632 132 16359 7077 0
...
S:1014631977.063592 145391 0 160647 0 167204 165579852 145931642 71 16546 144890 0
S:1014631987.457935 139986 1993 131855 0 138068 144289195 147683044 22 25460 79744 0
<                          ^ and so on.
S:1014631997.857827 56924 0 51794 0 57508 60640440 63126045 51 16848 6847 0
...
S:1014637583.395733 82402 0 85109 0 89933 89132913 83570190 139 17542 13992 0
S:1014637593.788293 150279 4714 182199 0 188692 195967675 155751704 58 3981 52839 0
S:1014637604.188202 165102 64157 212555 0 218217 237212247 183437745 0 5243 165102 0
S:1014637614.588008 30751 0 18284 0 23037 22869529 33605874 147 3226 13336 0
...
S:1014642623.746178 148740 0 150605 0 158189 155964364 149571236 31 17151 148072 0
S:1014642633.746479 127214 27358 176443 0 185878 198150700 138479962 50 13293 107855 0
S:1014642643.746553 134251 50060 160594 0 168377 180998686 148348228 1 16472 134250 0
S:1014642654.134881 22023 0 21996 0 30435 29287058 24159004 170 5727 2031 1
...
S:1014647855.671686 142034 0 154790 0 186061 183344466 151695329 31 30207 131340 0
S:1014647866.059864 105741 325 93312 0 120592 114396560 111917959 44 4876 105070 0
S:1014647876.468611 34161 0 34838 0 62716 52652182 36219653 92 6269 2891 0
...
S:1014653550.812066 99520 0 125997 0 166847 147396141 99303733 106 2742 98355 0
S:1014653561.188873 156270 15513 170961 0 208031 198561395 161589137 13 27940 90782 0
S:1014653571.588728 110433 1301 87604 0 119141 117118038 124641248 6 7301 69102 0
S:1014653581.989388 25825 0 25357 0 63692 49404451 28187719 130 358 9390 0
...
S:1014658949.122030 4464 0 4470 0 51871 31365883 3570214 86 22098 115 0
S:1014658959.122174 96677 11401 132805 0 193063 168195121 96158198 34 20471 93443 0
S:1014658969.521674 137541 18978 153507 0 205170 187966765 140056389 2 26940 90521 0
S:1014658979.921529 115286 0 93565 0 136425 131576643 129185684 4 11154 93994 0
...
S:1014664427.055062 69243 0 100984 0 160021 132998026 68546006 90 27458 66962 0
S:1014664437.454664 138958 21160 147423 0 194384 175974405 140393421 7 2576 111812 0
S:1014664447.854521 155094 35863 204661 0 263426 251271401 165321238 0 26598 155094 0
S:1014664458.254336 56381 1985 25912 0 65999 49587209 59655084 36 17443 45213 0
S:1014664468.654283 33015 0 33070 0 83278 64280118 35856883 55 17690 4268 0
...
S:1014706785.139327 1223 0 1223 0 7481 4371928 727505 240 7448 25 0
pcap_loop: User specified timeout occured

<          ^ I specified to libpcap that I wanted to quit at midnight or close
             to it:  1014706785 is Mon Feb 25 23:59:45 MST 2002

===============================================================================
< looks like I still might have some problem with my snort counts.

Snort analyzed 28221240 out of 28780841 packets, dropping 559601(1.944%) packets

Breakdown by protocol:                Action Stats:
    TCP: 26669755   (90.898%)         ALERTS: 7287      
    UDP: 1220552    (4.160%)          LOGGED: 7287      
   ICMP: 257564     (0.878%)          PASSED: 7598488   
    ARP: 0          (0.000%)
   IPv6: 0          (0.000%)
    IPX: 0          (0.000%)
  OTHER: 70991      (0.242%)
DISCARD: 0          (0.000%)
===============================================================================
Fragmentation Stats:
Fragmented IP Packets: 4747       (0.016%)
    Fragment Trackers: 2378      
   Rebuilt IP Packets: 2369      
   Frag elements used: 4738      
Discarded(incomplete): 0         
   Discarded(timeout): 9         
  Frag2 memory faults: 0         
===============================================================================
TCP Stream Reassembly Stats:
        TCP Packets Used: 26665859   (90.884%)
         Stream Trackers: 903596    
          Stream flushes: 3155332   
           Segments used: 5918480   
   Stream4 Memory Faults: 0         
===============================================================================

On Tue, Feb 26, 2002 at 05:12:43PM -0600, Chris Green wrote:
> Erek Adams <erek () theadamsfamily net> writes:
>
> > On Tue, 26 Feb 2002, Chris Green wrote:
> >
> > > FYI, its BPF/libpcap performance and not TCP stack performance that is the
> > > issue when it comes to snort
> >
> > Ok, with that being said, here's a question:  Is it worth upgrading to another
> > version of libpcap each time it comes out?  Or tracking it's CVS as well?
>
> If you are running linux on your IDS stuff, its worth it for hearing
> about the things they do to turbo packet stuff now and then.
>
> > Along those lines, would there be any useful TCP/IP stack parameters to
> > tune/change, or would that just be a waste of effort?
>
> I'm sure there are things like memcaps for bpf and the like to set and
> I'd love to see a good technical paper other than the winpcap one on
> pcap performance's as well as tuning.
>
> I'm also pretty sure one of the security websites would even pay a bit
> for an article on tuning of pcap performance.
>
> Most everything I've seen in the past few years ranks right above
> voodoo.
> -- 
> Chris Green <cmg () uab edu>
> Don't use a big word where a diminutive one will suffice.
>
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users

-- 
Phil Wood, cpw () lanl gov


_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users