Re: Poor performance with Snort 2.9.4.6 under OpenBSD 5.3

["C. L. Martinez" <carlopmart () gmail com>]

On Fri, Jun 7, 2013 at 10:36 AM, C. L. Martinez <carlopmart () gmail com> wrote:
> On Thu, Jun 6, 2013 at 7:36 AM, C. L. Martinez <carlopmart () gmail com> wrote:
> > On Wed, Jun 5, 2013 at 5:08 PM, Victor Roemer <vroemer () sourcefire com> wrote:
> > > Martinez, as Joel already mentioned, we'll want to see your Snort
> > > configuration. Shutdown stats would also be useful, but perfmon data would
> > > be better; if those can be provided.
> > >
> > > You mentioned that OpenBSD configured the network sysctl parameters "on the
> > > fly"; could you direct us to some documentation about this?
> > >
> > > You also mentioned that Snort was listening on em3, however the startup
> > > information in your email indicates that Snort is listening on em4, could
> > > you elaborate on this setup?
> > >
> > >
> > > Regarding Suricata, I personally do not have any experience in deploying or
> > > configuring it. That said, are you using, relatively, the same
> > > configurations? (e.g., any rules enabled, acquiring packets via libpcap,
> > > etc..)
> > >
> > > Also, why are "tcp.reassembly_gap" and "tcp.invalid_checksum" relevant?
> >
> > Ok, first of all sorry for this long post. I will try to answer to all
> > questions. First, my environment:
> >
> > Host: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 8 GiB RAM
> > Snort: release 2.9.4.6 compiled against libpcap 1.3.0
> > Suricata: release 1.4.2 compiled against libpcap 1.3.0
> >
> > Snort is listening in em4 interface and suricata in em5 (there was a
> > typo error previously).
> >
> > Performance data using my actual snort.conf (with few rules and
> > without so_rules or IP based rules):
> >
> > Packet Performance Monitor Config:
> >   ticks per usec  : 2405 ticks
> >   max packet time : 10000 usecs
> >   packet action   : fastpath-expensive-packets
> >   packet logging  : log
> >   debug-pkts      : disabled
> >
> > Rule Performance Monitor Config:
> >   ticks per usec  : 2405 ticks
> >   max rule time   : 4096 usecs
> >   rule action     : suspend-expensive-rules
> >   rule threshold  : 5
> >   suspend timeout : 10 secs
> >   rule logging    : log
> > pcap DAQ configured to passive.
> > Acquiring network traffic from "em4".
> > Reload thread starting...
> > Reload thread started, thread 0x1ee8c530a500 (4050)
> > Decoding Ethernet
> >
> >         --== Initialization Complete ==--
> >
> >    ,,_     -*> Snort! <*-
> >   o"  )~   Version 2.9.4.6 GRE (Build 73)
> >    ''''    By Martin Roesch & The Snort Team:
> > http://www.snort.org/snort/snort-team
> >            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> >            Using libpcap version 1.3.0
> >            Using PCRE version: 8.31 2012-07-06
> >            Using ZLIB version: 1.2.3
> >
> >            Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.17  <Build 18>
> >            Preprocessor Object: SF_DNP3  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_MODBUS  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_GTP  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_REPUTATION  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_SIP  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_SDF  Version 1.1  <Build 1>
> >            Preprocessor Object: SF_DCERPC2  Version 1.0  <Build 3>
> >            Preprocessor Object: SF_SSLPP  Version 1.1  <Build 4>
> >            Preprocessor Object: SF_DNS  Version 1.1  <Build 4>
> >            Preprocessor Object: SF_SSH  Version 1.1  <Build 3>
> >            Preprocessor Object: SF_SMTP  Version 1.1  <Build 9>
> >            Preprocessor Object: SF_IMAP  Version 1.0  <Build 1>
> >            Preprocessor Object: SF_POP  Version 1.0  <Build 1>
> >            Preprocessor Object: SF_FTPTELNET  Version 1.2  <Build 13>
> > Commencing packet processing (pid=4050)
> > ===============================================================================
> > Run time for packet processing was 368.552024 seconds
> > Snort processed 643495 packets.
> > Snort ran for 0 days 0 hours 6 minutes 8 seconds
> >    Pkts/min:       107249
> >    Pkts/sec:         1748
> > ===============================================================================
> > Packet Performance Summary:
> >    max packet time       : 10000 usecs
> >    packet events         : 0
> >    avg pkt time          : 8.95128 usecs
> > Rule Performance Summary:
> >    max rule time         : 4096 usecs
> >    rule events           : 0
> >    avg rule time         : 1.96408 usecs
> > ===============================================================================
> > Packet I/O Totals:
> >    Received:      2121798
> >    Analyzed:       643495 ( 30.328%)
> >     Dropped:       618918 ( 22.582%)
> >    Filtered:            0 (  0.000%)
> > Outstanding:      1478303 ( 69.672%)
> >    Injected:            0
> > ===============================================================================
> >
> > And here they are some statistics outputs:
> >
> > top output:
> >
> > load averages:  0.18,  0.18,  0.13
> > 21 processes: 20 idle, 1 on processor
> > CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.6% interrupt, 99.4% idle
> > CPU1 states:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.0% idle
> > CPU2 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> > CPU3 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> > Memory: Real: 506M/2942M act/tot Free: 5018M Cache: 2319M Swap: 0K/6142M
> >
> >   PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
> >  4050 root       4    0  417M  151M sleep/0   bpf       0:06  2.83% snort
> > 31522 root      10    0  356M  328M sleep/0   nanosle  35:09  0.73% suricata
> >
> > Status interfaces (systat ifstat):
> >
> >     2 users    Load 0.17 0.17 0.13
> >
> > IFACE            STATE  DESC
> >                      IPKTS   IBYTES    IERRS    OPKTS   OBYTES
> > OERRS    COLLS
> > em0              up
> >                          1       72        0        0       71
> > 0        0
> > em1              up
> >                          0        0        0        0        0
> > 0        0
> > em2              up
> >                          0        0        0        0        0
> > 0        0
> > em3              up
> >                          1      334        0        2     1732
> > 0        0
> > em4              up
> >                       6773  4771755        0        0        0
> > 0        0
> > em5              up
> >                       6773  4771755        0        0        0
> > 0        0
> >
> > Interfaces configruation options:
> >
> > em4: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> > mtu 1514
> >         lladdr 00:50:56:17:fc:cd
> >         priority: 0
> >         media: Ethernet autoselect (1000baseT full-duplex,master)
> >         status: active
> >         inet6 fe80::250:56ff:fe17:fccd%em4 prefixlen 64 scopeid 0x5
> > em5: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> > mtu 1514
> >         lladdr 00:50:56:18:79:51
> >         priority: 0
> >         media: Ethernet autoselect (1000baseT full-duplex,master)
> >         status: active
> >         inet6 fe80::250:56ff:fe18:7951%em5 prefixlen 64 scopeid 0x6
> >
> > Checking mbuffers (systat mbufs):
> >
> >     2 users    Load 0.18 0.17 0.13
> >
> > IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> > System                    0   256   361          59
> >                                2k   351         457
> > lo0
> > em0                            2k     7     4   256     7
> > em1                            2k     6     4   256     6
> > em2                            2k     4     4   256     4
> > em3                            2k     7     4   256     7
> > em4                            2k    69     4   256    69
> > em5                            2k    73     4   256    73
> >
> > sysctl.conf options enabled:
> > kern.bufcachepercent=80
> > net.bpf.bufsize=65536
> > net.bpf.maxbufsize=2097152
> >
> > Ok, now some Suricata stats:
> >
> > 5/6/2013 -- 14:29:09 - <Info> - stream "max-sessions": 262144
> > 5/6/2013 -- 14:29:09 - <Info> - stream "prealloc-sessions": 32768
> > 5/6/2013 -- 14:29:09 - <Info> - stream "memcap": 33554432
> > 5/6/2013 -- 14:29:09 - <Info> - stream "midstream" session pickups: disabled
> > 5/6/2013 -- 14:29:09 - <Info> - stream "async-oneside": disabled
> > 5/6/2013 -- 14:29:09 - <Info> - stream "checksum-validation": enabled
> > 5/6/2013 -- 14:29:09 - <Info> - stream."inline": disabled
> > 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "memcap": 67108864
> > 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "depth": 1048576
> > 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toserver-chunk-size": 2560
> > 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toclient-chunk-size": 2560
> > 5/6/2013 -- 14:29:09 - <Info> - all 7 packet processing threads, 3
> > management threads initialized, engine started.
> > 5/6/2013 -- 14:29:13 - <Info> - No packets with invalid checksum,
> > assuming checksum offloading is NOT used
> > 6/6/2013 -- 06:27:20 - <Info> - Signal Received.  Stopping engine.
> > 6/6/2013 -- 06:27:20 - <Info> - 0 new flows, 0 established flows were
> > timed out, 0 flows in closed state
> > 6/6/2013 -- 06:27:21 - <Info> - time elapsed 57492.090s
> > 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Packets 7317731, bytes 5500227933
> > 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Pcap Total:2974089715
> > Recv:2974089715 Drop:0 (0.0%).
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Total flow handler queues - 6
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 0  - pkts: 2754800
> > flows: 19921
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 1  - pkts: 2158501
> > flows: 15267
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 2  - pkts: 1276685
> > flows: 9932
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 3  - pkts: 678835
> > flows: 5843
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 4  - pkts: 358226
> > flows: 3109
> > 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 5  - pkts: 127487
> > flows: 1488
> >
> > -------------------------------------------------------------------
> > Date: 6/6/2013 -- 06:27:21 (uptime: 0d, 15h 58m 44s)
> > -------------------------------------------------------------------
> > Counter                   | TM Name                   | Value
> > -------------------------------------------------------------------
> > capture.kernel_packets    | RxPcapem51                | 2974088271
> > capture.kernel_drops      | RxPcapem51                | 0
> > capture.kernel_ifdrops    | RxPcapem51                | 0
> > decoder.pkts              | RxPcapem51                | 7317731
> > decoder.bytes             | RxPcapem51                | 5500227933
> > decoder.ipv4              | RxPcapem51                | 7317731
> > decoder.ipv6              | RxPcapem51                | 0
> > decoder.ethernet          | RxPcapem51                | 7317731
> > decoder.raw               | RxPcapem51                | 0
> > decoder.sll               | RxPcapem51                | 0
> > decoder.tcp               | RxPcapem51                | 7317660
> > decoder.udp               | RxPcapem51                | 0
> > decoder.sctp              | RxPcapem51                | 0
> > decoder.icmpv4            | RxPcapem51                | 71
> > decoder.icmpv6            | RxPcapem51                | 0
> > decoder.ppp               | RxPcapem51                | 0
> > decoder.pppoe             | RxPcapem51                | 0
> > decoder.gre               | RxPcapem51                | 0
> > decoder.vlan              | RxPcapem51                | 0
> > decoder.teredo            | RxPcapem51                | 0
> > decoder.ipv4_in_ipv6      | RxPcapem51                | 0
> > decoder.ipv6_in_ipv6      | RxPcapem51                | 0
> > decoder.avg_pkt_size      | RxPcapem51                | 752
> > decoder.max_pkt_size      | RxPcapem51                | 1506
> > defrag.ipv4.fragments     | RxPcapem51                | 0
> > defrag.ipv4.reassembled   | RxPcapem51                | 0
> > defrag.ipv4.timeouts      | RxPcapem51                | 0
> > defrag.ipv6.fragments     | RxPcapem51                | 0
> > defrag.ipv6.reassembled   | RxPcapem51                | 0
> > defrag.ipv6.timeouts      | RxPcapem51                | 0
> > defrag.max_frag_hits      | RxPcapem51                | 0
> > tcp.sessions              | Detect                    | 41460
> > tcp.ssn_memcap_drop       | Detect                    | 0
> > tcp.pseudo                | Detect                    | 5705
> > tcp.invalid_checksum      | Detect                    | 12
> > tcp.no_flow               | Detect                    | 0
> > tcp.reused_ssn            | Detect                    | 0
> > tcp.memuse                | Detect                    | 36175872
> > tcp.syn                   | Detect                    | 80827
> > tcp.synack                | Detect                    | 80035
> > tcp.rst                   | Detect                    | 33845
> > tcp.segment_memcap_drop   | Detect                    | 0
> > tcp.stream_depth_reached  | Detect                    | 148
> > tcp.reassembly_memuse     | Detect                    | 67770240
> > tcp.reassembly_gap        | Detect                    | 2222
> > detect.alert              | Detect                    | 29
> > flow_mgr.closed_pruned    | FlowManagerThread         | 49711
> > flow_mgr.new_pruned       | FlowManagerThread         | 3558
> > flow_mgr.est_pruned       | FlowManagerThread         | 1974
> > flow.memuse               | FlowManagerThread         | 3884096
> > flow.spare                | FlowManagerThread         | 10001
> > flow.emerg_mode_entered   | FlowManagerThread         | 0
> > flow.emerg_mode_over      | FlowManagerThread         | 0
> >
> > As you can see, packets dropped counter shows 0% (well, in fact this
> > is not always the case. Suricata may lost between 1-2% of packets).
> > And this suricata sensor is configured to use IP based rules, when
> > snort not.
> > But there are two counters that can shows if some problems exists:
> > tcp.reassembly_gap and tcp.invalid_checksum. But as you can see, are
> > minimal according to Suricata docs:
> >
> > https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Statistics
> >
> > When I say: "OpenBSD tunes them "on the fly" according to network
> > load", I would say that OpenBSD 5.1 and later will dynamically adjust
> > the TCP send and receive window sizes. Please, refer to:
> >
> > http://marc.info/?l=openbsd-misc&m=130804020930481&w=2
> > http://marc.info/?l=openbsd-misc&m=128904951710762&w=2
> > http://www.openbsd.org/faq/faq6.html#Tuning
> > http://quigon.bsws.de/papers/2013/AsiaBSDcon/cksums.pdf
> >
> > Do you need to add more info or perform more tests?
> >
> > And many thanks for your help.
>
>
> Ok, more info. I have disabled SMP kernel version and changed by
> uni-processor kernel (bsd) and now performance has grown: packets
> dropped are between 5-20% now ...
>
> Maybe all my problems are with system interrupts??

One question: it could be possible to use libpcap provided by OpenBSD
instead of install from www.tcpdump.org's site??

For example, I am trying to build daq against OpenBSD's libpcap and:

checking libnetfilter_queue/libnetfilter_queue.h usability... no
checking libnetfilter_queue/libnetfilter_queue.h presence... no
checking for libnetfilter_queue/libnetfilter_queue.h... no
checking for linux/netfilter.h... (cached) no
checking for pcap.h... (cached) yes
checking for pcap_lib_version... checking for pcap_lib_version in
-lpcap... (cached) yes
checking for libpcap version >= "1.0.0"... no

    ERROR!  Libpcap library version >= 1.0.0  not found.
    Get it from http://www.tcpdump.org

Suricata builds without problems using libpcap provided by OpenBSD ...

I have attached config.log.

Thanks.

Attachment: config.log
Description:

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!