Snort mailing list archives
Gigabit and Snort
From: "Gross, Mark" <mgross () microstrategy com>
Date: Tue, 10 Aug 2004 22:18:59 -0400
Hello List, I am having a few problems with losing packets at high speeds. First a bit about the hardware. Assume that all connections are fiber. Cisco 65xx <---------> Cisco 65xx (failover with 4 fiber trunks) | | <-- Sniffing Interfaces Snort Sensor Snort Sensor | | | Database | | | | <-- Management Network ------ Cisco 65xx ------ Each sensor is a Dell 2650 , 2 x 2.8 Xeon, 4GB Ram, 720GB Internal Storage, 2 x Intel 1000 PCI-X Gigabit NICs. The Database is a Dell 2650, 2 x 2.8 Xeon, 12GB Ram, 720GB Internal Storage, Powervault Array with 1.7TB Storage using the PERC4 Raid Controller, and 1 x Intel 1000 PCI-X Gigabit Nic. OS is RedHat Enterprise Linux AS. Users passing through the sensors (via 14 VLANS) are about 800. All traffic and users MUST pass the sensors, even for internal communications. That's the basic setup. So my problem is that the sensors are dropping ungodly amounts of traffic at very high speeds (see below). I have read some of the other articles about the buffers and all, and on one box I set them and it showed minimal improvement. So I am kind of lost as to a cause. Here are the stats of one of the sensors. There arn't actually 4 CPUs , that's the Hyperthreading crap.. DoH. Snort is hanging right around 50 Procent CPU time and Memory is good. There is no logging on the sensors and everything is going over the management network to the MySQL database. When I do a tcpdump, snort on the interface you can see the packet loss (see below). Also included are the Interface stats. The card is not maxed either. I plan on setting up Barnyard very soon, but I can't imagine this would create such a packet loss. The snort.conf is a straight out of the box with the home_net as ANY. Otherwise just the MySQL output plugin is configured. Thanks in advance. Mark ------------------------------------------------------------------------ ------------------------------------------ CPU/Memory Statistics 21:40:31 up 3:01, 1 user, load average: 0.87, 0.74, 0.65 58 processes: 56 sleeping, 2 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 14.8% 0.0% 5.2% 1.8% 6.5% 0.2% 71.3% cpu00 0.0% 0.0% 0.0% 7.0% 26.0% 0.0% 66.9% cpu01 42.4% 0.0% 15.4% 0.2% 0.0% 0.2% 41.8% cpu02 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% cpu03 17.0% 0.0% 5.6% 0.0% 0.0% 0.6% 76.8% Mem: 3998688k av, 2519888k used, 1478800k free, 0k shrd, 90812k buff 411456k active, 1974520k inactive Swap: 2040244k av, 0k used, 2040244k free 2173236k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 6982 root 25 0 53620 50M 1192 R 18.6 1.2 45:36 1 snort ------------------------------------------------------------------------ ------------------------------------------ TCPDUMP Statistics after about a 5 min dump at night. 13559511 packets received by filter 13503328 packets dropped by kernel ------------------------------------------------------------------------ ------------------------------------------ INTERFACE statistics after 30 seconds. Interface Received Sent Total Kbps Kbps Kbps eth2 863163.60 0.00 863163.60 |---- Max 866570.72 0.00 866570.72 |---- Avg 831803.76 0.00 831803.76 ------------------------------------------------------------------------ ------------------------------------------ SNORT after 30 seconds: ======================================================================== ======= Snort analyzed 2930003 out of 5366187 packets, dropping 2436184(45.399%) packets Breakdown by protocol: Action Stats: TCP: 491713 (9.163%) ALERTS: 181 UDP: 1633 (0.030%) LOGGED: 181 ICMP: 65 (0.001%) PASSED: 0 ARP: 91 (0.002%) EAPOL: 0 (0.000%) IPv6: 0 (0.000%) IPX: 0 (0.000%) OTHER: 274 (0.005%) DISCARD: 0 (0.000%) ======================================================================== ======= Wireless Stats: Breakdown by type: Management Packets: 0 (0.000%) Control Packets: 0 (0.000%) Data Packets: 0 (0.000%) ======================================================================== ======= Fragmentation Stats: Fragmented IP Packets: 3 (0.000%) Fragment Trackers: 2 Rebuilt IP Packets: 0 Frag elements used: 0 Discarded(incomplete): 0 Discarded(timeout): 0 Frag2 memory faults: 0 ======================================================================== ======= TCP Stream Reassembly Stats: TCP Packets Used: 491713 (9.163%) Stream Trackers: 1000 Stream flushes: 72 Segments used: 257 Stream4 Memory Faults: 0 ======================================================================== ======= Final Flow Statistics ,----[ FLOWCACHE STATS ]---------- Memcap: 10485760 Overhead Bytes 16400 used(%1.906099)/blocks (199869/1284) Overhead blocks: 1 Could Hold: (73326) IPV4 count: 1283 frees: 0 low_time: 1092189785, high_time: 1092189816, diff: 0h:00:31s finds: 493492 reversed: 176397(%35.744652) find_sucess: 492209 find_fail: 1283 percent_success: (%99.740016) new_flows: 1283 Protocol: 1 (%0.013171) finds: 65 reversed: 5(%7.692308) find_sucess: 20 find_fail: 45 percent_success: (%30.769231) new_flows: 45 Protocol: 6 (%99.654098) finds: 491785 reversed: 176168(%35.822158) find_sucess: 490786 find_fail: 999 percent_success: (%99.796862) new_flows: 999 Protocol: 17 (%0.330907) finds: 1633 reversed: 224(%13.717085) find_sucess: 1401 find_fail: 232 percent_success: (%85.793019) new_flows: 232 Protocol: 89 (%0.001621) finds: 8 reversed: 0(%0.000000) find_sucess: 2 find_fail: 6 percent_success: (%25.000000) new_flows: 6 Protocol: 103 (%0.000203) finds: 1 reversed: 0(%0.000000) find_sucess: 0 find_fail: 1 percent_success: (%0.000000) new_flows: 1 database: Closing connection to database "" Snort exiting
Current thread:
- Gigabit and Snort Gross, Mark (Aug 16)
- Re: Gigabit and Snort Edin Dizdarevic (Aug 16)
- <Possible follow-ups>
- RE: Gigabit and Snort Kreimendahl, Chad J (Aug 17)