RE: Looking for advice to improve performance

["Price, Jason" <Jason.Price () thomson com>]

> What type of link are you trying to monitor?
It's a gig fiber link, pulling between 20 and 25Mbps.

> What is the hardware platform? CPU type & speed? 
IBM Netfinity 6000.  Quad PIII 700mHz processors.  1GB memory.

> What interface/driver/version are you using?
Intel Pro/1000 with the Intel e1000 default driver.
 
> What is the storage medium? Local? Network? Array? Single disk?
Originally was writing to an external array, but testing using the local
drives produced the same % of dropped packets.  Have tested 3 disk systems:
- Local RAID 1
- External RAID 5
- External RAID 0
No tcpdump performance differences between any of them, which leads me to
believe the disk subsystem isn't the issue.

> What is the tcpdump commandline & filter being used? 
The command being called by Shadow is:
"/usr/sbin/tcpdump -i eth1 -w - -F /usr/local/SHADOW/sensor/std.filter"

Additionally, I have tried running tcpdump with no arguments (other than
interface and output file), and the % of packets dropped decreases, but it's
still dropping a fair amount.  It appears that the filter is contributing to
the # of dropped packets, but is not the root cause.

> Is the sensor system being asked to do anything else besides sniff & serve
sshd?
It's doing httpd, sshd, tcpdump, and gzip.  Shadow gzips the traffic logs on
the fly.  Httpd is for another part of Shadow, but isn't being used
currently.  The daemon is up, but it's not doing any work for now.

When running Shadow, CPU utilization maxes out around 80-85% on a single
processor (primarily due to gzip) - the other 3 are relatively unused.  When
running just tcpdump, cpu utilization is 5-10%.

As for the rest of the suggestions, they are already in place.  The box was
built very bare-bones (outside of recompiling the kernel), and is solely for
this purpose.

Thanks for the input so far.

Jason

-----Original Message-----
From: George Bakos [mailto:gbakos () ists dartmouth edu]
Sent: Wednesday, January 07, 2004 11:19 PM
To: Price, Jason
Cc: 'tcpdump-workers () tcpdump org'
Subject: Re: [tcpdump-workers] Looking for advice to improve performance


The "xxx packets dropped by kernel" message you are seeing indicates that
there is a critical path bottleneck, likely in the storage channel.

A few questions: 
What type of link are you trying to monitor?
What is the hardware platform? CPU type & speed? 
What interface/driver/version are you using? 
What is the storage medium? Local? Network? Array? Single disk?
What is the tcpdump commandline & filter being used? 
Is the sensor system being asked to do anything else besides sniff & serve
sshd?

Here are a few simple guidelines that may help:

 - Simple is better. Minimize the filter used on the sensor & limit the
snaplen for your broad capture. 

 - Faster is better. Locally attached disks are much faster than network
attached storage. If you are using an array, be sure it has plenty of RAM
available for buffered writes. 

 - Lighter is better. Minimize the load on the sniffer. Redhat likes prety
stuff; get rid of it. If you MUST run a GUI, use a lightweight one, and no
xscreensaver, for cryin' out loud. Best bet is to boot to runlevel 3
(non-graphical mode) before putting it into production. When sniffing at
peak network load, what is the cpu utilization? Use the "top" utility to
view running processes, memory usage & CPU states.

 - Wide awake is better. Ensure there are no power management options
turned on in the BIOS or kernel. You don't want your disks spinning down
during quiet times, only to drop packets when that attack comes down the
pipe at 3am.

g

On Wed, 7 Jan 2004 16:10:49 -0600 
"Price, Jason" <Jason.Price () thomson com> wrote:

> I am trying to use tcpdump in conjunction with Shadow (on RedHat Advanced
> Server 3) to log all data coming into our organization.  This is a very
high
> volume of data, and tcpdump seems unable to handle it.  Currently, about
40%
> of incoming packets are being dropped by the kernel.
>
> What are my options for improving the throughput of tcpdump?
>
> I'm relatively new to the linux world, so be gentle...  :)
>
> Jason
> -
> This is the TCPDUMP workers list. It is archived at
> http://www.tcpdump.org/lists/workers/index.html
> To unsubscribe use
mailto:tcpdump-workers-request () tcpdump org?body=unsubscribe


-- 
George Bakos
Institute for Security Technology Studies - IRIA
Dartmouth College
gbakos () ists dartmouth edu
603.646.0665 -voice
603.646.0666 -fax
-
This is the TCPDUMP workers list. It is archived at
http://www.tcpdump.org/lists/workers/index.html
To unsubscribe use mailto:tcpdump-workers-request () tcpdump org?body=unsubscribe