Re: HUGE packet-drop

[Gabe Black <Gabe.Black () jdsu com>]

Just some ideas to help.  I have an application that is trying to capture, modify, resequence, and route very small 
packets.  The configuration I found the best is the following (for kernel parameters):

sysctl -w net.core.wmem_max=1310710
sysctl -w net.core.rmem_max=1310710
sysctl -w net.core.wmem_default=1249280
sysctl -w net.core.rmem_default=1249280
sysctl -w net.core.netdev_budget=1024
sysctl -w net.core.dev_weight=128
sysctl -w net.core.netdev_max_backlog=10000
sysctl -w net.ipv4.udp_mem="11416320 15221760 22832640"
sysctl -w net.core.optmem_max=40960

I also used ethtool with the -G option to increase the nic's hardware buffers to their max. 

The machine it is running on is pretty beefy, but cpu cycles are not limiting the throughput.  I found the factor that 
effected performance the most is cache coherency and limiting inter-process interrupts.

The best way I found to do this was to have the application that is receiving the packets running on the same cores 
that the kernel is pulling them off the nic.  Since my machine had two chips each with 4-6 cores (8-12 logical cores), 
I limited my application to run on the same chip as the nic was receiving its interrupts.

To control which logical core the nic handles the interrupts:

cat /proc/interrupts | grep eth[number]

That should show you the irq(s) that handles the interrupts.  

Then do something like:

echo "1" > /proc/irq/[irq#_from_above]/smp_affinity 

to set what core handles the interrupt (1 is a bitmask saying to use the first logical core).

Finally do something like:

taskset -p 0x5 [your application i.e. tcpdump] (0x5 is a bitmask saying which cpu cores to use) to control what cores 
the process gets scheduled on.

I found this the best way to get the most throughput out of the application, and the most likely way to reproduce 
results.  Without doing this, I would get some runs that performed well, and others that were junk, and most that 
wavered.

Oh, stop the irqbalancer service as that may overwrite what you did to the /proc/irq/#

Also use top to monitor your top processes.  If you see a ksoftirqd service getting lots of cpu cycles, you will be 
dropping packets and may be a good indicator that you should try a different irq#.

Good luck.  If you find any other tips to squeeze more performance please write back as I am also trying to squeeze the 
most performance possible out of my application.


- Gabe Black

-----Original Message-----
From: tcpdump-workers-owner () lists tcpdump org [mailto:tcpdump-workers-owner () lists tcpdump org] On Behalf Of M. V.
Sent: Sunday, January 23, 2011 9:42 AM
To: tcpdump-workers
Subject: [tcpdump-workers] HUGE packet-drop

hi guys,

i'm stucked with a confusing problem. i'm trying to dump gigabit network traffic 
to file. i'm using Debian5.0.3 AMD64 on an HP Proliant DL360 G5 (2*Quad-Core 
2.33 hz - 4GiB RAM - 2*76GiB HardDisk). but whatever i try, even at 600Mbps 
rate, i got a huge packet-drop.

*) i tried both tcpdump (1.1.1) && dumpcap (1.4.3) : (got similar results)
- dumpcap -i eth0 -w /dump_folder/dump.pcap [-b filesize:150000] -s 0
- tcpdump -i eth0 -w /dump_folder/dump.pcap -s 0

*) i tried recompiling those two, with different libpcap versions (0.9.8 has 
best result, and i tried 1.0 and .1.1.1)

*) i tried playing with kernel parameters:
net.core.rmem_default = 40971510
net.core.rmem_max = 40971510
net.core.wmem_default = 40971510
net.core.wmem_max = 40971510
net.core.netdev_max_backlog = 300000
net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1
net.core.optmem_max = 2048000
....

*) i even tried writing on different disks (SDD, RamDisk): no change in results.
-----------------------------------------------------------------------------------------

the result is not good at all. these are my tests and results:
* iperf server: iperf -s -u 5
* iperf client : iperf -c ... -u -b 600m -t 10000

for dumping traffic at 600Mbps for 150 seconds, this is the best result i had so 
far:
total received packets: ~8,000,000
dropped packets: ~3,500,000

so my best result is more than 40% packet-drop, which is a disaster.

can anyone help me with this? any suggestions? am i doing something terribly 
wrong?

thank you.


      -
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.