tcpdump mailing list archives
libpcap delivering the same packet more than once under high load?
From: Jim Lloyd <jlloyd () silvertailsystems com>
Date: Fri, 10 Sep 2010 11:34:57 -0700
We use libpcap 1.1.1 on linux 2.6.18 (CentOS 5.5) to sniff http traffic. I have a test driver application written with libcurl than uses the curl multi API to simultaneously download (using HTTP GET requests) N jpg images from a server running apache httpd. This combination is able to saturate a GigE connection in the download direction, and our libpcap application is able to reconstruct the TCP streams and parse the HTTP messages with no errors. This is excellent (great job guys!) and we're very pleased with the performance. But it appears that libpcap does have anomalous behavior when the GigE link is saturated. We track the amount of bits received from libpcap by accumulating pcap_pkthdr.len*8 for each packet received. We also track other useful statistics. Here are results from a number of runs: N cap mbps packets data dup requests eth mbps 1 350 22100 6988 34 469 369 2 752 44547 14013 364 961 757 3 977 52289 15603 1400 1138 906 4 1143 55223 16025 2569 1200 950 5 1333 57891 16681 4885 1234 978 6 1424 57834 16192 6033 1262 998 10 2149 58213 15662 12529 1271 1003 20 3282 49607 14133 17101 1272 997 30 3169 42122 13481 14953 1271 975 The columns are: N number of concurrent requests cap mbps The Mbps as measured by accumulating pcap_pkthdr.len*8 (and then dividing by 2^20, not 10^6). packets The total packets per second received from lipbcap data The number of data packets per second that were used in reconstructing the TCP stream dup The number of data packets that were discarded as duplicates while reconstructing the TCP stream requests The number of HTTP requests parsed from the TCP streams eth mbps The mbps seen on the eth NIC, computed by polling /proc/net/dev. Includes both inbound & output data. The data shows that we effectively saturate the GigE link at around 5 concurrent connections, where we see peak or near-peak rates for most metrics. As we increase the number of concurrent connections, the number of requests per second stays relatively flat, as does the Mbps seen at the ethernet NIC. But strangely, the Mbps of the data delivered by libpcap to the application continues to climb, to up to about 3Gbps. This increase in the data rate seems to be due to the continuing increase in the number of duplicate packets per second. These duplicate packets cannot be unique packets that were retransmitted between the two machines on the layer 1 GigE link, because if there was a significant increase in retransmission duplicates on the link, they would have competed for the fixed 1Gbps channel capacity. But the data is consistent with libpcap delivering the same packets more than once. Is this known behavior? It's relatively benign behavior and we don't expect to see saturated links in any production environment, so if this is a bug in libpcap, it's probably low priority to fix. FYI we use pcap_dispatch, and our dispatch callback simply copies the packet and then queues the copied packet onto one of several queues that are each consumed by a different worker thread. The statistics for "packets" are gathered by the main thread. The statistics for "data" and "dup" are gathered in the worker threads. We do have cross-checks to ensure that the total packet rate seen by the worker threads is consistent with the packet rate seen by the main thread. Thanks, Jim Lloyd Silver Tail Systems - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Current thread:
- libpcap delivering the same packet more than once under high load? Jim Lloyd (Sep 10)
- <Possible follow-ups>
- Re: libpcap delivering the same packet more than once under high load? Alexander Dupuy (Sep 14)
- Re: libpcap delivering the same packet more than once under high load? Jim Lloyd (Sep 14)