Re: libpcap delivering the same packet more than once under high load?

[Jim Lloyd <jlloyd () silvertailsystems com>]

On Tue, Sep 14, 2010 at 5:48 AM, Alexander Dupuy <alex.dupuy () mac com> wrote:

> Jim Lloyd writes:
>
> > These duplicate packets cannot be unique packets that were retransmitted
> > between the two machines on the layer 1 GigE link, because if there was a
> > significant increase in retransmission duplicates on the link, they would
> > have competed for the fixed 1Gbps channel capacity. But the data is
> > consistent with libpcap delivering the same packets more than once.
> How are you capturing these packets (i.e. where is your libpcap application
> running, and what is it monitoring)?

For this testing I am running the application on the same machine that is
running apache httpd.


> If you are using a switch mirror or SPAN port to capture the traffic, there
> are many configurations that will capture packets multiple times, e.g. both
> on receive from one port and transmit on another.


Yes, we've see that in production environments. One of our customers has
their SPAN configured so that we see duplicates of every packet. But that
isn't the case with this test run in our lab.


> If your traffic is crossing multiple (V)LANs which are being monitored, you
> can also see packets that are not exact duplicates (different MAC addresses,
> IP TTL/hop count & checksum) but which are effectively duplicates at the TCP
> layer.

The client test generator application (a custom app written using libcurl)
runs on a different machine, but the two machines are attached to the same
switch. No VLANs are involved.

> If your libpcap application is running on the sender or receiver, and
> monitoring the host's interface that is used for the test traffic, you won't
> have this problem (although depending on your hardware you may see
> transmitted traffic that is not what actually goes out on the wire - e.g.
> TCP segmentation offload will split/fragment packets but libpcap will not
> see that).

This all makes sense, and shouldn't apply in the given test.

> Unless you are using an optical tap or port replicator or something like
> that to perform the traffic capture, it is generally the case that what you
> see is not 100% identical to what is actually going across the wire.

> Some other non-libpcap-related reasons why what you see is not exactly what
> was on the wire are in the kernel - on some multiprocessor systems (Linux at
> least, perhaps this is also true for *BSD) you may get packets in a
> different order than the one in which they were actually received (you can
> even see this in the libpcap packet receive timestamps).

I've been assuming that some reordering of packets may occur even in this
idealized lab setup with two machines connected to the same switch. Our app
handles reordering fine, and will not count reordered packets as
duplicates.


> All of these are not the fault of libpcap - although it is possible to add
> functionality to libpcap (or your application) to eliminate duplicate
> packets and/or reorder packets based on receive timestamps - we have done
> the former in our applications.

Yes, the application is written to expect duplicates and eliminate them when
they occur. I'm not surprised to see some duplicates. I even expect to see a
higher percentage of duplicates when I saturate a link. I just don't expect
to see total traffic close to 3x the physical capacity of the link.

I am of course still open to the possibility that the cause of the
unexpected behavior is something outside of libpcap, but I am at a loss for
a good explanation. Thanks for your ideas. Please let me know if you think
of others.

Jim
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.