Re: Multiple interface capture and thread safety status in libpcap

[Guy Harris <guy () alum mit edu>]

On May 10, 2012, at 7:43 AM, Wiener Schnitzel wrote:

> I need to perform packet sniffing on several interfaces at the same time.

Are you processing packets from each interface independently, so that a packet on interface A is not looked at when 
processing packets from interface B, or are you processing the packets from all of the interfaces as a single stream, 
so that you need to see packets from multiple interfaces in order?

> My natural approach would be to open a pcap_t object for each interface and place a "select" - considering Linux 
> -call to deal with packet dispatching. My only constraint is that I have to treat the received packets in 
> chronological order: indeed, I would like to process the data as it gets to the interfaces, without introducing any 
> reordering.

Even if you're sniffing on *one* interface, I think that, with at least some versions of the Linux kernel, "as it gets 
to the interfaces" and "as it gets delivered to the PF_PACKET socket from which libpcap reads" are not necessarily the 
same thing on multi-core machines.  I seem to remember that some people have seen packets with out-of-order timestamps, 
and have the impression that the problem is that if two packets are processed on different cores, the packet that 
arrived second might be queued up on the socket before the packet that arrived first if, for whatever reason, the 
thread on the second core manages to get its job done faster.  I don't know whether this is still a problem with 
reasonably recent versions of the kernel.

> If I am not mistaken, it might be possible that a "select" call does not read data in temporal order, if multiple FDs 
> are ready at the time the process is scheduled for running by the OS. Is that correct ?

Well, to be technical, the select() call doesn't read data, it's the calls to pcap_dispatch() made as a result of 
select() saying various FDs are ready, but, yes - if pcap_dispatch() processes more than one packet, you'll be 
processing the currently-available packets from the the first interface you find when scanning select()s results, 
followed by the currently-available packets from the second interface, and so on, even if the last packet from an 
earlier interface has a later time stamp than the first packet from a later interface.

Now, if you put all the pcap_t's into non-blocking mode, and pass a count of 1 to pcap_dispatch(), so it processes only 
one packet, or if you use pcap_next() or pcap_next_ex(), you could try reading from each of the interfaces, process the 
packet with the lowest time stamp, and, in the next trip through the loop, read another packet from the interface from 
which the packet you processed came and re-check the packets read previously from the other interfaces, you'd process 
the packets in time stamp order (modulo any out-of-order delivery from the kernel on any single interface).

> A work-around to this problem might be to move the capture on different threads: each thread has its own pthread_t 
> object and captures traffic on a different interface.

If each thread is processing packets independently, so that you don't have to worry about processing packets from 
multiple interfaces in the right order for all of those interfaces, then you could do it in one thread - for each call 
to pcap_dispatch(), do the processing for packets from the interface in question.  Doing it in multiple threads would 
make better use of multiple cores in your application, however.

If that's *not* the case, doing the capture in different threads still requires some scheme to process packets from 
different interfaces in order.

> In this case, I do not have a clear picture about which parts of libpcap are thread-safe and which not (my version of 
> reference is the 1.1.1); I have found really old posts about thread-safety issues in pcap_compile and pcap_setfilter 
> (which I would need: 1 common filter for each thread) but nothing more.

pcap_compile() uses YACC and Lex, or uses replacements thereof in YACC-compatible/Lex-compatible mode, so they're *not* 
thread safe - the lexical analyzer and parser have non-thread-safe state.

pcap_setfilter(), however, should be thread-safe, and the rest of the APIs are thread-safe as long as any given pcap_t 
is only being processed in one thread at a time; there's no locking to allow a given pcap_t to be processed in more 
than one thread simultaneously.-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.