tcpdump mailing list archives
Re: Memory-mapped capture and thinking the packet's
From: Eloy Paris <peloy () chapus net>
Date: Sat, 26 Sep 2009 18:09:13 -0400
Hi Guy, On 07/10/2009 05:07 PM, Guy Harris wrote:
pcap_next() and pcap_next_ex() rely on the packet data pointer handed to their pcap_dispatch() callback still pointing to the same packet data after the callback returns. If the packet data is being read into a buffer with read()/getmsg()/recvfrom()/etc., that works. If, however, the packet data is being fetched from a memory-mapped buffer shared with the kernel, so that the pointer points into that buffer, it doesn't necessarily work - when the callback returns, libpcap might release that entry in the buffer, allowing the kernel to reuse it and put new packet data there. A problem that was probably the result of this was reported in a thread titled "Buffer overwrites with pcap_next_ex"; I attempted to fix it by, in the Linux memory-mapped capture implementation, not releasing the packet until the *next* packet was fetched. Unfortunately, *that* broke select()/poll(), as reported in a thread titled "select() regression in libpcap-devel?"; select() thinks there's an unconsumed packet, and always reports that reading a packet is available, so I had to back that fix out. The pcap_next() and pcap_next_ex() problem could be fixed by having those routines, when capturing in memory-mapped mode, use a callback that copies the packet to a separate buffer and returns a pointer to *that* buffer. This means you do one more copy, but if that bothers you, you need to stop using pcap_next()/pcap_next_ex() and just use pcap_dispatch()/pcap_loop() and live with having to deal with the callback architecture. Unfortunately, it wouldn't fix any applications that use pcap_dispatch()/pcap_loop() and *still* assume that the packet data won't be changed until the next packet is read. I don't know whether any such applications exist; given that there is no guarantee that the data for packet N is still available after packet N+1 is read (and, in fact, there's a good chance that it *isn't* available - a certainty when, for example, capturing on Linux in non-memory-mapped mode), I *suspect* few applications make that assumption.
My application Network Expect ran into a regression and after spending a couple of hours troubleshooting this I narrowed the problem down to packet data getting overwritten after returning from pcap_next() and before the next call to pcap_next(). I then found the thread titled "Buffer overwrites with pcap_next_ex", which you mention above and is the exact same problem I ran into, and then I recalled seeing a recent message from you on this topic (the message above).
So it seems like the only option I have to fix the regression is to convert the pcap_next() call to pcap_dispatch()/pcap_loop() semantics. I don't think that copying the packet to a safe place as soon as pcap_next() returns is good enough since, while the exposure window is narrower, there's still a chance to have packet data overwritten by the kernel.
This problem is really nasty, by the way, and I'm surprised I haven't seen more reports of this problem -- the regression in Network Expect ranged from weird, incorrect results to segfaults caused by starting to dissect a packet to later have the entire packet changed while in the middle of the dissection. Really hard to troubleshoot too.
Under these conditions pcap_next() and pcap_next_ex() are completely unusable. Perhaps we should document that pcap_next() and pcap_next_ex() should not be used when libpcap uses mmap()?
Along the same lines, is there a way to disable libpcap's use of mmap() at run-time even if it's available? That would be a better workaround than to re-write my application...
Cheers, Eloy Paris.- netexpect.org - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Current thread:
- Memory-mapped capture and thinking the packet's still there after a pcap_dispatch()/pcap_loop() callback returns Guy Harris (Jul 10)
- Re: Memory-mapped capture and thinking the packet's Eloy Paris (Sep 26)
- Re: Memory-mapped capture and thinking the packet's Guy Harris (Sep 26)
- Re: Memory-mapped capture and thinking the packet's Guy Harris (Sep 26)
- Re: Memory-mapped capture and thinking the packet's Eloy Paris (Sep 26)
- Re: Memory-mapped capture and thinking the packet's Guy Harris (Sep 26)
- Re: Memory-mapped capture and thinking the packet's Eloy Paris (Sep 26)