Re: Libpcap And snap_len

[Guy Harris <guy () alum mit edu>]

On Dec 11, 2010, at 4:35 AM, Vikram Roopchand wrote:

>                  We have been using Libpcap 1.1.1 heavily

On what operating system are you using it?  The code used for capturing is very different on different OSes.

> and noticed
> something of the following nature. It seems that during a packet read,
> internal buffer to read each segment is allocated a size of "snap_len"
> bytes.

The buffer into which packets are read is allocated in pcap_activate(), not in the read code.  libpcap doesn't allocate 
a separate buffer for each packet.

> This is okay since we do not know how many bytes will arrive so it
> makes sense to have snap_len number of bytes as buffer to read into. But
> after that it looks that no realloc is done to reduce the buffer to
> the captured length (caplen) size. Is this correct ?

Yes.  There's no good reason to do so, as the next packet to be read into the buffer might be larger.

And this only applies on those platforms where

        1) only one packet is read into the buffer at a time

and

        2) the packets aren't read from a memory-mapped buffer shared between the kernel and userland.

If more than one packet is read into the buffer at a time - as is the case, for example, if your OS has BPF as the 
capture mechanism (*BSD, Mac OS X, AIX, some newer OpenSolaris builds), the buffer has to be as big as BPF's internal 
packet buffer, which can be, and has to be, much bigger than a single packet.

If it's a memory-mapped buffer, that buffer is also bigger than one packet.

> If this is the
> case, isn't some memory getting wasted each time a segment is caught.

For any given packet, not all the memory allocated for the buffer contains packet data, so, in that sense, it's 
"wasted".  It might be used for the next packet.

That "waste" is not cumulative, i.e. it's not as if you lose more and more memory with each packet that's captured.

> For
> example, in case of Ethernet, If the snap_len was kept to 65535, and the
> ethernet frame can go atmost to 9022 (jumbo),

Actually, the "frame" can be bigger; if the network adapter is doing TCP reassembly (TCP Segmentation Offloading, or 
TSO), it might hand a reassembled chunk of TCP data to the host, so what you're getting wouldn't be a single Ethernet 
frame.

> we lose on quite a bit of
> internal buffer each time a segment is caught.

Again, it's not as if memory is lost for each packet captured; there might be some memory that, during the entire 
capture, is never used, but that amount of memory is allocated at the start of the capture, not allocated for every 
packet captured.

> I understand there must be
> some performance drop doing realloc, may be that is the reason ?

If we did a realloc() to shrink the buffer after the packet's been read, we would have to do *another* realloc() to 
grow it again before reading the next packet; that would increase the CPU time required to capture, *and* would 
potentially fragment memory significantly and waste a lot more memory.-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.