Re: questions on -B, performance, mbufs, and libpcap buffers

[Guy Harris <guy () alum mit edu>]

On Sep 13, 2011, at 8:47 PM, Jon Schipp wrote:

> I have a few questions, hopefully someone can set me straight.
> Info: *I'm on FreeBSD 8.2* *out of a couple million packets, in a few hours
> time, I drop around 4000*
> First off are all packets stored in mbufs?

With most network drivers, I think packets are stored in the driver's ring buffer, and those are "loaned" to mbufs.  Or 
maybe I'm remembering SunOS 4.x, and FreeBSD does things differently.

> And if so, is it possible to increase the amount of mbuf clusters to improve
> tcpdump performance?

That's probably not the problem.  The mbuf or mbuf chain containing an input packet is handed to BPF if you're 
capturing traffic with libpcap (which tcpdump does), and they're copied to BPF's own buffer, which is not made of mbufs.

> Again, questioning myself, I wasn't sure if packets for tcpdump were stored
> in mbufs, I would expect them to run out if the kernel was dropping them.
> Though, I'm assuming that all packets are stored in mbufs and that bpf just
> gets a copy of what's in the mbufs.

They are, but the buffer into which the copy is made is not mbufs, it's set on a BPF device with a BIOCSBLEN ioctl.  
("man bpf".)

> Second, does -B set an application buffer independent of the libpcap buffer?

No, it tells libpcap how big the buffer size should be; in BPF (*BSD and Mac OS X), that's the kernel buffer size as 
set by BIOCSBLEN.

> What's the default size?

In the current top-of-tree libpcap, 524288 bytes.

In FreeBSD 8.2, 32768 bytes.

> Can capture improvements be made by increasing this value?

Probably.  32768 may have made sense back when a machine with 32 megabytes was a Big Machine With Tons Of Memory and 
100 MB/sec Ethernet was a Really Fast Network, but the laptop on which I'm typing this has 4GB of memory (when I bought 
it, that was twice the base memory size; it now *is* the base memory size) and its Ethernet adapter is a 10/100/1000 
interface.  (It was boosted in response to a gripe from a networking developer in the OS group for the machine in 
question, because those of us in the remote file system group wanted tcpdump to default to 65535 as the snapshot 
length, which caused packet drops with the old default buffer size.  I took one look at the default buffer size and 
said "hello, 1998 called, it wants its default buffer size back" - although maybe I'm being unfair to 1998.)

> *I'm dropping packets at the kernel, according to tcpdump*
> Is the argument for -B in Kilobits?

No, it's in bytes.

> I see that the libpcap buffer can be increased with sysctl since libpcap
> initializes its buffer amount with bpf sizes:
> <majordomo () lists tcpdump 
> org?subject=subscribe%20to%20tcpdump-workers&body=subscribe%20tcpdump-workers>net.bpf.maxbufsize:

That's the *maximum* buffer size; you can try to set a bigger buffer with "-B" (or with pcap_set_buffer_size() in 
libpcap, which tcpdump calls to implement "-B", or with BIOCSBLEN, which libpcap does with the buffer size from 
pcap_set_buffer_size()), but the kernel will silently truncate that size at net.bpf.maxbufsize.

> net.bpf.bufsize:

That's the default, which you *can* set bigger, which will set the buffer size if you don't specify "-B", but it'll 
also set the buffer size used by any application or daemon that doesn't explicitly set it.  I don't know what else used 
BPF on FreeBSD, but in Mac OS X, the DHCP client uses it, and when I set it (and the maximum size) to a very large 
value (on a machine older and smaller than the machine on which I'm typing this) that meant I ran out of memory when 
running tcpdump, so the DHCP client couldn't re-acquire the machine's configuration when it woke up, leaving it unable 
to access the network.  So if you crank net.bpf.bufsize up, be careful....

> I plan on doing tests with these soon. I'm trying to collect as much
> information as possible first.

I'd consider using "-B 524288" rather than using sysctl to set either net.buf.maxbufsize or net.bpf.bufsize.  *If* 
524288 isn't big enough, crank net.bpf.maxbufsize up and pick a bigger value to pass to "-B".

> Third, when tcpdump reports how many packets are dropped by the kernel is
> this the same value as the packets dropped by libpcap?

It's "ps_drop" from the result of pcap_stats(), which is "bs_drop" from BIOCGSTATS on systems using BPF, which is the 
number of packets that the BPF code in the kernel dropped because there wasn't enough space in the BPF buffer.

> In otherwords, does libpcap ask the kernel for the kernel drop amount

Yes.

> and report it as "packets dropped by kernel".

It reports it as "ps_drop", which tcpdump reports as "packets dropped by kernel".

> Or will they have different
> values?
> The reason I ask is that the program ntop has a field that shows "dropped by
> (libpcap)" but not a "dropped by kernel"

Hmm.  I'm looking at the ntop 4.1.0 source, and can't find "dropped by (libpcap)" in a message.  There's "STATS: {N} 
packets dropped (according to libpcap)", which is reporting ps_drop from pcap_stats(), and there's "Dropped (libpcap)" 
and "Dropped (pcap)" in some HTML-generating code, which is also reporting ps_drop from pcap_stats().

So ntop is saying "libpcap" where tcpdump is saying "kernel", but they're reporting the same value.

> If they're different, is there a way to find out the root cause: kernel or
> libpcap

libpcap doesn't itself have any buffer that can overflow - it's all kernel.

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.