Re: bpf/pcap performance

[Guy Harris <guy () alum mit edu>]

On Apr 13, 2004, at 3:18 PM, Darren Reed wrote:

> Ok, now that I read it, I find it tries to do too much (in my
> opinion.) The part that makes it go off and read more data is
> of questionable value ?

I.e., the fact that it - like "pcap_next()" - will fill the packet 
buffer if it's empty?

> As for non-blocking more...
>
> How exactly is pcap_next_ex() meant to work in non-blocking mode ?

In a fashion similar to the way "pcap_next()" does - if no packets are 
available, it should return 0, so the documentation for a return value 
of 0 should indicate that.

> Looking more closely at the non-blocking side of things, there seems
> to be no use of either p->nonblock or p->timeout apart from in
> pcap_setnonblock() ?

"p->nonblock" is there for the benefit of, among other things, 
"pcap-linux.c" if, as, and when it implements mmaped-capture support - 
"reading" isn't done with a "read()" or "recvfrom()" in that case, so 
it needs to know whether, if the mmapped buffer is empty, it should do 
a "select()" to wait for a packet or should just return "no packets".

I need to look at the "p->timeout" issue.

> Maybe pcap_next_ex() should also return -2
> if "p->cc == 0" prior to calling p->read_op if nonblock is true ?

No, -2 is an "end-of-file, don't bother calling 'pcap_next_ex()' ever 
again, there will never be another packet" return.  0 means "no packets 
now, but try again later".

> Another thought that came to me on "how it might work", going back
> to the quick API that I wrote up, if I could open a pcap file and
> mmap it in,

...which you can do only if you have enough address space; what would 
be best here would be a way in which a window could be mapped in, so 
that really large capture files (which people *do* have and *do* 
process) can still be handled.

> I think the current design (pcap_dispatch(), etc) is what's led
> to strange API hooks like pcap_breakloop() appearing, when I think
> a better design could have side stepped the need for them O:-)

Well, "pcap_breakloop()" also works around the workaround :-) for the 
"don't choke when we get ptraced" problem, i.e. that the implementation 
on some platforms treats EINTR not as an indication that it should quit 
but as an indication that it should try reading again.  I don't know 
whether an EINTR gets delivered if you're debugging a libpcap-based 
application, but, if that *is* a problem and that workaround *is* 
necessary, some scheme is needed to let a signal handler cause the read 
loop to terminate.

> There's pcap.h with pcap_pkthdr and pcap-int.h with pcap_sf_pkthdr.
> At present, applications are meant to include pcap.h and use 
> pcap_pkthdr.
> What should happen is that internally, pcap_int_pkthdr

Presumably meaning "pcap_sf_pkthdr", or possibly a renamed 
"pcap_sf_pkthdr"?

> should be used by libpcap

I.e., used in capture files?

> and pcap_pkthdr should be used in pcap_dump() and when reading pcap 
> dump files.

I.e., used in APIs?

That's what we do now - we have "pcap_pkthdr", used in API's, and 
"pcap_sf_pkthdr", used in capture files.

> But that would only make sense if the pcap file format was actually
> considered to be a public interface.  Is it?

I call it quasi-public - you can write code that reads and writes it 
(in fact, I have - the code in Ethereal), but

	1) you shouldn't do so merely out of ignorance of the existence of 
libpcap (and Net::Pcap and the like)

and

	2) you should be prepared for us to come up with a new capture file 
format which your code won't be able to read but that the next version 
of libpcap *will* be able to read (without the applications having to 
change, as long as they ignore all the new capabilities of the new file 
format).

> If not, I'd really like
> you to consider making it a public interface.- which I think is the
> other, large, ongoing email conversation here is about.

Once the new format is defined and implemented, we should have a 
"pcap(5)" (or "pcap(4)", or whatever - that's one problem with that man 
page, different flavors of UN*X have different conventions for which 
section is to have file formats :-( ) man page that documents the new 
format; it can also document the current format.

> It just dawned on me that you might want a universal file format that
> uses particular sizes for things and have pcap_pkthdr use what ever
> happens to be the native size and that this fits the current design
> and implementation.

Yes, that's what I think is the right answer.

> So we're stuck having to build an API that has to be able to work on
> top of a broken kernel interface implementation.  Fun.  NOT.

The only way *not* to ever be so stuck is to abandon libpcap support 
for anything other than Linux and BSD (and *maybe* Windows, if none of 
the underlying limitations get in the way of the WinPcap drivers), and 
even that might require supporting only shiny new versions or supplying 
kernel patches.  We don't control all the mechanisms atop which libpcap 
is implemented.

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.