Re: advice for heavy traffic capturing

[Darren Reed <darrenr () reed wattle id au>]

Hi Fulvio,

> Fulvio Risso, Loris Degioanni, An Architecture for High Performance Network
> Analysis, Proceedings of the 6th IEEE Symposium on Computers and
> Communications (ISCC 2001), pg. 686-693, Hammamet, Tunisia, July 2001.

Is there any way you can get this (and the other date info.) into those
PDFs ?  It really helps put them in perspective.

> > Winpcap appears, by design, to be the same as BPF.  If you reduced the
> > number of buffers in the ring used with NPF to 2 buffers, I suspect it
> > would be the same as BPF ?
>
> No, there are two different architectural choices.
> The ring does not have buffers; it has just space for packets; space
> occupancy is exactly the size of the packet.

Ah, so you're using the buffers that the data is read into, off the NIC,
to put into the ring ?  Or to put in BSD terms, the ring is made up of
mbuf pointers ?

You would have to be careful to not hold on to the buffers for too long
(or too many of them) or else surely you would run out ?

That would make direct access to the buffers from user space (using mmap
or similar) more involved.

> > Interestingly, there are a few large areas for improvement: timestamp
> > (1800 -> 270), Tap processing (830->560) and filtering (585 -> 109).
>
> ... and NIC drivers and Operating system overhead which, as you can see,
> account for more or less 50% of the total overhead.

Yup.

> > The Intel 100 ProS have 128K for receieve, as I recall, the same as
> > the 1000MX card.  There wasn't much between these two, that I was able
> > to observe, except that the 100ProS was slightly better.
>
> The amount of memory you have on the NIC is not very significant.
> I cannot give you numbers right now, but this is not the parameter that
> changes your life.

Why not ?  Well I suppose your results (if the 3com really does only have
16 or 32k of buffer) would support this.

But maybe buffering is more important for BPF where you have the interrupt
masked out for longer while the data is copied ?

> =========================================
> A valuable result of this study is the quantitative conclusion that,
> contrary to common belief, filtering and buffering are not the most critical
> factors in determining packet capture performance. Optimization of these two
> components, that received most attention so far, is shown to bring little
> improvement to the overall packet capture cost, particularly in case of
> short packets (or when small snapshot length are needed). The profiling done
> on a real system shows that the most important bottlenecks lie in hidden
> places, like device driver, interaction between application and OS,
> interaction between OS and hardware.
> =========================================

Hmmm, the testing I did would disagree with that, or at least so far as
to say that there is a "sweet spot" for buffer sizes and data rates (at
least with BPF.)  The hardware does make some difference - one of our
other test cards was a Netgear (FA-311?) and it was shit.

My recollection was that with the data sample we were using, with 1MB
captures enabled for BPF, at full speed, most reads were between 64k
and 256k, at a time.

There were other changes to BPF, unrelated to what you've changed, that
reduced packet loss from X% to 0%.  I copied these, this year, from
FreeBSD to NetBSD but I don't recall their vintage on FreeBSD.

> And, I would like to say, you need a global picture of where the bottleneck
> are before doing optimizations.

Oh, sure.  And one of those limiting factors is PCI.

> For instance, we're now working to decrease the 50% of the time spend by
> each packet in the operating systems.

You're still working with Windows ?

> > In the NetBSD emails, I think I ponder making changes to the buffering
> > so that it is more ring-buffer like (similar to what exists within NPF
> > if I understand the diagram right.)
>
> Eh, what you're saying is good but... the double buffering in the BPF has an
> advantage: it is much simpler, and if you're not interested in memory
> occupancy, it is a very good choice.

Yes.

> We didn't realize it in 2001; now, we can see less black and white in the
> choice between a double buffer and a ring buffer...

What have you found that makes you say this ?
The simplicity in cpu cycle cost ?

> > Is the JIT code easily ported to other platforms ?
>
> Yes, as far as the platform is Intel ;-)

That's fine with me :)
Do you have a URL for this ?

Darren
-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.