Re: Proposed new pcap format

[Stephen Donnelly <stephen () endace com>]

Jefferson Ogata wrote:
> > > 50% extra space and 50% extra disk bandwidth cost? So my 250
> > > Megabyte per second
> > > pcap stream to disk becomes 375MB/s?

...

> Raw packet data would typically be base64-encoded. This expands data by 
> 33%; three octets become four. You don't have to write one octet as two.
>
> In any case, if you're trying to capture every packet off the wire, you 
> might not want to use the newer binary pcap format under discussion 
> either. It's looking to impose some not insignificant overhead as well.

When capturing network data at hundreds of megabytes per second for extended 
periods and hence dealing with hundreds of gigabytes of captured data at a time, 
even 33% overhead is very expensive in storage space and disk bandwidth, as well 
as the cpu time required to perform XML output with base-64 encoding.

This is why my interest in the new format is to encourage keeping the fixed 
overhead per packet record small. This can be done by a) keeping per-packet meta 
data optional where possible, and b) keeping space efficiency in mind when 
encoding per packet (meta)data.

It may well be true that for analysis XML is useful either internally for 
processing, or for results, but libpcap is primarily about packet capture.

Stephen.
--
-----------------------------------------------------------------------
    Stephen Donnelly BCMS PhD           email: sfd () endace com
    Endace Technology Ltd               phone: +64 7 839 0540
    Hamilton, New Zealand               cell:  +64 21 1104378
-----------------------------------------------------------------------

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.