Re: Proposed new pcap format

[Jefferson Ogata <Jefferson.Ogata () noaa gov>]

Fulvio Risso wrote:
> > [mailto:tcpdump-workers-owner () lists sandelman ca]On Behalf Of Stephen
> > Donnelly
> >
> > Jefferson Ogata wrote:
> > > Yes, fully fledged decoded captures would use a lot of extra
> > disk, but a
> > > raw no-frills capture could be recorded with maybe only 50% or
> > so overhead.
> >
> > 50% extra space and 50% extra disk bandwidth cost? So my 250
> > Megabyte per second
> > pcap stream to disk becomes 375MB/s?
>
> No, more than 500 MB/s.
> You have to trasform everything in ascii, so an 8bit value becomes a 2 bytes
> ascii value.

As I imagine you know, XML is not ASCII; it's Unicode.

Raw packet data would typically be base64-encoded. This expands data by 33%; 
three octets become four. You don't have to write one octet as two.

In any case, if you're trying to capture every packet off the wire, you might 
not want to use the newer binary pcap format under discussion either. It's 
looking to impose some not insignificant overhead as well.

Again, pay attention to the discussion; there are many optional features being 
suggested for the pcap storage format. What prompted my remark was the 
discussion about which hash algorithms to include in the storage format, what 
data gets hashed, and whether any particular algorithm is designated as a 
default. That's the kind of stuff that says, to me, that a binary file format is 
going to grow out of itself pretty fast.

--
Jefferson Ogata <Jefferson.Ogata () noaa gov>
NOAA Computer Incident Response Team (N-CIRT) <ncirt () noaa gov>

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.