Re: Proposed new pcap format

[Jefferson Ogata <Jefferson.Ogata () noaa gov>]

Stephen Donnelly wrote:
> When capturing network data at hundreds of megabytes per second for 
> extended periods and hence dealing with hundreds of gigabytes of 
> captured data at a time, even 33% overhead is very expensive in storage 
> space and disk bandwidth, as well as the cpu time required to perform 
> XML output with base-64 encoding.
>
> This is why my interest in the new format is to encourage keeping the 
> fixed overhead per packet record small. This can be done by a) keeping 
> per-packet meta data optional where possible, and b) keeping space 
> efficiency in mind when encoding per packet (meta)data.

It can also be done by using your own personal file format for your 
actually quite rare application of long time-period, high-bandwidth 
capturing. I don't see why the general format would be tailored to this 
application. Most people aren't interested in saving off a whole OC48 
for any period of time. The more usual use is to try to identify 
problems with specific protocols, or perform IDS functions. In these 
cases, people are more interested in saving selected packets and metadata.

Another thing you might do with your own high-bandwidth capture format 
would be to design it to facilitate merging streams from multiple 
capture sources, which you might split up using a toplayer or similar 
box. Again, this is not an application for the general tcpdump user pop.

> It may well be true that for analysis XML is useful either internally 
> for processing, or for results, but libpcap is primarily about packet 
> capture.

I disagree, at least in the sense in which you appear to mean this. 
libpcap includes BPF and the pcap expression compiler, which are about 
packet filtering. And remember that this is the tcpdump-workers list.

--
Jefferson Ogata <Jefferson.Ogata () noaa gov>
NOAA Computer Incident Response Team (N-CIRT) <ncirt () noaa gov>

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.