tcpdump mailing list archives
Re: Proposed new pcap format
From: Stephen Donnelly <stephen () endace com>
Date: Thu, 15 Apr 2004 10:03:00 +1200
Jefferson Ogata wrote:
50% extra space and 50% extra disk bandwidth cost? So my 250 Megabyte per second pcap stream to disk becomes 375MB/s?
...
Raw packet data would typically be base64-encoded. This expands data by 33%; three octets become four. You don't have to write one octet as two.In any case, if you're trying to capture every packet off the wire, you might not want to use the newer binary pcap format under discussion either. It's looking to impose some not insignificant overhead as well.
When capturing network data at hundreds of megabytes per second for extended periods and hence dealing with hundreds of gigabytes of captured data at a time, even 33% overhead is very expensive in storage space and disk bandwidth, as well as the cpu time required to perform XML output with base-64 encoding.
This is why my interest in the new format is to encourage keeping the fixed overhead per packet record small. This can be done by a) keeping per-packet meta data optional where possible, and b) keeping space efficiency in mind when encoding per packet (meta)data.
It may well be true that for analysis XML is useful either internally for processing, or for results, but libpcap is primarily about packet capture.
Stephen. -- ----------------------------------------------------------------------- Stephen Donnelly BCMS PhD email: sfd () endace com Endace Technology Ltd phone: +64 7 839 0540 Hamilton, New Zealand cell: +64 21 1104378 ----------------------------------------------------------------------- - This is the tcpdump-workers list. Visit https://lists.sandelman.ca/ to unsubscribe.
Current thread:
- Re: Proposed new pcap format, (continued)
- Re: Proposed new pcap format Guy Harris (Apr 13)
- Re: Proposed new pcap format Michael Richardson (Apr 16)
- Re: Proposed new pcap format Loris Degioanni (Apr 13)
- Re: Proposed new pcap format Darren Reed (Apr 13)
- Re: Proposed new pcap format Guy Harris (Apr 13)
- Re: Proposed new pcap format Jefferson Ogata (Apr 13)
- Re: Proposed new pcap format Jefferson Ogata (Apr 13)
- Re: Proposed new pcap format Stephen Donnelly (Apr 13)
- Re: Proposed new pcap format Fulvio Risso (Apr 14)
- Re: Proposed new pcap format Jefferson Ogata (Apr 14)
- Re: Proposed new pcap format Stephen Donnelly (Apr 14)
- Re: Proposed new pcap format Jefferson Ogata (Apr 14)
- Re: Proposed new pcap format Ronnie Sahlberg (Apr 14)
- Re: Proposed new pcap format Jefferson Ogata (Apr 14)
- Re: Proposed new pcap format Ronnie Sahlberg (Apr 14)
- Re: Proposed new pcap format Fulvio Risso (Apr 14)
- Re: Proposed new pcap format Stephen Donnelly (Apr 14)
- Re: Proposed new pcap format Christian Kreibich (Apr 13)
- Re: Proposed new pcap format Jefferson Ogata (Apr 14)
- Re: Proposed new pcap format Christian Kreibich (Apr 14)
- Re: Proposed new pcap format Hannes Gredler (Apr 14)