Pcap next-gen file format

["Mark Bednarczyk" <voytechs () yahoo com>]

I haven't kept up on the fact that there is a new file format in the works.
I find it very interesting that my own efforts for a 'NAP' file format for
my own project (http://jnetstream.sf.net) nearly mirror pcap design while I
wasn't even aware that pcap team was working on a format as well. So this
was a completely oblivious design to pcap-ng format, which makes it all that
much more interesting, atleast for me.
 
I've spent significant amount of time designing NAP file format last year
and lots of experimentation. Some key points I can draw from comparing the 2
similar designs.
 
(Note that NAP terminology is different section==block, block==record etc,
but I will use pcap terminology to avoid confusion.)
 
1) pcap uses variable length headers - that is something I avoided in NAP.
In NAP all headers are 24-bytes long. No exceptions.
 
2) both formats use sections - NAP sections are by default always 512Kb in
length, padded if needed and a new section is created every 512kb. They
don't have to be this length (can be made upto 4gig), but unless there is a
compelling reason, they should be generated at this standard length. This
not only allows unix "cat" be used to combine multiple capture files, just
like in pcap-ng, but also allows "split" be used to split files into smaller
sections. Keeping sections at constant and not too long a length is also
very!!! very!!! important for indexing packet records.
 
3) pcap-ng stores the length of the block at the beginning and the end - nap
only at beginning and not the end, as sections are of constant and
relatively small size and any record can easily be accessed by first
aligning on the enclosing section and then iterating over from the start of
the section until the record is found. 
 
4) Also uses interface descriptor blocks, although in NAP is a little more
general and called CaptureDevice since its not assumed that the source of
the data is always an interface, (i.e. file, memory generated, etc..) but in
general the same type description information is stored.
 
5) NAP includes a vendor block. The vendor block is vendor specific and does
not have to be interpreted by implementations not understanding the vendor
record. This is similar to pcap-ng definition of a block in general, except
vendor block is a specific "record type" in NAP file format. It contains
vendor_id field after which everything else is vendor specific.
 
6) NAP file format stores protocol-binding overrides. This was a key goal of
the format which is to allow overriding the default bindings for protocols.
It is often found that port numbers and application don't allows use the
same mappings. NAP allows the user to define those mappings once and they
are stored within the capture file. Along the same lines, a more project
specific goal was to allow the capture file to store protocol definitions
themselves be stored along with the captured data. This way propriatory
protocols, not in the mainstream can be included within the capture file and
be fully decoded on a system/application normally not aware of those
protocols within packets contained in the capture file. This is possible
becase of NPL which is a text based protocol definition language that can be
compiled on the fly by jnetstream.
 
I could go on and on with the comparison, but I consider these all very
small differences. There are certainly a lot more similarities then
differences between the 2 formats.
 
I wrote the NAP definition using an internet-draft format. You can view the
complete definition here:
 
http://jnetstream.svn.sourceforge.net/viewvc/*checkout*/jnetstream/jNetStrea
mAPI/trunk/docs/draft-slytechs-network-nap-00.doc
 
And html version but "viewvc" is substituting some wierd characters and
screwing up the formatting a little bit.
 
http://jnetstream.svn.sourceforge.net/viewvc/*checkout*/jnetstream/jNetStrea
mAPI/trunk/docs/draft-slytechs-network-nap-00.htm
 
May be you guys can find something usefull in my format.
 
And for my benefit. There were a number of ideas I wasn't sure if they were
practical. But now that I see that pcap-ng also uses those same concepts,
they are validated for me as a the right decision. So all I can say is wow
and thank you.
 
Cheers,
mark...
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.