tcpdump mailing list archives
Pcap next-gen file format
From: "Mark Bednarczyk" <voytechs () yahoo com>
Date: Tue, 28 Aug 2007 11:36:18 -0400
I haven't kept up on the fact that there is a new file format in the works. I find it very interesting that my own efforts for a 'NAP' file format for my own project (http://jnetstream.sf.net) nearly mirror pcap design while I wasn't even aware that pcap team was working on a format as well. So this was a completely oblivious design to pcap-ng format, which makes it all that much more interesting, atleast for me. I've spent significant amount of time designing NAP file format last year and lots of experimentation. Some key points I can draw from comparing the 2 similar designs. (Note that NAP terminology is different section==block, block==record etc, but I will use pcap terminology to avoid confusion.) 1) pcap uses variable length headers - that is something I avoided in NAP. In NAP all headers are 24-bytes long. No exceptions. 2) both formats use sections - NAP sections are by default always 512Kb in length, padded if needed and a new section is created every 512kb. They don't have to be this length (can be made upto 4gig), but unless there is a compelling reason, they should be generated at this standard length. This not only allows unix "cat" be used to combine multiple capture files, just like in pcap-ng, but also allows "split" be used to split files into smaller sections. Keeping sections at constant and not too long a length is also very!!! very!!! important for indexing packet records. 3) pcap-ng stores the length of the block at the beginning and the end - nap only at beginning and not the end, as sections are of constant and relatively small size and any record can easily be accessed by first aligning on the enclosing section and then iterating over from the start of the section until the record is found. 4) Also uses interface descriptor blocks, although in NAP is a little more general and called CaptureDevice since its not assumed that the source of the data is always an interface, (i.e. file, memory generated, etc..) but in general the same type description information is stored. 5) NAP includes a vendor block. The vendor block is vendor specific and does not have to be interpreted by implementations not understanding the vendor record. This is similar to pcap-ng definition of a block in general, except vendor block is a specific "record type" in NAP file format. It contains vendor_id field after which everything else is vendor specific. 6) NAP file format stores protocol-binding overrides. This was a key goal of the format which is to allow overriding the default bindings for protocols. It is often found that port numbers and application don't allows use the same mappings. NAP allows the user to define those mappings once and they are stored within the capture file. Along the same lines, a more project specific goal was to allow the capture file to store protocol definitions themselves be stored along with the captured data. This way propriatory protocols, not in the mainstream can be included within the capture file and be fully decoded on a system/application normally not aware of those protocols within packets contained in the capture file. This is possible becase of NPL which is a text based protocol definition language that can be compiled on the fly by jnetstream. I could go on and on with the comparison, but I consider these all very small differences. There are certainly a lot more similarities then differences between the 2 formats. I wrote the NAP definition using an internet-draft format. You can view the complete definition here: http://jnetstream.svn.sourceforge.net/viewvc/*checkout*/jnetstream/jNetStrea mAPI/trunk/docs/draft-slytechs-network-nap-00.doc And html version but "viewvc" is substituting some wierd characters and screwing up the formatting a little bit. http://jnetstream.svn.sourceforge.net/viewvc/*checkout*/jnetstream/jNetStrea mAPI/trunk/docs/draft-slytechs-network-nap-00.htm May be you guys can find something usefull in my format. And for my benefit. There were a number of ideas I wasn't sure if they were practical. But now that I see that pcap-ng also uses those same concepts, they are validated for me as a the right decision. So all I can say is wow and thank you. Cheers, mark... - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Current thread:
- Pcap next-gen file format Mark Bednarczyk (Aug 28)