Re: parsing very large tcpdump files

[Ron Gula <rgula () tenablesecurity com>]

At 06:29 PM 11/18/2004 -0500, Tom wrote:

> moderator: sorry if this is vague.  My requirements are not fixed yet and 
> will probably change from case to case, therefore I am just looking for 
> generic info now.
>
> I was wondering if anyone on this list can recommend some tools 
> (Opensource or commercial) to automate the parsing of very large (many GB) 
> tcpdump files.  I am trying to put together a generic toolset but in 
> general some things I'd like to do are:
>
> 1. Filter out traffic to/from a specific IP address or range
> 2. Reconstruct all reconstructable sessions in an easy to parse way: 
> emails, web sites visited (and content uploaded/downloaded), voip, 
> anything else imaginable.
> 3. Be able to search all of this data for keywords.
>
> This may seem like a tall order.  I know of a few tools to do individual 
> tasks on a small scale, such as mailsnarf, vomit, ethereal,  etc. but it's 
> not practical to use ethereal to parse these by hand. I've tried 
> chaosreader.pl but it bogs down on files as small as 200 MB.
>
> I'd appreciate any input.
>
> Thanks.

Hey there,

For an excellent review of various ways to slice and dice TCPDUMP
files with opensource tools, I would highly recommend the book,
"Tao of Network Security Monitoring" which you should be able to
order from Amazon or get it from Borders. It has really good
sections on the regular opensource stuff as well as really good
tools that are not that common.

On the commercial side, our NeVO product can take in TCPDUMP
files, give you a list of unique hosts, open/browsed ports,
which hosts communicate with which other hosts, any detected
client/server apps, any vulnerabilities contained within them
and any successful intrusions observed. The resulting file is
compatible with Nessus, so you can load it into any tool that
parses Nessus data and do your analysis. NeVO does this sort
of stuff on live feeds as well.

Other commercial tools I would recommend, but don't know if
they can take a raw TCPDUMP feed are Niksun (http://www.niksun.com/)
and NetIntercept (http://www.sandstorm.com/).

Ron Gula, CTO
Tenable Network Security
http://www.tenablesecurity.com





















--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------