RE: parsing very large tcpdump files

["Bowes, Ronald (EST)" <RBowes () gov mb ca>]

If you know perl, you can probably put together a program in a matter of
minutes to suit your needs.  I needed a tool which could parse a proxy's log
dump (several gigabytes), and it could do it in a matter of sections,
writing any accesses from a specific ip or user to a file.  I don't have the
source handy, but my program looked something like:

----------------------------------------------
#!/usr/bin/perl

fopen(OUTFILE, "/tmp/test");
fopen(INFILE, "/var/log/proxy001.log");
foreach $line(<INFILE>)
{
        my ($ip, $username, $date, $time, $alltherest) = split(/, /, $line);
        if($ip eq "10.1.2.3" || $username eq "rbowes")
        {
                print OUTFILE $line;
        }
}
print "---=== COMPLETE ===---";
----------------------------------------------  

It's likely it wouldn't be very difficult to adapt something similar to that
to your needs.  If you need any perl pointers, let me know.



Ron Bowes
Information Protection Centre
Government Of Manitoba

-----Original Message-----
From: Tom [mailto:digitalevidence () excite com] 
Sent: Thursday, November 18, 2004 5:30 PM
To: focus-ids () securityfocus com
Subject: parsing very large tcpdump files


moderator: sorry if this is vague.  My requirements are not fixed yet and
will probably change from case to case, therefore I am just looking for
generic info now.

I was wondering if anyone on this list can recommend some tools (Opensource
or commercial) to automate the parsing of very large (many GB) tcpdump
files.  I am trying to put together a generic toolset but in general some
things I'd like to do are:

1. Filter out traffic to/from a specific IP address or range
2. Reconstruct all reconstructable sessions in an easy to parse way: emails,
web sites visited (and content uploaded/downloaded), voip, anything else
imaginable.
3. Be able to search all of this data for keywords. 

This may seem like a tall order.  I know of a few tools to do individual
tasks on a small scale, such as mailsnarf, vomit, ethereal,  etc. but it's
not practical to use ethereal to parse these by hand. I've tried
chaosreader.pl but it bogs down on files as small as 200 MB.

I'd appreciate any input.

Thanks.



_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------