Re: Network Storage

[Michael J McCafferty <mike () m5computersecurity com>]

more in-line...

On Thu, 2012-04-12 at 17:16 -0400, Maverick wrote:
> Thank you very much for your suggestions.
>
> 1) My goal is to store the traffic may be fore ever, and analyze it in
> the future for security related incidents detected by ids/ips.

The poor man's way to do this is to use the space you have and use the
-C and -W options in tcpdump. You have as much history as you have disk
space. Maybe make 500M files, and a count of 1800 to use 900G of disk
space. When you have an event, you copy off the files that are relevant
to the time period of the events, to a workstation. Another option is -G
for rotating the files by time instead of size.

> 2) I am storing just header and initial few bytes but still it gets
> filled up quite quickly.

You can use the -z option to gzip compress the files to save space.
However, I don't know how this will affect your disk io... will it be
fast enough to keep up with the writing of the raw data and doing a
concurrent gzip of the last file. If you have enough hardware
performance, but are limited on space, then it's worth a shot.

> 3) Netflow approach is nice but I also want to have traces available
> for reasons mentioned in 1).
>
> 4) Are there any issues having an external storage as a solution for
> this problem.

There is also some advice in the man page for tcpdump regarding the -z
option. You can write a shell script that takes the capture file as the
only argument, to do other stuff you want done... in this case, copy the
file off to another drive. It could be a network location too... of
course, don't forget to not capture *that* traffic (feedback!).

> Best,
> Ali
>
> On Thu, Apr 12, 2012 at 5:06 PM, Michael J McCafferty
> <mike () m5computersecurity com> wrote:
> > Ali,
> >        Do you need to capture the whole packet, including the payload? You
> > will save a lot of space by just capturing the headers. For example,
> > tcpdump doesn't capture the whole packet by default anyway. You may not
> > be able to capture at line rate anyway depending on what you are using
> > to capture with (drivers, libraries, software, etc). See the -s option
> > in tcpdump man page for info.
> >
> > Good luck,
> > Mike
> >
> > On Thu, 2012-04-12 at 16:25 -0400, Maverick wrote:
> > > Hello Everyone,
> > >
> > > Can you please comment on what is best solution for storing network
> > > traffic. We have been graciously granted access by our network
> > > administrator to capture traffic but the one Tera byte disk space is
> > > no match with the data that we are seeing, so it fills up quickly. We
> > > can't get additional space on the server itself so I am looking for
> > > some external solutions. Can you please suggest something that would
> > > be best for Gbps speeds .
> > >
> > >
> > > Best,
> > > Ali
> >
> > --
> > ************************************************************
> > Michael J. McCafferty
> > CEO
> > M5 Hosting
> > http://www.m5hosting.com
> >
> > Like us on Facebook for updates and photos:
> > https://www.facebook.com/m5hosting
> > ************************************************************

-- 
************************************************************
Michael J. McCafferty
CEO
M5 Hosting
http://www.m5hosting.com

Like us on Facebook for updates and photos:
https://www.facebook.com/m5hosting
************************************************************