Re: Data archiving

[Erek Adams <erek () snort org>]

On Wed, 19 Mar 2003, Sammy wrote:

> I'd like to get an idea of what people are doing regarding archiving
> their snort data.  How long do you keep data online and then what are
> you doing with it?  Are people archiving to tape w/ encryption?  Also,
> if anyone is using Snort for capturing all packet traffic, how do you
> deal with the tremendous amount of traffic generated?  I'd like to set
> up my system so that I can go back as far as possible to look through
> both alerts as well as all packet data but I'm finding it really hard to
> deal with the large amounts of data (one of my switches has about
> 20GB/hour running through it).  In the event of a break-in, since I
> wouldn't know how long the system has been compromised, I need to go
> back as far as possible.  Any advise/assistance is greatly appreciated!

Standard answer:  It depends on your network and needs.  :)

Real answer:  Data archiving has been an issue for years.  Ever since
people started doing serious backups, there's been a real need for some
_good_ methods to this maddness.

What I've done in the past for data archiving:  On-line, Near-line, and
off-site backups.  The length of retention depends upon policy.

        On-line:  Files are located on a central server with large amounts
of disk--Basically, the DB server.  As things are needed, they can simply
be pulled right of of disk, making it quick and easy to recover data.
        Near-Line:  Files are moved from the main server, back to a large
data storgage box.  Throw disks at this machine...  As many as you can
cram into it.  As data is needed, it can be loaded over the net back to
the online server for pushing/reading/restoring to wherever.
        Offline:  Magneto Optical, DLT-4, DVD-R, CD-R, whatever.  Save and
store offsite, but with clear labeling so you can retrieve what you need.

Since your're working with packets, you might consider only saving header
info on the offline storage.  That would greatly reduce the amount of data
that you have to deal with.  This would imply that you need to have a good
retention on the near-line backups so you can pull the data as/when you
need.  Once a specified time has passed, (6 months, a year, whatever) the
packet payload can be ditched, saving time and space.

Anyway...  Those are just my thoughts....  Hope they are of some use.

Cheers!

-----
Erek Adams

   "When things get weird, the weird turn pro."   H.S. Thompson


-------------------------------------------------------
This SF.net email is sponsored by: Does your code think in ink? 
You could win a Tablet PC. Get a free Tablet PC hat just for playing. 
What are you waiting for?
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr5043en
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users