nanog mailing list archives
Re: Network Storage
From: Leo Bicknell <bicknell () ufp org>
Date: Sun, 15 Apr 2012 06:38:28 -0700
In a message written on Thu, Apr 12, 2012 at 05:16:27PM -0400, Maverick wrote:
1) My goal is to store the traffic may be fore ever, and analyze it in the future for security related incidents detected by ids/ips.
Let's just assume you have enough disk space that you can write out every packet, or even just packet header. That's a hard problem, but you've received plenty of suggestions on how to go down that path. Once you have that data, how are you going to process it? Yes, disk reads are faster than disk writes, but not by that much. If it takes you 24 hours to write a day of data to disk, it might take you 12 hours just to read it all back off and process it. Processing a weeks worth of back data could take days. I'm also not even starting to count the CPU and memory necessary to build state tables and statistical analysis tables to generate useful data. There's a reason why most network traffic tools summarize early, as early as on the network device when using Netflow type collection. It's not just to save storage space on disk, but it's to make the processing of the data fast enough that it can be done in a short enough time that the data is still relevant when the processing is complete. -- Leo Bicknell - bicknell () ufp org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Attachment:
_bin
Description:
Current thread:
- Re: Network Storage, (continued)
- Re: Network Storage Valdis . Kletnieks (Apr 12)
- Re: Network Storage John T. Yocum (Apr 12)
- Re: Network Storage Dan Olson (Apr 12)
- Re: Network Storage Matthew Luckie (Apr 12)
- Re: Network Storage Jared Mauch (Apr 12)
- Re: Network Storage George Herbert (Apr 15)
- Re: Network Storage Andrew Thrift (Apr 15)
- Re: Network Storage Simon Leinen (Apr 16)
- RE: Network Storage Drew Weaver (Apr 16)
- Re: Network Storage Michael J McCafferty (Apr 12)
- Re: Network Storage Leo Bicknell (Apr 15)
- Re: Network Storage Jimmy Hess (Apr 12)
- Re: Network Storage Kyle Creyts (Apr 14)