Re: challenges in capturing Gigabit ethernet

[hank.schupp () mantech-ist com]

I've tried some experiments with collecting at high-speed off gigabit cards.  Created a test system that generated 
traffic at varied rates from 10 mbs to 1 gbs.  Though the cards themselves reported very little loss (Endace products) 
the ability to write the data to disk started failing around 250 mbs.  Though I intend to continue testing in the near 
future I first see that I need to become a little smarter on storage and which is the quickest for write-speed. 

Understand also that the tests were not done under the most scientific conditions - still repeated testing showed our 
breakdown was primarily the ability to write the data to disk and not the gigabit card's ability to process the data.

The next question would be, if we COULD write to disk as fast as we collected the data, at which point do other factors 
such as the OS, the CPU, the memory, or the aplication itself become a factor?  

The testing I am doing is for a Windows-based application and, as such, I have not yet looked at the same conditions 
under *nix.  I have some interest in that area though so may see what I can find out there as well.  

I know I have seen other folks discuss these same concerns here in the past.  I seem to recall someone else having 
found the ability to write to disk a severe limitation.  Mind you that most ID/IPS system do not try to write every 
packet to disk - they usually only write the packet or session that that triggered an event.  So, for IDS/IPS sensors 
any speed limitations tend to deal with how much "analysis" must be done on the traffic passing by/through the sensor.  
The more rules in effect - the less they are able to process.  Which is a big reason why "tuning" of sensors to your 
network is so important.  Normalizing the sensor to your particular network configuration, addressing scheme, etc. will 
reduce a LOT of the false positives.  Disabling rules that do not apply to your network also improves the sensor's time 
to spend analyzing the traffic you DO determine is valid.  

OK folks ... Flame on!  <grin>

Hank  

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------