RE: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk

["Brian Smith" <bsmith () tippingpoint com>]

On Nov 2, Marty Roesch wrote:

> As far as pcaps are concerned, pcaps in a vacuum don't really add a  
> whole lot beyond just testing basic detection capabilities.  
> You need  
> to have real high grade network testing equipment like the stuff  
> Spirent makes so that you can develop normalized, repeatable test  
> environments in which to test detection capabilities.  Measuring  
> latency, throughput, etc is also best done in an environment 
> where you  
> can setup repeatable test environments or at least where you 
> can setup  
> repeatable baseline environments to transmit your pcaps over the top  
> of.  Tcpreplay doesn't meet this requirement particularly 
> well all by  
> itself, nor will the TippingPoint software.

Actually, tomahawk is less designed for testing IPS detection/blocking
capabilities (although it can be used for that) and more geared toward
setting up realistic, repeatable background traffic mixes.  I developed
it to directly address several limitations in the switch/router test gear.

As you say, the good thing about the Spirent, et. al. gear is that it is
very precise -- it will tell you latency within 10 ns, for example.  The
limitation is that the traffic generated looks nothing like the traffic
that appears on a real network once you go past the headers.  The same is
true for most of these tools.  One exception is WebAvalanche/WebReflector,
which generates fairly realistic traffic, but only for a few protocols, so
the mix is unrealistic.

When testing a switch, router, or firewall, the traffic generated
by these devices is fine.  But when testing anything that goes deep
into the stream, like an IPS, you need to make sure the data that it's
inspecting is as realistic as possible.  The traditional router/switch
test gear just doesn't do that.

Worse yet, they can give you misleading results because the IPS may be
optimized for the traffic.  As a trivial example, suppose you use smartbits
to send ethernet frames padded with zeros (so that it's all zeros above
layer 2).  An IPS can look at that data, quickly determine that it's
not IP, and send it on its way with no further processing.  The test will
show that the IPS has great latency and throughput, but predicts nothing
about how it will perform in a real network (unless you deploy it in bypass
mode :-).  

Tomahawk can be used to set up a repeatable traffic test using pcaps
from the target network, which gives a more realistic protocol mix.
The throughput stabilizes after a minute or two and is repeatable to
a few percent -- some of this noise is caused by the sampling methods
used to compute throughput (we sample the NIC stats, which are only
updated every 2 seconds), some from the non-determinism in some IPSs
(e.g., a software product using a non real-time OS will always have a
tiny amount of noise, plus there's caching effects, etc.).

So how do you test this stuff, given these considerations?  A reasonable
compromise that we use is to find the throughput limits using tomahawk with
a pcap taken from the target network, and then test latency by loading the
box to, say, 90% of that limit with tomahawk and use smartbits to find the
latency by sending a modest stream of UDP or ICMP traffic (so the IPS can't
just ignore them).

Ultimately, the quality of any test is it's predictive power: that is, how
well does the test predict the performance of the product in the real
situation.  That's how we should be measuring the effacy of these tools.
If the prediction is off by an order of magnitude, those extra significant
figures in the measurement don't do you a lot of good.

        Brian

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------