Re: Testing IDS with tcpreplay

[Bob Walder <bwalder () spamcop net>]

I agree with Aaron - the only way to ensure two test runs are identical for
two different IPS' is to use replay tools. If the traffic is captured
properly (cleanly) and the replay tool is intelligent enough then there is
no way for the IPS to discern that the replayed traffic is different from
the original.

We have numerous tools like Canvas, Metasploit, etc, etc as well as live
exploits, but the first thing we do whenever we get a new tool/exploit is
run it, capture the traffic, verify the effect, and then we use the PCAP
from that point onwards.

There are some things that cannot be captured and replayed - fragroute
evasion traffic, for example, as well as some complex attacks that mess up
the replay tool - and for those we have to run them live whenever we test.

The replay tools keep things accurate and repeatable, ensure the same test
for everyone, and save us huge amounts of time during the run allowing us to
run far more tests in the same test slot.

Bear in mind that many exploits will DOS the server or alter its behaviour
to subsequent exploit attempts - even with Vmware images it is a pain to
keep re-initialising between runs - forget to do that, and a subsequent
exploit could fail and you might take that as a "miss" for the IPS when the
altered traffic stream is no longer detected as malicious.

IMHO, replayed traffic is the most accurate means of testing IPS/IDS
products - but you DO need a source of good, clean, valid PCAPs, and to
create those, a lot of the exploit tools mentioned in threads like this are
invaluable. In other words, you need both!

Bob Walder
The NSS Group


On 22/2/06 03:59, "Aaron Turner" <synfinatic () gmail com> wrote:

> Inline...
>
> On 2/20/06, Ivan Arce <ivan.arce () coresecurity com> wrote:
> > Aaron Turner wrote:
>
> [snip]
>
> > > Generally speaking,  tcpreplay is better when one or more of the
> > > following is true:
> > >
> > > 1) Trying to do comparative analysis and you want to make sure each
> > > device sees exactly the same thing
> >
> > Hmm, why is that harder to accomplish with Metasploit than with tcpreplay?
>
> Because metasploit, other tools and exploits incorporate PRNGs and
> other methods of altering the attack so that it isn't exactly the same
> each time.   Sure it's the same "exploit", but the bits on the wire
> are different.  That makes "each device sees exactly the same thing"
> really difficult.   Of course some tools allow you to control these
> via seeds but others don't.
>
> > > 2) Need to automate or do a lot of regression testing and want a
> > > stable and relatively simple lab environment
> >
> > same as above....
>
> Again, less complex (no 2nd box and vmware to maintain/automate) and
> 100% repeatable data streams.  Also what about attacks that Metasploit
> doesn't have?  What if you want background traffic?
>
> > > 3) Already have a library of pcap's (either from customers, the wild
> > > or capturing traffic of real tools like Metasploit)
> >
> > Yeah, but that is an entirely different kind of testing. Replaying the
> > packets captured from the execution of an exploit is not the same as
> > executing the same exploit again.
>
> If you're testing a vulnerable application then I agree.  but if you
> are testing an IDS/IPS,  then I would argue that it is for all intents
> and purposes it's the same thing.  If you believe otherwise, then
> please explain.
>
> > > 4) Don't want to worry about re-installing or fixing target systems
> > > after they've been 0wn3d.  VMware of course helps, but there is still
> > > a lot more administrative overhead.
> >
> > Hmm, maybe or maybe not... Actually you can pretty much automate the
> > entire process (or a big part of it):
> >
> >  1. set up of the proper VMware images (specially if you're using GSX or
> >  a similar virtualization server that lets you manage images
> > programatically and from remote)
> >  2. run a set of exploits in the appropriate order
> >  3. generate reports or other output with the results
> >  4. correlate output with IDS/IPS  alerts/logs/etc.
>
> The point being you have to put forth the effort to automate
> vmware/metasploit is a lot more work then just tcpreplay.  Not to
> mention, if you want to add other tools other then metasploit now you
> have more automation work to do.  tcpreplay is exploit/tool agnostic
> and provides a simple and unified method of generating all kinds of
> traffic- not just exploits.
>
> [snip]
>
> > > In general, tcpreplay isn't all that useful IMHO when you're first
> > > starting off and "want to do some IDS/IPS testing" or only intend to
> > > run a few tests or tests only once or twice unless you already happen
> > > to have a nice pcap library.
> >
> > Ahh that's interesting, I see it in exactly the opposite way: tcpreplay
> > is ok when you want to scratch the surface of your IDS capabilities or
> > perhaps more appropriate for stress or throughput testing or very basic
> > regression testing. However, if you truly want to check if the IDS
> > recognized real attacks you need to test with real exploit runs not with
> > a replay of their captured traffic.
>
> Why?  What is the different between "real exploit runs" vs. "replaying
> real exploit runs" from the viewpoint of an IDS/IPS/UTM/etc?
>
> > > Obviously the biggest limitation of tcpreplay is it doesn't come with
> > > a library of pcaps.  Maybe one of these days I can figure out the
> >
> > In my view, the biggest limitation is that replaying captured packets an
> > overly simplified manner of modeling real world attacks. Today's exploit
> > code is a lot "smarter" than simple PoC that send the same fixed data on
> > each run. modern exploits make runtime decisions based on the state of
> > the target system/application and several other things. To successfully
> > simulate the execution of real exploits you need to maintain state about
> > both endpoints (target and attacker's systems) and properly simulate the
> > meaningful state changes in them that would change exploit-code's
> > execution flow and elicit different traffic patterns that those from
> > previous runs.
>
> Perhaps you're not aware, but tcpreplay fully emulates both client and
> server state, from protocol setup, to the actual exploit and all the
> way through getting a shell/whatever if that is what you captured.  An
> IPS/IDS has absolutely no way of knowing if traffic was generated by
> tcpreplay or not, because it is EXACTLY like the traffic that was
> originally generated/captured.  If you want to run the same exploit
> 500 times with different parameters with Metasploit, that's great.
> Just capture each of those 500 attacks and replay each of them against
> 10 different IDS/IPS's.
>
> The point is that with tcpreplay, your test cases are 100% repeatable.
>  You *know* that the all those runtime decisions that happened in the
> original are replayed exactly the same way each and every time.
>
> Why is this important?  Well consider this:
>
> Say you have two IPS's you want to test.  You an send an "attack" with
> Metasploit against the first one and it detects it.  You run it again
> against the second one and it doesn't.  Does that mean the first one
> is better?  Well unless the byte stream was exactly the same in both
> cases (because Metasploit chose different runtime parameters based on
> a PRNG, time of day or whatever) or maybe you upgraded Metaspolit
> since the first time and things changed, you don't know.  That's why
> people should use tcpreplay.
>
> --
> Aaron Turner
> http://synfin.net/
>
> ------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it
> with real-world attacks from CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
> to learn more.
> ------------------------------------------------------------------------



------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------