Re: Testing IDS with tcpreplay

[Ivan Arce <ivan.arce () coresecurity com>]

Hi Aaron.

See my comments inline as well ...

Aaron Turner wrote:
> Inline...
>
> On 2/20/06, Ivan Arce <ivan.arce () coresecurity com> wrote:
> > Aaron Turner wrote:
>
> [snip]
>
> > > Generally speaking,  tcpreplay is better when one or more of the
> > > following is true:
> > >
> > > 1) Trying to do comparative analysis and you want to make sure each
> > > device sees exactly the same thing
> > Hmm, why is that harder to accomplish with Metasploit than with tcpreplay?
>
> Because metasploit, other tools and exploits incorporate PRNGs and
> other methods of altering the attack so that it isn't exactly the same
> each time.   Sure it's the same "exploit", but the bits on the wire
> are different.  That makes "each device sees exactly the same thing"

That was exactly my point.

If you are testing you IDS you'd like to know that it accurately detects
an exploit for a given vulnerability not that it detects the packets
that comprise a particular instance of the exploit's execution.

What I tried to point out is that with tcpreplay you end up testing
different things than with Metasploit or similar tools. Using tcpreplay
it good to build a valid and easily repeatable set of tests but it's
important to know the difference with the real thing lest you'll get
burned when a different exploit for the same bug or even the same
exploit in a different operational environment hits your IDS.

> really difficult.   Of course some tools allow you to control these
> via seeds but others don't.
>
> > > 2) Need to automate or do a lot of regression testing and want a
> > > stable and relatively simple lab environment
> > same as above....
>
> Again, less complex (no 2nd box and vmware to maintain/automate) and
> 100% repeatable data streams.  Also what about attacks that Metasploit
> doesn't have?  What if you want background traffic?

Those are certainly good points.
Regarding attacks that Metasploit does not have; I guess then you'd need
to find an attack for the bug elsewhere (either publicly available or in
a commercial product), otherwise how would you have the replayable
attack traffic anyway?

As for background traffic, yep that is certainly helpful and I recognize
the value of tcpreplay there, which is way smarter than simply using a
traffic generator to inject random packets that are completely out of
context for the test environment

> > > 3) Already have a library of pcap's (either from customers, the wild
> > > or capturing traffic of real tools like Metasploit)
> > Yeah, but that is an entirely different kind of testing. Replaying the
> > packets captured from the execution of an exploit is not the same as
> > executing the same exploit again.
>
> If you're testing a vulnerable application then I agree.  but if you
> are testing an IDS/IPS,  then I would argue that it is for all intents
> and purposes it's the same thing.  If you believe otherwise, then
> please explain.

Ok, I think I've touched on the point in my above comment but I will
explain my view a bit more.

I would argue that you are testing the IDS to figure out if it will be
capable of detecting exploitation attempts against real applications in
their real operational environment. If that is in fact the case, then
you need to make sure that the exploit traffic you're using to test your
IDS is likely the same that the IDS will see coming after the real
applications. It follows then that if the exploit that generates the
traffic will behave differently in the face of the real application then
your captured-replayed traffic can't really assure that the IDS will
work properly (unless you really understand how the exploit will behave
and what traffic it will generate in any random execution).


My point is that modern exploits do behave differently based on the
runtime characteristics of the target system and application. I am not
talking about shellcode obfuscation or NIDS evasion attempts but about
exploit logic. For example if a given exploit needs to fill-up the
process heap after/before triggering the bug or obtain heap layout data
to change execution flow on the target app. then it will send more/less
-or even different- traffic depending on the endpoint's application state.

> The point being you have to put forth the effort to automate
> vmware/metasploit is a lot more work then just tcpreplay.  Not to
> mention, if you want to add other tools other then metasploit now you
> have more automation work to do.  tcpreplay is exploit/tool agnostic
> and provides a simple and unified method of generating all kinds of
> traffic- not just exploits.


agreed and understood, I am just pointing out that you will accomplish
different things depending on which test strategy you choose

> [snip]
>
> > > In general, tcpreplay isn't all that useful IMHO when you're first
> > > starting off and "want to do some IDS/IPS testing" or only intend to
> > > run a few tests or tests only once or twice unless you already happen
> > > to have a nice pcap library.
> > Ahh that's interesting, I see it in exactly the opposite way: tcpreplay
> > is ok when you want to scratch the surface of your IDS capabilities or
> > perhaps more appropriate for stress or throughput testing or very basic
> > regression testing. However, if you truly want to check if the IDS
> > recognized real attacks you need to test with real exploit runs not with
> > a replay of their captured traffic.
>
> Why?  What is the different between "real exploit runs" vs. "replaying
> real exploit runs" from the viewpoint of an IDS/IPS/UTM/etc?

I think I explained this in the paragraphs above.

> Perhaps you're not aware, but tcpreplay fully emulates both client and
> server state, from protocol setup, to the actual exploit and all the
> way through getting a shell/whatever if that is what you captured.  An


You mean at the TCP/IP stack level or does it emulate the application's
state as well?

> IPS/IDS has absolutely no way of knowing if traffic was generated by
> tcpreplay or not, because it is EXACTLY like the traffic that was
> originally generated/captured.  If you want to run the same exploit
> 500 times with different parameters with Metasploit, that's great. 

You can run the same exploit 500 times with exactly the same parameters
and the generated traffic will not be the same. Some examples:
- The Apache-SSLv2 exploit used by the Slapper worm
- An exploit for the php_memory_limit bug
- An exploit for MS License Logging Service (LLSRV)

The reason for this is that they have logic to take different decisions
or to use different data (pointers,etc) based on the heap state of the
target application.


> Just capture each of those 500 attacks and replay each of them against
> 10 different IDS/IPS's.
>
> The point is that with tcpreplay, your test cases are 100% repeatable.

Yes, but that does not assure 100% code coverage of the exploit you used
to generate the captured traffic.

As I said, I see the value of using tcpreplay but the implications need
to be clearly understood so you know what you're testing.

-ivan

---
"Buy the ticket, take the ride" -HST

Ivan Arce
CTO

CORE SECURITY TECHNOLOGIES
http://www.coresecurity.com

PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836  B25D 207B E78E 2AD1 F65A


------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------