Re: newbie quetsions

[Mike Paquette <paquette () toplayer com>]

In-Reply-To: <41DD51DF.9080407 () immunitysec com>

Dave,

I've been following this thread for the last week or so.  As you may know, we like your CRI tool, and indeed we use it 
to test RPC handling and inspection in Top Layer's network IPS products.  I also appreciate many of your comments 
regarding the importance of the IDS/IPS properly handling IP fragments, TCP segments, and RPC fragments in order to 
defeat evasion attempts.

I'm not quite sure, however, why you're bashing the NSS IPS tests.  Your comments seem to be applying a very narrowly 
defined criterion as the basis for dismissing the entire NSS IPS test suite.  Specifically, I must take exception to 
your claim that "They largely test for things you don't care about, such as pushing packets down a wire."

As a vendor of IPS products, I can tell you that organizations planning to deploy network IPS technology are VERY 
interested in how well the IPS can push packets down the wire!  They all run businesses, and the packets being "pushed 
down their wire" are their lifeblood: payment transfers, sports bets, media delivery, internal application requests, 
etc.  In my experience, the ability of the IPS to handle legitimate traffic as a "good networking device" is often used 
as the *first* set of criteria in selecting an inline IPS product.  I've had many a customer who literally said that 
they didn't even want to *talk* about the protection mechanisms until we'd proven that our device could operate as a 
"good network citizen."

We've run our products through the NSS IPS tests, and I just can't agree with the rest of your comments:

> "They're not open tests."

The NSS test methodologies are published in full.

> "They're outdated."
 
The first IPS test was a year ago and the NSS methodology was brand new.  You're right that it's mostly the same this 
year, save for some new exploits, but I would not consider it outdated.  I don't know of a more recent or more 
comprehensive set of tests for a network IPS.

> "They largely test for things you don't care about, such as pushing packets down a wire..."
 
My experience shows that organizations DO care about the things that NSS tests for: signature coverage, baseline 
performance, performance under load, latency, application response times, anti-evasion capabilities, stateful 
operation, management and configuration.  I already  mentioned my view about "pushing packets down the wire."
Bob Walder from NSS can chime in here, but my understanding is that the NSS signature coverage tests include many 
RPC-related exploits and their variants, run both "in the clear" and with various evasion techniques, including 
modified exploit code and RPC fragmentation.

> "No scientific test should be non-repeatable"

We've been able to repeat the majority of the NSS tests consistently in our lab.  You might be talking about the fact 
that the capture files for the attack recognition tests are not publicized.  This topic was addressed in the thread 
regarding the Tipping Point Tomahawk tool already.  Clearly the set of "attacks" used is the result of work that NSS 
has performed, and I understand their desire to keep that proprietary.

> "and no scientific test should require such large amounts of money to change hands."

Do you mean the test fees? The report fees? If so, why not? It's called "business."  Only by charging money can a test 
house spend the amount of time necessary to REALLY test advanced products like network IPS.  In fact, you might be able 
to use CRI to create your own mini-test, and charge IPS vendors to participate in it!  Or why not work with Walder 
directly to have him use CRI to enhance his evasion section of his test?

Mike Paquette
VP Technology,
Top Layer Networks, Inc.

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------