Re: newbie quetsions

[avi chesla <chesla () 012 net il>]

In-Reply-To: <41DD51DF.9080407 () immunitysec com>

> Received: (qmail 13780 invoked from network); 7 Jan 2005 00:27:04 -0000
> Received: from outgoing.securityfocus.com (HELO outgoing2.securityfocus.com) (205.206.231.26)
>  by mail.securityfocus.com with SMTP; 7 Jan 2005 00:27:04 -0000
> Received: from lists.securityfocus.com (lists.securityfocus.com [205.206.231.19])
>       by outgoing2.securityfocus.com (Postfix) with QMQP
>       id 110EB14375C; Thu,  6 Jan 2005 17:32:05 -0700 (MST)
> Mailing-List: contact focus-ids-help () securityfocus com; run by ezmlm
> > Dave Aitel wrote:

> I guess the interesting thing is that you actually bought something for 
> your millions of dollars. Or perhaps it's a look into the Speed vs. 
> Accuracy trade off. Lots of other people have spent millions of dollars 
> on professional engines, but still fail the simple tests like this 
> because all nss.co.uk is testing for is extremely old attacks and 
> whether an IDS can take the load of millions of packets at once. This is 
> going to favor Snort-like systems largely at the expense of parsing 
> engines. I think it's telling that nss doesn't test MSRPC at all. It's 
> funny how the IDS industry has tuned itself. But set your MTU low 
> enough, and you can bypass some systems even if you're the only packets 
> on the wire. Doing SMB fragmentation basically guarantees it.
>
> If you're looking for a misleading test, the NSS.CO.UK tests are what 
> you want. They're not open tests. They're outdated. They largely test 
> for things you don't care about, such as pushing packets down a wire. No 
> scientific test should be non-repeatable, and no scientific test should 
> require such large amounts of money to change hands.

I really suggest reading the reports that NSS issues including their market overview and test methodology in order to 
learn about how to analyze and test security devices or any other communication devices. To say that NSS's tests are 
out of date is simply not true.        

Evaluation of IPS products raises a great challenge for the evaluator. In my experience, the NSS group does a very 
thorough and, perhaps most importantly, un-biased work with their round of tests of IPS devices.
By examining NSS's test methodologies (published in their site and in every report they issue), it is easy to recognize 
the level of understanding that the NSS group has in regarding to the IPS market and product positioning (this 
understanding is the first step in establishing the correct test scenarios and success criteria).  
 
Regarding to Evasion techniques, NSS's tests comprise more than enough methods that try to evade detection. These 
include: Packet fragmentation which include 19 different methods of IP packet fragmentation and Stream segmentation, 
URL Obfuscation which include 9 URL obfuscation techniques (e.g., URL encoding, premature URL ending, session splicing 
etc), other miscellaneous evasion techniques... Of course there will always be new evasion techniques but it seems that 
NSS has chosen to use the most updated and common ones. Let's remember that no test can include all the possible 
evasion techniques but the important thing is to aim as high as possible. 

NSS includes also special evasion techniques in order to test rate-based NIPS which are usually based on time-dependant 
thresholds. In order to test these detection engines NSS generates DoS attacks, network scans and self-propagating Worm 
activities with different delays between packets(e.g., very slow scans, random time between events, slow TCP connection 
floods, slow SYN attacks etc.). In this way NSS analyzes how sophisticated these rate-based detection engines are.  

According to NSS reports, they have all the equipment and experience that is needed in order to simulate background 
traffic that emulates "real" world legitimate user behaviors (throughout several popular applications).  This is a very 
important capability that helps to reveal false positive and misdetection percentages of the detection and prevention 
engines – maybe the most important test for IPS devices (as high percentages of false positive renders the IPS devices 
useless). 

NSS indeed pushes the products to their limits. I think that this is certainly necessary in order to  reveal how much 
"brain" work was invested in the hardware and software architecture. NSS's performance test includes playing with 
parameters such as number of simultaneous TCP connections, TCP connection rates, Packet sizes, packet rates, etc. This 
capability allows an analysis of the immunity of the detection engines against false positive and misdetections.  It is 
also interesting and educating to see how NSS approaches differently rate-based NIPS and Content-based NIPS with their 
performances and false positive rate tests.

Avi Chesla.

--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------