RE: icsa ips testing vulnerability set

["Tim Holman" <tim_holman () hotmail com>]

Hi guys,

The vulnerabilities listed are outdated, yes:

http://www.icsalabs.com/icsa/docs/html/communities/nips/criteria/Vulnerabili
tySet_Baseline_060626.xls

They range from 2001-2005.  But what's more important is that they test a
wide range of applications and operating systems, to ensure that the network
IPS can cope with and decode a diverse range of packets that you'll see on a
typical network pipe.  I think it's fairly comprehensive, and based on this,
you can rest assured that if another vulnerability/exploit is discovered,
then the IPS in question has the relevant engines in place necessary to
decode the packets, and apply any new signatures without the developers
having to go back and create decodes for applications they don't understand
yet. 

Also, having a read through the reports, it appears that the ICSA tests
limit their SYN Flood attack tests to T3/45Mbps.

Whilst this is adequate for the majority of perimeter IPS installs, it
certainly doesn't put high-end IPS devices to the test, especially if
they're going to be installed in carrier networks, or used on enterprise
(even 100Mbps) networks.  

What could be better would be a scaled approach - for example, if ICSA could
test DOS attacks up to 25%, 50%, 75% and 100% of the device's rated
throughput.  This will then give punters a good idea of whether or not the
device in question is suitable for a large-enterprise or carrier-class
network, and allow a realistic comparison between vendors that submit their
flagship devices, to those that want to test their mid-range devices, or
perhaps don't have a device that sits in the carrier-class space.

We all know the TP 5000E is carrier-class, but running it through the same
tests as a smaller BroadWeb device is going to confuse and potentially
mislead punters who need reassurance that the carrier-class devices they've
just paid a good $150k for has been FULLY validated by ICSA.

Great work by ICSA so far - the tests are very comprehensive, but I really
think you should either draw a line and keep the Ferraris out of the GoKart
race, or put appropriate scaling into your tests so that ANY network IPS for
ANY purpose can be fairly validated.

Regards,

Tim




-----Original Message-----
From: Stefano Zanero [mailto:zanero () elet polimi it] 
Sent: 26 July 2006 13:51
To: Ronny Vaningh; Focus-Ids Mailing List
Subject: Re: icsa ips testing vulnerability set

Ronny Vaningh wrote:
> While I was reviewing ICSA "Network IPS Corporate Testing Criteria" I

Disclaimer: didn't read that document, so I'm commenting on your comment.

> really got the impression that they used a fairly outdated set of
> vulnerabilities.

The problem is more basic.

You are thinking of a coverage test, meaning "let's see how many attacks
they do block". Trouble is, this is misuse detection, so this does not
make much sense. If you shoot at those appliances an attack they have a
signature for, they'll almost invariably catch it. If it's a new attack,
or one they don't have a signature for, they won't.

> What do you think ?

> From my point of view, testing IDS coverage in width, in particular in
misuse detection systems, is pointless. It makes slightly more sense to
test for the ability to recognize classes of attacks.

Further details on my black hat federal presentation that I won't spam
anymore *eg*

Stefano

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------


------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------