About IPS testing (was: newbie quetsions)

["Julius Detritus" <julius.detritus () ifrance com>]

I will try to make constructive answers, as the purpose of the game is not
to flame NSS but to provide reliable information on how to choose an IPS.

* First, about methodology. 
Yes methodology is mandatory as it gives a unique way to build a homogeneous
checklist when testing. No doubt about that. However, methodology alone is
far from enough. How many CSO have strict methodology applied without any
technical backup. At last they can say their IT has been methodologically
rooted; and trust me, there are thousands of big companies out there that
have such certificate!

* Second, about up-to-date vulnerabilities. 
I easily understand that when you want to objectively test products such as
IPS you should test them with the same set of vulnerabilities. Then can't
those tests be performed in parallel with a methodology that would
explicitly notice that X vulnerabilities not older than 1 month old will be
tested ? 

Also everybody knows that IPS are subject to performance issues when they
are not tuned. So "old" and useless signatures are usually disabled in
production. Having them enabled for tests will make results unreliable as
they do not match production conditions.

* Third, about evasion techniques. 
I don't think that because some products are not able to handle old tricks,
tests should be limited so these ones. This would mean that products that
passed the tests are not crappy. Not that they can provide effective
protection...

* Last, about open tests.
There is no major risk in providing complete tests details (captures, list
of "standard" attacks and generic method for recent attacks). If they really
reflect real-world production environment, it will move vendors to meet
useful requirements and this is what we (end-user) want: real security
products. And we don't care if products were good or not at the beginning.
We are just concerned by results.

On the business point of view I don't think that NSS would be affected. You
may like it or not they are a "de facto" standard, and I hardly think that
somebody else can come out and say "hey I can do the same" and get the
business. Also, what has been very well pointed out, is that you can no more
perform performance tests with a few computers. So labs such the NSS one
will be needed anyway as most bosses (and mine first) will not give budget
for smartbits-like stuff. 

* Addendum : my sources
I said a lot of things about NSS tests. I have not been there. I got the
information from a guy working for a vendor (which got very good grade - I
wouldn't have trust him else) who was present when the tests were performed
and was quite disappointed (not to say hilarious). It was last year, maybe
things have changed since.

Julius


_____________________________________________________________________

Envie de discuter gratuitement avec vos amis ?
Téléchargez Yahoo! Messenger http://yahoo.ifrance.com


--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------