IDS mailing list archives
RE: ICSA Labs Network IPS Testing
From: "Walsh, John (Jack)" <jwalsh () icsalabs com>
Date: Wed, 12 Dec 2007 14:03:08 -0500
Rahul: It's true that the baseline set of criteria does focus on remotely exploitable, server-side vulnerabilities found in enterprise software. What is less well known is that there is an optional criteria module focused entirely on coverage protection for client-side vulnerabilities (also in enterprise software). As with the evolving server-side set of vulnerabilities, this client-side set is published online as well. You can find it here: http://www.icsalabs.com/icsa/docs/html/communities/nips/criteria/Vulnera bilitySet_ClientSide_070703.xls So, when we do the research twice a year (approximately) to determine the vulnerability set - it's done for both the server-side set required by the baseline set of criteria as well as for the optional client-side set. You probably noticed that no one has been tested successfully against this optional module. Enterprise end users may have to demand that products be tested for client-side vulnerability coverage protection (if they are interested in ensuring proper protection for attacks targeting both sets) before developers will pursue such testing. I can and have recommended it, but it really does take a push from end users. As I said before, if folks have questions about ICSA Labs Network IPS testing, please feel free to get in touch. Take care, Jack Walsh Technology Programs Manager, Intrusion Detection & Prevention ICSA Labs 717.790.8126 jwalsh () icsalabs com
-----Original Message----- From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Rahul K Sent: Tuesday, December 11, 2007 2:03 PM To: Stefano Zanero Cc: Focus-Ids Mailing List Subject: Re: ICSA Labs Network IPS Testing Hi, Having some experience in developing and testing IPS, I have my two bits to add. Most IPS tests, like Stefano said, are tricky at best and pointless at worst. I don't want to take any potshots at ICSA or anyone else, but it is not simple for anyone to do an exhaustive test of an IPS and that too with the same test plan for every IPS. ICSA, to their credit, say that of all the vulnerabilities they will only focus on remote server-side vulnerabilities and that too only those that they (and other vendors) think will affect enterprises. Fair enough. They don't care about client-side vulns, local vulns and vulnerabilities in Shoutcast. They test a particular subset (however small it may be) and certify the IPS. So even if one buys an IPS that blocks all server side attacks launched by ICSA, it does not mean that the server behind the IPS is secure from remote attacks. Vendors and buyers need such certifications so that it is easier to make a sale and deploy an IPS respectively - after all, not everyone subscribes to focus-ids. It would be reasonable to criticize ICSA if one finds out they are not doing what they promise correctly. But if the criticism is for not testing exhaustively, that seems excessive. Cheers, Rahul On 12/5/07, Stefano Zanero <s.zanero () securenetwork it> wrote:Hi, didn't mean to interfere in your ongoing flame, but:IPS certification testing, I thought I ought to correctsome misleadinginformationOh, good, let's see! You don't mind if instead of going through your whitepapers I just use your own email as a source, right?IPS certification testing program. The truth is that wedo not "pickspecific attacks and say that you must block these."That's wonderful to hear. So, what do you do instead?provides coverage protection for all attacks targeting anevolving setof medium-to-high severity vulnerabilities that we and aconsortium of15 network IPS vendors(http://www.icsalabs.com/icsa/topic.php?tid=6a87$5813f3e2-37b7
7ee3$3b4a-
f1d4a32d) believe are relevant to enterprise end users.So, you pick specific attacks (which are a snapshot of a set of vulnerabilities that you + the tested vendors believe arerelevant) andsay "you must block these", right ? This seems exactly the same sentence that Joel posted, onlya bit moreelaborate :) And just to shoot another shot in the dead horse of IDPS testing, testing MISUSE based detectors (as most IPS are) on"detection rate" ispointless. Testing them on coverage is tricky at best, and does not really provide any useful insight at all on IPS where (asJoel pointedout) having 60k signatures instead of 30k does not reallymean anything.Oh, and on a side note:a) is in no position to speak authoritatively about ICSALabs networkIPS testing,The sheer fact that someone is "in no position to speak" about your tests means that your tests are lacking. If a test is properly documented and scientific, everybody is in a position tospeak about it.In the particular case of Joel Snyder, who has been doing excellent tests for a long time, I'd say he is in a particularly goodposition tocomment. If this email sounds harsh, well, it is. I just don't like people commenting AGAINST other people, instead than pointing outthe specificflaws in their posts. Best, Stefano-------------------------------------------------------------- ----------Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go tohttp://www.coresecurity.com/index.php5?module=Form&action=impa
ct&campaign=intro_sfw
to learn more.-------------------------------------------------------------- ------------------------------------------------------------------------ ---------- Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.coresecurity.com/index.php5?module=Form&action=impa
ct&campaign=intro_sfw
to learn more. -------------------------------------------------------------- ----------
------------------------------------------------------------------------ Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw to learn more. ------------------------------------------------------------------------
Current thread:
- ICSA Labs Network IPS Testing Walsh, John (Jack) (Dec 04)
- Re: ICSA Labs Network IPS Testing Stefano Zanero (Dec 05)
- Re: ICSA Labs Network IPS Testing Rahul K (Dec 11)
- RE: ICSA Labs Network IPS Testing Walsh, John (Jack) (Dec 12)
- Re: ICSA Labs Network IPS Testing Rahul K (Dec 11)
- Re: ICSA Labs Network IPS Testing Stefano Zanero (Dec 05)