RE: ICSA [WAS: Re: Intrusion Prevention]

[smarkle () icsalabs com]

> On 12/29/02 Greg Shipley wrote:
> Over the past six years Neohapsis Labs has been testing products in the
> security space, with the vast majority of our results appearing in Network
> Computing magazine. Year after year we learn from our successes, and
> mistakes, and roll that knowledge into our ever evolving testing
> methodologies.  We tend to be leaders in this regard.  For example. the
> careful reader will note that our documented testing methods in 1999
> weren't mirrored by others until around 2001, and that our present-day
> methods are quite a bit beyond what anyone else has done, to-date.

All - I have remained silent on this list for years. I am interested in
helping mature an Industry. That is what ICSA Labs does and IDS has been one
of my responsibilities since early 1999. After cutting through the stinging
criticism and saber rattling, I have chosen to respond only to the paragraph
above. Any vendor that knows the ICSA Labs testing methodology knows that
for over ten years we have perfected pass/fail certification testing with
evolving test methodology and criteria. We did this when everyone else
argued that it was the wrong approach. This is the standard, and it is in
fact the ICSA Labs approach that has been mirrored by other test labs. 

> On 1/18/01 Greg Shipley wrote:
> [edit] Don't get me wrong, I think there is a huge need for 3rd-party
involvement, and dare I say it, "certification."  
> IMHO, there are some fronts to this that are REALLY important on.  For
example, I've heard that the ICSA team is working on >IPSEC *compliance* and
interoperability testing.  Ok, that's huge, as anyone who has worked with
multi-vendor VPN 
> deployments knows that the VPN space is a mess on that front.

> The problem is, I question whether or not people are being mislead, and how
much good some of these certifications (like the >firewall one) really do.
Ultimately, does this type of "branding" help provide for a false sense of
security? [end]

The problem, clearly stated by Greg, is whether people are being misled. The
answer is emphatically NO. The ICSA Labs NIDS test is geared toward three
different network types. ICSA Labs has never mirrored the 1999 Neohapsis
test, nor will we - it was flawed. We have built a real network to test
NIDS. We have always used working exploits that are targeting a victim
machine that is vulnerable to each specific attack. We have also included
the first false positive test...ever. You may be a bit beyond, however, your
F-1 vs. Garbage Truck analogy reminds me of the tortoise and the hare. You
may have gone farther in terms of performance but you yourself have admitted
errors caused by the pace. This is where people have been misled. They read
a magazine article that states vendor x has the best NIDS. End-users do not
need to know who has the best product in a snap-shot-in-time lab test, they
need to know the best product for their real live environment. That is where
ICSA Labs NIDS testing and certification has excelled and IMNSHO will never
be caught.

Greg - I sincerely ask you to contact me off-line and discuss a possible
visit to the ICSA labs. It is evident by your post that you do not have a
complete knowledge of what we do. This thread has also included reference to
the ICSA Labs Firewall program. I have asked one of our most vocal critics
in the past to give you his opinion on the current state of the ICSA Labs
Firewall program. Look for a post in the near future on that subject.

Scott Markle
IDS Program Manager
ICSA Labs

***********************************************************************
This message is intended only for the use of the intended recipient and
may contain information that is PRIVILEGED and/or CONFIDENTIAL.  If you
are not the intended recipient, you are hereby notified that any use,
dissemination, disclosure or copying of this communication is strictly
prohibited.  If you have received this communication in error, please
destroy all copies of this message and its attachments and notify us
immediately.
***********************************************************************