RE: NSS Certification - Credible?

[Andrew Plato <andrew.plato () anitian com>]

It is important to note that tests are laboratory experiments, not the
real world. While I respect the effort and energy it takes to perform a
sound IPS or firewall test, I find myself often disagreeing with them
because they test a fantasy environment, not reality. And there is not
any sound way to test in a real environment. 

For example, I find that most testers pay little to no attention to the
usability and on-going maintenance effort of a product. At the end of
the day, how a product is used has a much more profound impact on its
success than the quality of the engine.  The best IPS in the world is
not going to be useful in the hands of a unskilled or irresponsible
administrator.  Likewise, a mediocre or poor IPS can be made quite
useful, if the administrator uses it to its full potential. 

Furthermore, I have come to the conclusion that magazine tests are by
and large worthless.  They are all to often influenced by the
advertising in the magazine. I know people say that isn't true. But, I
just don't believe it. And any test that is performed by a single person
is also flawed. The test is entirely dependent upon that person's
obsessions and preferences. And honestly, most magazine tests I have
seen show obvious biases toward certain players. 

This is why any company considering an IPS or any complex security
technology should make a short list of products and then talk to other
companies using those products. You can learn a lot more about a product
from collaborating with other users than you can from a certification
report or a magazine review. 

As for NSS, its as credible as a lab test can be. I would not use NSS
exclusively as a buying guide. There are some products they have
"certified" that are, IMO, truly awful products. Nevertheless, NSS
certification should be taken as a positive for any short list of
vendors. 


Andrew Plato, CISSP, CISM, QSA
President/Principal Consultant
Anitian Enterprise Security 
 

-----Original Message-----
From: listbounce () securityfocus com [mailto:listbounce () securityfocus com]
On Behalf Of Joel M Snyder
Sent: Monday, March 02, 2009 10:43 AM
To: Ravi Chunduru
Cc: Focus IDS
Subject: Re: NSS Certification - Credible?

I would contend that this is "best of a bad thing."

I have done an enormous amount of testing myself on network security
products for over 20 years, and Bob Walder's NSS tests are the best out
there.

The first thing you have to understand is that this kind of testing is
VERY expensive; it costs a lot of money for the equipment, but it costs
even more money for the time.  Only when a lab like NSS is actually
getting paid do they have the luxury of doing a very good job.

When we test for publications like Network World, we are on a
dramatically lower budget--we'll test 5 to 10 products for about a 10th
of what NSS charges to test a single product.

I think that the "certification" thing is a pile of crap (not just with
NSS, but with every vendor that offers a 'check mark' or 'gold' or
'certified' level). 
However, what comes out of NSS, in addition to the useless badges, is an
ENORMOUS report based on what they actually saw and didn't see.  That's
the value of their work, and that's why I continue to believe that they
are the best private test lab in our space.

Yes, all of the criticisms you mount (such as the ability of the vendor
to have a 'do over') are valid, but if you want someone who at least has
the veneer of independence (despite their being paid by the vendor),
then the NSS reports are very worthwhile reading.

This may change over time---it's no longer Bob and the South of France;
it's now a real company in the US with bigger pressures to perform.  And
this is what has caused other previously-reputable testers to have lost
their reputation.

So, take it with a grain of salt, but anyone who does NOT read the NSS
reports on products that they have tested is cutting themselves off from
a huge supply of very high quality data. I won't make that statement for
most of the other "labs" out there who are doing commercial testing.

jms

Disclosure: I've never taken money from NSS, ever.  I'm just a fan.

--
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One       Phone: +1 520 324 0494
jms () Opus1 COM                http://www.opus1.com/jms