IDS mailing list archives
RE: NSS Certification - Credible?
From: Andrew Plato <andrew.plato () anitian com>
Date: Mon, 2 Mar 2009 13:55:34 -0800
It is important to note that tests are laboratory experiments, not the real world. While I respect the effort and energy it takes to perform a sound IPS or firewall test, I find myself often disagreeing with them because they test a fantasy environment, not reality. And there is not any sound way to test in a real environment. For example, I find that most testers pay little to no attention to the usability and on-going maintenance effort of a product. At the end of the day, how a product is used has a much more profound impact on its success than the quality of the engine. The best IPS in the world is not going to be useful in the hands of a unskilled or irresponsible administrator. Likewise, a mediocre or poor IPS can be made quite useful, if the administrator uses it to its full potential. Furthermore, I have come to the conclusion that magazine tests are by and large worthless. They are all to often influenced by the advertising in the magazine. I know people say that isn't true. But, I just don't believe it. And any test that is performed by a single person is also flawed. The test is entirely dependent upon that person's obsessions and preferences. And honestly, most magazine tests I have seen show obvious biases toward certain players. This is why any company considering an IPS or any complex security technology should make a short list of products and then talk to other companies using those products. You can learn a lot more about a product from collaborating with other users than you can from a certification report or a magazine review. As for NSS, its as credible as a lab test can be. I would not use NSS exclusively as a buying guide. There are some products they have "certified" that are, IMO, truly awful products. Nevertheless, NSS certification should be taken as a positive for any short list of vendors. Andrew Plato, CISSP, CISM, QSA President/Principal Consultant Anitian Enterprise Security -----Original Message----- From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Joel M Snyder Sent: Monday, March 02, 2009 10:43 AM To: Ravi Chunduru Cc: Focus IDS Subject: Re: NSS Certification - Credible? I would contend that this is "best of a bad thing." I have done an enormous amount of testing myself on network security products for over 20 years, and Bob Walder's NSS tests are the best out there. The first thing you have to understand is that this kind of testing is VERY expensive; it costs a lot of money for the equipment, but it costs even more money for the time. Only when a lab like NSS is actually getting paid do they have the luxury of doing a very good job. When we test for publications like Network World, we are on a dramatically lower budget--we'll test 5 to 10 products for about a 10th of what NSS charges to test a single product. I think that the "certification" thing is a pile of crap (not just with NSS, but with every vendor that offers a 'check mark' or 'gold' or 'certified' level). However, what comes out of NSS, in addition to the useless badges, is an ENORMOUS report based on what they actually saw and didn't see. That's the value of their work, and that's why I continue to believe that they are the best private test lab in our space. Yes, all of the criticisms you mount (such as the ability of the vendor to have a 'do over') are valid, but if you want someone who at least has the veneer of independence (despite their being paid by the vendor), then the NSS reports are very worthwhile reading. This may change over time---it's no longer Bob and the South of France; it's now a real company in the US with bigger pressures to perform. And this is what has caused other previously-reputable testers to have lost their reputation. So, take it with a grain of salt, but anyone who does NOT read the NSS reports on products that they have tested is cutting themselves off from a huge supply of very high quality data. I won't make that statement for most of the other "labs" out there who are doing commercial testing. jms Disclosure: I've never taken money from NSS, ever. I'm just a fan. -- Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719 Senior Partner, Opus One Phone: +1 520 324 0494 jms () Opus1 COM http://www.opus1.com/jms
Current thread:
- NSS Certification - Credible? Ravi Chunduru (Mar 02)
- Re: NSS Certification - Credible? Joel M Snyder (Mar 02)
- Re: NSS Certification - Credible? Jeremy Bennett (Mar 02)
- RE: NSS Certification - Credible? Andrew Plato (Mar 03)
- Re: NSS Certification - Credible? Joel M Snyder (Mar 05)
- Re: NSS Certification - Credible? Stefano Zanero (Mar 09)
- Re: NSS Certification - Credible? Joel M Snyder (Mar 02)