Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ?

[Rick Moy <rmoy () nsslabs com>]

I'm always glad to see there's interest in our test results, and would like to address a few points raised on the 
thread.

NSS Labs does not charge vendors to participate in these group tests. They're free in order to remove any bias and 
reason for a vendor not to participate. Just like consumer reports, the benefactor of the information pays for the 
testing by buying a magazine, report or subscription.

Compare this to the certification testing done by test labs who get paid by the vendors. Coincidentally, these same 
labs pass vendors' products that would not survive our tests. Just reference our IPS, AV, or recent firewall tests, 
where 5 of 6 firewalls failed, despite having dual certifications from other labs. Why do we make the tests so hard? 
Because we see our mission is to provide buyers with comprehensive assessments so they can make informed buying 
decisions, and compensate for any holes in their defenses.

About our IPS testing, there were some questions about attack surface. Our attack set includes exploits that return 
live shells against > 1200 CVSS 7+ vulns, and growing. So most of our content is relevant to typical enterprises. And 
this is the largest set of vulns in any test (10x the other labs). Includes client and server attacks against all major 
OS and apps and patch levels.  Less mainstream OS & apps? This is where custom testing becomes important. Lots of 
methodology info on our site. But then you need the right tools, vulnerable hosts and exploits...

In our reports, when we cite "default" or "recommended" policies, these are those that are defined by the vendors 
themselves, not by NSS Labs. Each vendor has a different approach, and we are baselining the out-of-the-box set of 
signatures. In addition, NSS Labs found that many IPS vendors were shipping with low default/recommended policies 
because they didn't want to generate False Positives during the evals ;p. Unfortunately, many customers were not 
tuning. So we test both default and tuned policies to show the range of protection. To get tuned settings, we invite 
vendors to do their best job of tuning for an enterprise network - without generating any false positives.

We also do TCO and performance comparisons, but you wont find us touting product X or Y as the best. There are many 
factors involved in selecting and tuning defenses and they should be weighed carefully. At the end of the day, security 
testing is tough, time consuming work to do scientifically and get right. If you can do that, more power to you (and 
BTW we're hiring ;-).

Regards,
Rick Moy
CEO, NSS Labs
www.nsslabs.com<http://www.nsslabs.com>

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users