Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ?

[Crusty Saint <saintcrusty () gmail com>]

Hi Rick,

Thank You for adding your take on this subject.

Before i wrote this mail i had not yet read thru the v6.1 Test Methodology
in full, the complementary information from this mail is sufficiënt to what
made me wonder.

Though after all the answers now are obvious.

S-C

To my regret i personally don't qualify to apply @ NSS Labs, for the Senior
Position, but i hope others will find their calling :)


2011/5/10 Rick Moy <rmoy () nsslabs com>

> I’m always glad to see there’s interest in our test results, and would like
> to address a few points raised on the thread.
>
>
>
> NSS Labs does not charge vendors to participate in these group tests.
> They’re free in order to remove any bias and reason for a vendor not to
> participate. Just like consumer reports, the benefactor of the information
> pays for the testing by buying a magazine, report or subscription.
>
>
>
> Compare this to the certification testing done by test labs who get paid by
> the vendors. Coincidentally, these same labs pass vendors’ products that
> would not survive our tests. Just reference our IPS, AV, or recent firewall
> tests, where 5 of 6 firewalls failed, despite having dual certifications
> from other labs. Why do we make the tests so hard? Because we see our
> mission is to provide buyers with comprehensive assessments so they can make
> informed buying decisions, and compensate for any holes in their defenses.
>
>
>
> About our IPS testing, there were some questions about attack surface. Our
> attack set includes exploits that return live shells against > 1200 CVSS 7+
> vulns, and growing. So most of our content is relevant to typical
> enterprises. And this is the largest set of vulns in any test (10x the other
> labs). Includes client and server attacks against all major OS and apps and
> patch levels.  Less mainstream OS & apps? This is where custom testing
> becomes important. Lots of methodology info on our site. But then you need
> the right tools, vulnerable hosts and exploits…
>
>
>
> In our reports, when we cite “default” or “recommended” policies, these are
> those that are defined by the vendors themselves, not by NSS Labs. Each
> vendor has a different approach, and we are baselining the out-of-the-box
> set of signatures. In addition, NSS Labs found that many IPS vendors were
> shipping with low default/recommended policies because they didn’t want to
> generate False Positives during the evals ;p. Unfortunately, many customers
> were not tuning. So we test both default and tuned policies to show the
> range of protection. To get tuned settings, we invite vendors to do their
> best job of tuning for an enterprise network – without generating any false
> positives.
>
>
>
> We also do TCO and performance comparisons, but you wont find us touting
> product X or Y as the best. There are many factors involved in selecting and
> tuning defenses and they should be weighed carefully. At the end of the day,
> security testing is tough, time consuming work to do scientifically and get
> right. If you can do that, more power to you (and BTW we’re hiring ;-).
>
>
>
> Regards,
>
> Rick Moy
>
> CEO, NSS Labs
>
> www.nsslabs.com
>
>
>
>
> ------------------------------------------------------------------------------
> Achieve unprecedented app performance and reliability
> What every C/C++ and Fortran developer should know.
> Learn how Intel has extended the reach of its next-generation tools
> to help boost performance applications - inlcuding clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users



-- 
- - -
Security Engineer - Tags: Analyst Systems Security Linux Firewall Network
Web Troubleshooting - If you think I deserve a rant, write me off-list

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users