NSS Certification - Credible?

[Ravi Chunduru <ravi.is.chunduru () gmail com>]

One interesting and provactive slide "Effectiveness" here:
http://nsslabs.com/webinars/NSS%20Labs%2010g%20webinar.pdf
I agree some what  with what was said there, but testing with private
exploits alone does not make NSS testing credible.  I feel that there
are some points which IDP buyers would like to know while selecting
the IDP vendor.

How many times vendor failed in testing before the product was
certified?  My understanding is that NSS allows vendors to provide
signature pack during testing if it does not meet the pass criteria.
Shouldn't this failed number be known to IDP buyers?  I also  feel
that buyers would like to know the Initial coverage number.  Without
that I don't see the difference between public testing houses and NSS.

To make buyers comfortable, I believe testing should be done
periodically (Once in a month?) on certified products and take them
off the certified list if they don't meet the criteria.  I noted that
there are some products in the certified list dating back 2004/2005.

> From the test report, it appears that NSS certifies if 30-40% of
client side attacks are detected.  Are buyers comfortable with this
number?

Number of tests made are dismal around 500+.  Does that number good enough?

Buyers know their internal assets (protocols, applications, operating
systems etc..) and would like to see certifications providing detailed
information on security effectiveness of common protocols and
applications.  I don't see these details on NSS reports.  I am not
sure whether this was the intention of testing by public houses, but
one knows clearly on products and their coverage with respect to
vulnerabilities and exploits.

By the way, are there any testing & certification houses targeting
measurement of security coverage with respect to individual protocols
servers HTTP, FTP, SSH, SIP, LDAP, SQL Server etc.?

Thanks
Ravi