Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ?

[beenph <beenph () gmail com>]

I look at 1) and i doubt you can do that nigel ...*punt*


On Mon, May 9, 2011 at 9:14 AM, Nigel Houghton <nhoughton () sourcefire com> wrote:
> On Mon, 9 May 2011 07:50:36 -0400, firewalZ wrote:
> > I personally am a bit sceptical of NSS and other similar so-called
> > third party tests. From a user perspective, you can not get any
> > information from them without paying big $. From a vendor perspective,
> > you cannot participate without paying big $ (i have heard this same
> > story from more than one vendor). Smells a bit like a racket to me.
> > Bottom line is that if you are able, build a small test lab (many open
> > source options), look into getting demo loaners from various vendors
> > and test for your self. Understand your environment, potential attack
> > entry points and unique vulnerabilities. I feel this would be a far
> > greater use of the time and money that an NSS report would cost.
>
> I understand your concerns, they are the same ones voiced by many. It's
> not for me to say who is a good resource and who is not and how the
> various testing vendors go about their testing. I don't have enough
> detailed information on a lot of that information anyway. However, I do
> know quite a lot about in-house testing.
>
> The following needs to be true:
>
>  1. You need to know exactly what you are doing.
>  2. The test lab needs to be set up correctly with actual testing
> equipment. (including traffic generation equipment)
>  3. The attacks need to be valid (see number 1).
>  4. The normal traffic needs to be valid and must emulate real-world
> situations (see number 1).
>  5. You must be able to combine 3 and 4. (see number 1)
>  6. You need to test thoroughly, and more than once.
>  7. Your test results should be reproducible by someone else.
>  8. Essentially, you should follow the scientific method, see [0].
>  9. There is no such thing as a "small" test lab.
>  10. It will never be perfect.
>
> At this point, you might realize that the time involved and cost is not
> insignificant and it certainly makes the cost from a testing vendor to
> buy a report seem much more reasonable. Of course, you should try to
> test the equipment you are about to buy as you suggest as well,
> especially since no two networks are alike, but I wouldn't rely on only
> that testing to make a decision. (just as I wouldn't rely on only a
> testing vendors report to make a decision)
>
>  [0]  - http://en.wikipedia.org/wiki/Scientific_method
>
> --
> Nigel Houghton
> Head Mentalist
> SF VRT Department of Intelligence Excellence
> http://vrt-blog.snort.org/ && http://labs.snort.org/
>
> ------------------------------------------------------------------------------
> WhatsUp Gold - Download Free Network Management Software
> The most intuitive, comprehensive, and cost-effective network
> management toolset available today.  Delivers lowest initial
> acquisition cost and overall TCO of any competing solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users

------------------------------------------------------------------------------
WhatsUp Gold - Download Free Network Management Software
The most intuitive, comprehensive, and cost-effective network 
management toolset available today.  Delivers lowest initial 
acquisition cost and overall TCO of any competing solution.
http://p.sf.net/sfu/whatsupgold-sd
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users