Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ?

[Martin Holste <mcholste () gmail com>]

> The following needs to be true:
>
>  1. You need to know exactly what you are doing.
>  2. The test lab needs to be set up correctly with actual testing
> equipment. (including traffic generation equipment)
>  3. The attacks need to be valid (see number 1).
>  4. The normal traffic needs to be valid and must emulate real-world
> situations (see number 1).
>  5. You must be able to combine 3 and 4. (see number 1)
>  6. You need to test thoroughly, and more than once.
>  7. Your test results should be reproducible by someone else.
>  8. Essentially, you should follow the scientific method, see [0].
>  9. There is no such thing as a "small" test lab.
>  10. It will never be perfect.
>
> At this point, you might realize that the time involved and cost is not
> insignificant and it certainly makes the cost from a testing vendor to
> buy a report seem much more reasonable.

All good points.  I'd add this tidbit to ponder: As per #10, even a
perfect setup will yield imperfect results because of the dichotomy
between actual, real-world traffic and reproducible traffic.  As a
universal truth, every moment in time is absolutely unique, and
therefore you'll never get a perfect test.

So, if you're not going to get a perfect test anyway, here's what I
would suggest: get demos from the vendors and run them during the same
time of day on weekdays (or whatever you have to do to get reasonably
similar traffic patterns).  This means your results are irrelevant to
other orgs, but that's the price one pays for quick, usable results
for your own org.  Validate the similar traffic load by knowing what
your average flows per second and megabits per second were during the
testing (you should also pay attention to max values).  If they're
reasonable close (and in a business network, they are almost always
very close unless there's a holiday, etc.), then you should be getting
a good (but not perfect!) representation of the throughput
capabilities.  Then, create a "heartbeat" signature which will fire on
a web request to a site you have permission to make regular requests
to, and have a cron job or something make a web get at a regular
interval.  Count the number of hits, and you've got what I consider to
be the most accurate, usable representation of what you should expect
with regard to the device's throughput inspection capacity.

This allows you to separate tests based on signature quality,
inspection thoroughness, protocol decode, etc. from tests based on
throughput.  This is valuable because if the system can't handle the
load of your pipe to begin with, then the quality of all of the other
factors is irrelevant.  If you can be sure that it's finding your
heartbeats with all of the rules loaded on a full pipe, then you've
got solid footing for the rest of the considerations.  As a bonus,
it's also a great way to evaluate how easy it is to write signatures
for the platform.  I have a write-up on how to do this with Snort on
my blog at ossectools.blogspot.com.

------------------------------------------------------------------------------
WhatsUp Gold - Download Free Network Management Software
The most intuitive, comprehensive, and cost-effective network 
management toolset available today.  Delivers lowest initial 
acquisition cost and overall TCO of any competing solution.
http://p.sf.net/sfu/whatsupgold-sd
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users