Snort mailing list archives
Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ?
From: Martin Holste <mcholste () gmail com>
Date: Mon, 9 May 2011 09:50:32 -0500
The following needs to be true: 1. You need to know exactly what you are doing. 2. The test lab needs to be set up correctly with actual testing equipment. (including traffic generation equipment) 3. The attacks need to be valid (see number 1). 4. The normal traffic needs to be valid and must emulate real-world situations (see number 1). 5. You must be able to combine 3 and 4. (see number 1) 6. You need to test thoroughly, and more than once. 7. Your test results should be reproducible by someone else. 8. Essentially, you should follow the scientific method, see [0]. 9. There is no such thing as a "small" test lab. 10. It will never be perfect. At this point, you might realize that the time involved and cost is not insignificant and it certainly makes the cost from a testing vendor to buy a report seem much more reasonable.
All good points. I'd add this tidbit to ponder: As per #10, even a perfect setup will yield imperfect results because of the dichotomy between actual, real-world traffic and reproducible traffic. As a universal truth, every moment in time is absolutely unique, and therefore you'll never get a perfect test. So, if you're not going to get a perfect test anyway, here's what I would suggest: get demos from the vendors and run them during the same time of day on weekdays (or whatever you have to do to get reasonably similar traffic patterns). This means your results are irrelevant to other orgs, but that's the price one pays for quick, usable results for your own org. Validate the similar traffic load by knowing what your average flows per second and megabits per second were during the testing (you should also pay attention to max values). If they're reasonable close (and in a business network, they are almost always very close unless there's a holiday, etc.), then you should be getting a good (but not perfect!) representation of the throughput capabilities. Then, create a "heartbeat" signature which will fire on a web request to a site you have permission to make regular requests to, and have a cron job or something make a web get at a regular interval. Count the number of hits, and you've got what I consider to be the most accurate, usable representation of what you should expect with regard to the device's throughput inspection capacity. This allows you to separate tests based on signature quality, inspection thoroughness, protocol decode, etc. from tests based on throughput. This is valuable because if the system can't handle the load of your pipe to begin with, then the quality of all of the other factors is irrelevant. If you can be sure that it's finding your heartbeats with all of the rules loaded on a full pipe, then you've got solid footing for the rest of the considerations. As a bonus, it's also a great way to evaluate how easy it is to write signatures for the platform. I have a write-up on how to do this with Snort on my blog at ossectools.blogspot.com. ------------------------------------------------------------------------------ WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Current thread:
- NSS Labs : CheckPoint 97.3% recommended profile hoax ? Crusty Saint (May 05)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Joel Esler (May 05)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? firewalZ (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Nigel Houghton (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? beenph (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Martin Holste (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Crusty Saint (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Nigel Houghton (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Jason Brvenik (May 09)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Crusty Saint (May 09)
- <Possible follow-ups>
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Rick Moy (May 10)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Martin Holste (May 10)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Seth Hall (May 10)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Paul Halliday (May 10)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Martin Holste (May 10)
- Re: NSS Labs : CheckPoint 97.3% recommended profile hoax ? Crusty Saint (May 11)