Industry test reports... (was RE: Proverbial appliance vs software based firewall)

["Marcus J. Ranum" <mjr () ranum com>]

Ryan, Kennedy wrote:
> Funny thing..  I'm right now looking at a Netscreen commissioned Tolly Group
> test report from March 2002, (no.202121) which amazingly (not) shows
> Netscreen outperforming Checkpoint/Nokia IP740 and Cisco 545 by leaps and
> bounds..    

Tolly group's tests are all bought and paid for. The vendor pays
to have a test of their own devising run, and Tolly reports the
results. Remember the big scandal from last year when Miercomm
published the results of their "test" of Intrusion.com's IDS?
The test was rigged (the protocol was suggested by Intrusion.com)
to yield completely ridiculous (and deliberately deceptive)
results. Typically, in a software company, it's the marketing
organization's budget that pays for the tests. :)  Then the testing
company carefully words the results so that they don't step on
their other customers that have _also_ paid for tests.

The thing to look for is when they run the SAME TEST protocol
every time and make up their own test protocol with input from
outsiders not beholden to the vendors. I know I used to bash
them 8 years ago or so, but ICSA labs is doing some really good
stuff with their firewall tests these days. But they're looking
at functionality, not "marketing numbers" - don't believe _ANY_
of the marketing number tests.

When you buy a car do you believe the manufacturer's claim
that it goes 0-60 in a "blazing" 6 seconds, or Car and Driver's
test results where they used their standard test methods and
got 0-60 in 8 seconds? ;)

mjr.
---
Marcus J. Ranum                         http://www.ranum.com
Computer and Communications Security    mjr () ranum com

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards