IDS mailing list archives
Testing Manifestos [WAS: OSEC, ICSA, Cats and Dogs, etc.]
From: Greg Shipley <gshipley () neohapsis com>
Date: Fri, 10 Jan 2003 00:39:55 -0600 (CST)
[Warning: this one is long, folks, my apologies in advance....] I've given a fair amount of thought in my "free time" to the recent OSEC / Intrusion-Prevention thread (and in particular, Marcus's post) and I wanted to pass on a few ramblings. I'm going to try to focus more on the philosophy behind testing here, and less on the names. I realize that much of my earlier post on testing efforts, specific certs, etc., could have come across as very "mud-slingy'ish," which honestly was not my intention (although I'm certainly guilty of sending some zingers once in a while). In addition, I think that the mentioning of vendor/company names really took the heat up a notch and detracted from the true points of the discussion. Ironically, *I* was triggered by this very scenario in the suggestion that OSEC was similar to what ICSA was doing, and while I still stand by my statement that OSEC is much different my knee-jerk reaction is a topic for social analysis in possibly another forum, at another time.... :) Second, Marcus, I appreciate hearing many of your views from the vendor's perspective. I've never worked for a product company, so while I can empathize with many of your positions, I probably can't relate to them at the same level as some of the other "vendor folks" on this list. Your opinions on such matters are obviously insightful, and I appreciate you sharing them. Thank you. However, with that said I invite you to consider my position as a public reviewer. I will humbly suggest that my world is a bit different then yours, with different drivers and different goals, and will attempt in this e-mail to explain why. Here goes... -------------- First, I think it is important that consumers identify that there are MULTIPLE forums in which products are reviewed, each with its own set of dynamics. The big points I think people should make note of are: A. Who funded the test? B. Who defined the criteria? C. Does the testing methodology make sense? D. Were the tests executed properly? E. Do the testers have a good reputation / have they *successfully* done this type of testing before? Taking a sub-set of answers to those questions, some of the more common combinations our industry sees: 1. Vendor performed benchmarking based on vendor-defined criteria: vendor does the tests, vendor publishes the results. 2. Vendor-sponsored, 3rd-party performed bake-offs based on vendor-defined criteria: vendor funds the tests, 3rd-party performs the tests, vendor or 3rd party publishes the results. 3. Reviews (bake-offs) in trade publications w/ light testing: funded by the trade publication, internal staff writes an article based on little or no real testing, trade publication publishes the results. 4. Vendor-sponsored, 3rd-party performed bake-offs based on 3rd-party defined criteria: vendor funds the tests, 3rd-party performs the tests, 3rd party publishes the results. 5. Reviews (bake-offs) in trade publications w/ heavy testing: funded by the trade publication, 3rd-party (or internal staff) perform the tests, trade publication publishes the results. ....the list goes on. My personal opinion is that anything that is a) totally vendor sponsored, and b) has criteria that is defined solely by one vendor, is instantly suspect. In fact, I often wonder why ANYONE gives ANY weight to a test of a single product by a supposed "objective 3rd-party" when it is vendor sponsored, and vendor defined. I mean, come on - honestly, like we're going to see the report if the tests DON'T come out positive for that vendor? Right.... I'd like to believe the consumer base is a little smarter then this, but I digress... --------------------- Second, I found the following statements interesting: On Mon, 30 Dec 2002, Marcus J. Ranum wrote:
With respect to industry certification efforts - that's a tricker matter. The objective is to set a bar and continually raise it. It flat-out doesn't work if you start with the bar too high.
I would humbly suggest that this is entirely based on the motives for the "certification" effort. For example, if I'm launching a certification effort and I want EVERYONE to pass (or have a hope of passing), I will inevitably be reduced to lowest-common-denominator criteria definition. There simply isn't a way around this. In particular, if the participating vendors are driving the criteria, the least-functional, lowest-performing vendor will drag the whole thing down. So the "certifier" has a choice: exclude that bottom tier, or risk not achieving consensus. It's a tough spot. I'VE BEEN THERE. And in the case where you want everyone to participate, you are absolutely right: you set the bar low, and you continually inch it up. I will also suggest that this is one way of trying to nudge a particular product space forward, albeit very slowly, and with little effect over long periods of time. HOWEVER, I will *also* suggest that there is another approach: set the bar high, with the understanding that not everyone is going to achieve it. Those that do, have bragging rights to the areas that they achieve. Those that don't, well, don't. But the bottom line is that you don't HAVE to set the bar low. I will agree that most efforts have gone that way, however. I will let this list take it from here...
What I gather you're trying to do with OSEC is test stuff and find it lacking or not. Basically you want to say what products you think are good or bad - based on your idea (with input from customers and vendors) of good and bad.
Er, close, but no, and perhaps I am at fault for the confusion. What OSEC does is VERIFY certain aspects of a product. There are no GOOD and BAD ratings. The *test* is not pass/fail, the *criteria points* are pass/fail. For example in OSEC NIDS v1.0, test E7 uses "HTTP (500 Mbps, 536 MSS)" as background traffic, uses 10,000 addresses (which does some table flexing), at approximately 116k pps. HOWEVER, if a vendor doesn't market their product to run at 500+Mbps (and under similar traffic profiles), this test is irrelevant. Further, if a consumer doesn't need something to inspect traffic at those speeds, the test is irrelevant TO THEM. So while I will assume that these criteria points are relevant to SOME PEOPLE, the criteria is not based off of a "all of these points are good" or "if you don't pass this, you suck" mentality. It's far more intelligent then that. And *unlike*, say, a Network Computing article, there are no "soft analysis" angles to these tests - they are simply sensor tests, and they only measure/verify SOME of the many components to NIDS solutions.
Of course, if I were a vendor, I'd skewer you as publicly and often as possible for any bias I could assign you. Because your approach is inherently confrontational.
"Confrontational" - that's an interesting word to choose. Does the approach validate on a yes/no level? Yup - I would argue that's what performing good testing is about. So if that's what you mean by confrontational, um, yeah, absolutely - it's "confrontational."
Back when I worked at an IDS vendor, I tried to talk our marketing department out of participating in your reviews because, frankly, the vendors are forced to live or die based on your _opinion_. That's nice, but we've seen before that opinions of how to design a product may vary. Many industry expert types "Don't Get This" important aspect: products are often the way they are because their designers believe that's how they should be. Later the designers defend and support those aspects of their designs because that's how they believe their products should be - not simply out of convenience. The egg really DOES sometimes come before the chicken. :)
There is a smile on my face, as I now sense that I've been doing my job. :) So many things to respond to in this one.... For starters, I typically do not try to review products based on how they are designed. If I've mislead people in the past on this point, that's my bad, and I need to correct this. But honestly, I don't really care that much how products are designed - I care about how effectively they address the needs of their customers. IMHO, my job, as a reviewer, is to objectively test products and use quantitative results to come to qualitative conclusions. I could be wrong here, but I believe readers follow publications like Network Computing because the authors *DO* have opinions. They *do* rate things based on a given criteria, and offer readers advice on purchasing. The final opinions are, absolutely, opinions - but they are based on criteria, and hard, objective testing - not some engineering feat that someone pulled off with a piece of silicon. (Although that can be cool, and we will write about it!) As a side note, as a CTO (no, it's not just a title) of a consulting firm I am tasked with keeping an eye on numerous production systems with 3 Internet points-of-presence spread across 3 (US) states. I *am* a consumer of security technology, so while I do have my opinions, they are also based on very real and tangible business needs. While my needs aren't the same as those of some of our F500 customers, they are often similar. This also factors into the criteria I/we use. I am a consumer, and I work with consumers, too. Finally, I'm not sure that the vendors "live and die" by an opinion - it's just one opinion. They live and die by successfully running a company and meeting client needs. Taking a slight tangent, there is a scene in the movie "The Contender" where a senior Congressman advises the junior Representative to "cross out the word objectivity." "Your constituents want you for your opinion, for your philosophy, for your subjectivity." As odd as this may sound, I think much of that applies to a good product reviewer...as long as he or she is clear why/where they have gathered their opinions from. (Side note: I highly encourage the watching of "The Contender" back-to-back with "The Big Lebowski." There is something about watching Jeff Bridges playing the President of the United States and The Dude, in the same day, that just warms my heart...)
So here's the problem: how do you test a product without holding any subjective beliefs in your criteria? Man, that's hard. I wish you luck. (By the way, I've noted a very strong subjective preference on your part to Open Source solutions, most notably Snort. You've consistently cast their failures in kinder light than anyone else's, etc... So be careful...)
Ok, now this just cracks me up. ISS RealSecure wins a product review in like 1999, and I get accused (on this list even!) of being biased towards ISS. Years later Enterasys Dragon "wins," and, gee, now Greg is biased towards Dragon. And we've got Cisco gear here at Neohapsis! Man, those Neohapsis guys are in bed with Cisco!! Yeah! Yeah! Look at all of that gear! I can't win. I find it particularly amusing at how "biased" I am...but Snort? Snort??!?!? I mean, sure, I like Snort as much as the next NIDS, but heck, Snort hasn't even "won" any past Network Computing reviews. So ya totally lost me on this one....where did this come from? Another side note: Sometimes I think people confuse "objective" with "not having an opinion." I had a rep from a vendor e-mail me once because they were told by somebody that I supposedly said something negative about their product. I was asked "I thought you were vendor neutral?" I may not be in bed with any vendors, but I am entitled to have opinions...sheezus....but I digress again....
behind the scenes. I was pretty impressed. The thing that impressed me the most was getting a bit of the inside skinny on how many vendors passed the test the first time (many have failed DOZENS of times) and I thought that was cool. Obviously, it'd be best if all products going into the test were perfect going in. But I'd be happy, as a vendor or a customer, if they were BETTER coming out.
Agreed - and I fear that this is the point that got lost in my last e-mail: they *are* different. So are the NSS tests/results. But most of them have some value. It all depends on what you want to get out of the "results."
So, I think there's a place for *ALL* these different tests and it's a bad idea to throw mud at any of them.
Agreed, good point, and I apologize for confusing things. However, I do think it is within our rights as professionals to point out misleading tests, misleading results, and things that are generally just not on the mark. Unfortunately I think there are more bad testing efforts than good ones, primarily as a result of what you pointed out earlier: it is HARD to do this stuff right. But on this front, I *am*, absolutely, biased. :)
Honestly, I think that a smart customer should do their own. It's flat-out INSANE to spend more than $100,000 on a product without doing an operational pilot of it and 2 competitors. Yet, many companies do exactly that. They get what they deserve.
*nod* Another great point: pilot efforts are essential. Good thread...but I'm biased. :) I hope some of this is useful. White-Russians optional, -Greg
Current thread:
- Testing Manifestos [WAS: OSEC, ICSA, Cats and Dogs, etc.] Greg Shipley (Jan 11)