OSEC [WAS: Re: Intrusion Prevention]

[Greg Shipley <gshipley () neohapsis com>]

On Wed, 25 Dec 2002, Rick Williams wrote:

> [snip]... I will be beginning evals of IntruVert soon, with NetScreen IDP to
> follow. For functionality ("speeds and feeds") criteria, I am relying
> heavily on OSEC, because the Neohapsis crew knows their stuff and nothing is
> hidden .... [snip]
>
> I like the NeoHapsis guys too, but the OSEC stuff is very like the ICSA
> certification for firewalls, etc - you get your checklist and your
> "PASS/FAIL" mark - "Just Another Certification Scheme"

Please allow me to take a moment to clarify something. OSEC tests are
positively, absolutely NOT, at all, like ICSA tests. OSEC CERTAINLY is
*not* "just another certification scheme."  While it is true that the
testing results are pass/fail, that's about where the similarities end.
Please allow me to explain:

Over the past six years Neohapsis Labs has been testing products in the
security space, with the vast majority of our results appearing in Network
Computing magazine. Year after year we learn from our successes, and
mistakes, and roll that knowledge into our ever evolving testing
methodologies.  We tend to be leaders in this regard.  For example. the
careful reader will note that our documented testing methods in 1999
weren't mirrored by others until around 2001, and that our present-day
methods are quite a bit beyond what anyone else has done, to-date.

Year after year we're approached by vendors wanting to leverage our
reputation for their marketing material, and year after year we refuse to
engage in these so-called paid bake-offs.  We've built a reputation in
this industry for Doing Things Right, and we're not about to tarnish that
for some cheesy testing and a quick buck.

However, with that said we've noticed over time that a) various industry
"certification" efforts were watered-down b) many of those efforts were
more or less irrelevant, and c) there as a definite need in the industry
for a trusted 3rd party to validate vendor claims with REAL testing.

Enter: the Open Security Evaluation Criteria (OSEC).

----------------------

OSEC NIDS criteria (BTW, the firewall criteria is coming) are based on our
(Neohapsis) experiences, consumer concerns, and vendor input.  It's
important to note that ALL THREE of those entities helped drive the
criteria, and it took me close to twelve months to get it all together for
the NIDS space.  (Anyone remember me rambling about standardized testing,
on this very list, last year?  OSEC is the result of that rambling...)

OSEC sets the bar up a few notches, focusing on the highest common
denominator, not the lowest. (*BIG* difference over, er, other efforts)
But don't take my word for it, take a look at it yourself:

http://osec.neohapsis.com/criteria/nids-v1/testsummary.html

That's v1.0 of the NIDS criteria, with close to 80 tests in 7-core areas.
Now, I haven't finished the NSS report yet, but I guarantee you that they
didn't verify all of those areas - but more on this later.  In addition to
those tests, OSEC profiles products and components, arming consumers with
vulnerability profiles, as well.  It continues to amaze me how many
vendors push out security products with vulnerable versions of underlying
components (i.e. OpenSSH, OpenSSL, bad TCP ISN generation, etc.).  OSEC's
criteria also takes this into consideration, and these results are public,
as well.

Side note: Where are these tests in other "certification" efforts?
(Answer: they don't exist)

Further, OSEC has broken out all of its tests in a pass/fail manner so
that consumers can view the areas that they care about, and ignore the
ones that they don't.  For example, if a consumer doesn't care about
"Complex IP Fragmentation (ordered 8-byte fragments, marked last frag
first)" then a "N/A" or "fail" in that category is of little consequence.
Same goes for speed.  Our industry publicly appears to be extremely caught
up in "who can go the fastest" games, but privately I know consumers ARE
cost-conscious.  The price/performance ratio IS a big deal when it comes
to purchasing.  In a down economy, a Porsche might be nice but not if you
just need a vehicle for under $20k to get you to and from the train
station...you're going to want something more cost-effective.  This is why
the OSEC tests are broken out into speed ranges, and you can see the price
tags - it's not an "all or nothing"  scenario.  (We're hoping to see
100Mbps and 500Mbps verified testing results in the coming months, not
just Gbps, for example...)

And we *expect* products to fail some areas.

Rick, you identified that the NSS tests are a bit different, and IMHO they
are.  Readers on this list should look at both OSEC results and NSS
results in a complimentary manner, but this is not an "either/or"
scenario, and OSEC covers things the NSS reports don't (and vice-versa).
Much of what you will find in traditional NSS reports and articles in
publications like Network Computing are some of what I call "soft
analysis," which is definitely needed and very helpful.  However, today's
OSEC tests are based on "hard analysis" / hard-testing, using a range of
tests.  OSEC doesn't cover everything, it just does a damn thorough job at
the areas it does cover.  (And it does cover quite a few areas.)

Finally, IMNSHO comparing OSEC criteria to ICSA criteria is akin to
comparing a Formula-1 racer to, say, a garbage truck.  Both have some
degree of engineering behind them, but serve very different purposes.
When consumers see that a product has gone through OSEC verification, they
know that one of the best test testing labs in the world has put that
product through one of the most rigorous criteria sets in the world, and
unlike other watered-down test sets, OSEC results MEAN SOMETHING.

In short, I dare say that the industry has not seen anything like OSEC
before.  And we're just getting warmed up - it will only get more
interesting from here.

For whatever it's worth,

-Greg

P.S. Don't take my word for all of the above, see for yourself:
http://osec.neohapsis.com - it's...all...right..there.