RE: Re: I love the smell of whining in the morning...

[Greg Shipley <gshipley () neohapsis com>]

A few thoughts for whatever they are worth:

While I agree that deploying products in "real-world" environments is 
valuable, I think there is a lot that can be learned by doing tests in 
controlled environments as well.

Both are valuable IMO.

Admittedly it's been years since I formally tested any of these 
products myself, but in the decade or so of testing I *DID* perform I 
have run tests in both real-world scenarios (e.g. The Network 
Computing bake-off we did at DePaul University back in 2001) and lab 
tests (e.g. subsequent Network Computing tests and the later OSEC 
tests from 2001-2004).  I - and arguably "we" - have learned from 
both.

It has been my experience that if you design the test criteria and 
methods properly you can emulate *most* real-world scenarios.  For 
example, one of the side-effects we found during the DePaul testing 
back in 2001 was that not all management protocols were effective in 
bandwidth constrained environments; we had placed the sensors down at 
the University but the management platforms were back in our lab 
connected via a VPN tunnel.  Bandwidth was constrained and that 
created problems for *some* of the access methods.

Would we have designed a test for that situation before experiencing 
it at DePaul?  No.  Could we emulate and test that scenario now that 
we know about it?  Certainly.

There are drawbacks to real-world testing, too.  For example, we once 
ran into a potential hash collision problem in a firewall vendor's 
state table management process that we suspect would only creep up in 
limited situations - large VOIP deployments being one of them.  So if 
you did a test in a real-world environment that didn't include VOIP 
you probably wouldn't have ever come across the problem...and then 
been hosed if you suddenly implemented VOIP and had it going through 
this firewall.

How did we find that problem?  Synthetic traffic generation.  The 
chances of us stumbling upon that otherwise were slim to none...

So yeah - there's value in each.

With this latest round of NSS tests, I have yet to see the report (I 
don't have an extra $1800 sitting around right now!) but the posted v6 
methodology and approach looks pretty solid IMO.  If there were 
specific flaws in the testing or process I'd like to hear the 
complaints specifically, as from my vantage point right now it appears 
to be a reasonable approach.

Regardless, like any product evaluation process I think it's valuable 
for the reader to:

- consider the criteria and methodology
- consider the author(s) history, reputation, and funding model
- take the results as a single data point
- consider how the results apply to them
- ...come to their own conclusion

Finally - and I'm probably not going to make new friends in saying 
this - but if we think any signature-based approach is going to do 
anything other than identify really low-hanging fruit we're fooling 
ourselves.

If, for example, we integrated custom web applications into the 
vulnerability / exploitation test mix we'd probably see a big 0%.  
Working for a firm that has actually done the IR work for some of the 
breaches we're reading about in the news I can tell you 
authoritatively that the exploitation of web application vulns and the 
use of custom (often packed) tools will sail past most (all?) of the 
commercial IDS/IPS products.  Relevant?  Depends on who you are, what 
you are doing, what data you're protecting...but I digress...

I will say that seeing identification percentages for KNOWN, STOCK 
VULNERABILITIES fall below the 50% line is...well...depressing.  
Makes you wonder if it's worth investing in the technology, period...

My .02,

-Greg


-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate 
on your web server, you can securely collect sensitive information online, and increase business by giving your 
customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194