RE: False-negatives in several Vulnerability Assessment tools

["Craig H. Rowland" <crowland () cisco com>]

> > My current employer, which is a Fortune 10 company, shall be
> > referred to as "Ralph Co."  I've been with Ralph Co for 2 years now. 
> > Our security there is relatively pathetic.  I have had to go to 
> > upper managment because our security manager will run a scan at 
> > random and decide a given service needs to be terminated because the 
> > scanning tool that he's demo-ing that week says that it's a 
> > "critical vulnerablity".  I have had to try to explain to him 
> > several times that he pays us a lot of money to exercise our 
> > professional judegement in verifying what is and is not a real 
> > vulerablity.  His answer is that "The tool says so, so it must be."
>
> The nadir of this process was him insisting that we shut down a "Code 
> Red Infected Server".  Too bad it turned to out be a developers Apple 
> iBook.
>
> My point with all this is what you do with the scans AFTER you run 
> them.  If you want intelligent analysis of the report, you get a 
> security professional that knows how to check things manually and 
> knows when output from the scanner looks dubious.  Any reasonably 
> intelligent person can operate the scanner software and print out the 
> report when its done.  The skill and expertise comes in interpreting 
> the output and making meaningful suggestions that actually improve 
> security.

Exactly. When you go to the hospital for a broken bone you have a X-Ray
technician operate the machine, and an experienced radiologist who
interprets the results. They don't simply hand you the X-Ray for
personal interpretation and the bill. 

This is an important point that is frequently overlooked. I've seen a
number of audits that were paid for by customers and consisted of
nothing more than a nicely bound printout of a commercial scanner with
almost no interpretation. Personally, I think this is a serious breach
of responsibility. 

The results of a scanner can be misleading if you don't have a good
knowledge of common vulnerabilities, commonly affected hosts, and
patterns indicating misuse. Expecting a scanner alone to identify 100%
of all threats is not practical for several reasons:

1) The author of the vulnerability check may have written it
incorrectly. Or, more likely, it worked in their testlab environment but
failed out in the field for a variety of reasons.

2) Performing an exhaustive scan against all the systems in a large
enterprise is usually not feasible due to network constraints, stability
of the backbone and scanned systems, and the dynamic nature of network
deployments (wireless, DHCP, etc.).

3) The scanner does not have an internal view of the host being audited
and can miss critical mis-configurations that result in an insecure
setup, but appear "secure" from the outside with automation.

I guess my point in all this is that proper interpretation of security
tool results is critical. As much as the security industry would like to
have the software do everything for the inexperienced user, it just
isn't practical or advisable given the nature and seriousness of this
business. 

-- Craig

Opinions are my own. There is no endorsement of the (random)
advertisement appended to this message.


---------------------------------------------------------------------------
Attend Black Hat Briefings & Training Europe, May 12-15 in Amsterdam, the 
world's premier event for IT and network security experts.  The two-day 
Training features 6 hand-on courses on May 12-13 taught by professionals.  
The two-day Briefings on May 14-15 features 24 top speakers with no vendor 
sales pitches.  Deadline for the best rates is April 25.  Register today to 
ensure your place.  http://www.securityfocus.com/BlackHat-pen-test 
----------------------------------------------------------------------------