Re: ISS Internet Scanner Cannot be relied upon for conclusive

[coley () LINUS MITRE ORG (Steven M. Christey)]

Interesting discussion...  I understand why various checks won't
always be accurate, and may be subject to false positives and false
negatives.

One of the problems I have with the scanners I've examined, however,
is that it would be easier to perform an overall risk assessment if
the tools are explicit with some sort of confidence measurement in
each of their tests.  Some checks may have descriptive text that say
"this test is not always reliable," but I don't necessarily see that
in all the checks that have reliability problems.  Considering the
hundreds of checks that most tools have nowadays, it's generally
infeasible to hunt through all that descriptive text anyway.  A
"reliability" field for each check, just like a Risk field, could help
me know what results are accurate and what results are questionable.
By attaching the Reliability to the check itself as opposed to each
individual result for each host, you could avoid the ensuing mountain
of data as described by David.

I don't mean to turn this into a public wish list, but another thing
I'd like to see in auditing tools is a "methodology" description for
each check - do you just read banners and compare to known vulnerable
versions, or do you perform the exploit?  That has some implications
on reliability.  Also, some checks can have a negative impact on the
systems they probe, and yet they aren't labeled "denial of service"
(nor should they be; but consider tests that could dump core in /, for
example, which may accidentally cause a denial of service).  If you're
a consultant and you're auditing a network with skittish managers who
don't want any adverse impact on their machines at all, knowing the
methodology for each check would help you construct an appropriate
scan.  (The typical "Light," "Medium," and "Heavy" scan configurations
provided by some tools may be too coarse in some cases and would
require a lot of manual effort to review all the checks).

Reliability and Methodology fields, each with say 3-5 unique values,
could be effective in informing the user of the inherent limitations
in a particular scanner and/or all scanners.

This is my two cents, not MITRE's.

- Steve
Steven Christey        | coley () linus mitre org   (781)271-3961
The MITRE Corporation  | "I'm not really sure what the question is, but
202 Burlington Road    |  I think the answer is 'I don't know.'"
Bedford, MA 01730      |     - an anonymous but honest former MITRE employee