Re: Penetration testing via shrinkware

["tqbf" <ashland () pobox com>]

> TBQF observes that I have a mis-conception about scanners, asserting that a
> scanner's stated purpose is to scan for a finite list of bugs, not all
> possible bugs.  Fair enough, if that is what is meant by "verifying" a
> scanner, then I agree that it is theoretically possible to achieve
> verification that a scanner can reliably detect a finite list of bugs.  It
> just makes the idea of verifying a scanner a whole lot less interesting.

We're not talking about verification of scanners. We're talking about
scanners themselves. Scanners are not designed to detect all possible
bugs. That would be an unrealistic design goal. Scanners are designed to
detect a finite number of bugs. That is what a scanner does. 

Verification of a scanner involves ensuring that the scanner reliably
detects all the bugs in that finite list. I do not understand what else
one could "verify" about a piece of software other than that it performs
the task it was designed to do, without errors. 

Note also that the fact that there are a finite number of bugs to check
for does NOT mean that scanner verification is simple (or even practically
possible). Recall that there are two accuracy problems with scanner
software --- false negatives, where the scanner fails to report the
presence of an exploitable bug that it was designed to detect, and false
positives, where the scanner reports spurious invalid vulnerabilities. It
is much easier to address the first problem than the second problem, and
given the number of different operating environments deployed on modern
networks, it is impractical to exhaustively test vulnerability tests for
this problem.

Of course, this whole argument is simply a matter of semantics. You could
respond to my assertions by saying "a firewall is simply a device designed
to enforce a specific network access control policy" --- ie, "this
firewall is designed to block TCP port 111", and we can verify that
behavior much more readily than we can verify how well the firewall meets
the more vague design goal of "stop attacks". 

-----------------------------------------------------------------------------
Thomas H. Ptacek                          Network Security Research Team, NAI
-----------------------------------------------------------------------------
                                 "If you're so special, why aren't you dead?"