Re: Penetration testing via shrinkware

["Ivan Arce,CORE SDI" <ivan () securenetworks com>]

On Sun, 20 Sep 1998, Adam Shostack wrote:

> On Sun, Sep 20, 1998 at 06:47:08AM +1000, Christopher Nicholls wrote:
> | At 12:44 AM 18/09/98 -0700, Crispin Cowan wrote:
> | >tqbf () pobox com wrote:
> | >I beg to differ.  A firewall can at least theoretically be verified:  if
> | it is 
> | >formally proven to enforce a policy of (say) allowing through traffic on
> | ports X
> | >and Y, and no others, then the firewall is verified.  A scanner, on the other
> | >hand, can never be verified, because the potential list of vulnerabilities
> | that
> | >it could reasonably be expected to check for is infinite.  Scanners can
> | never be
> | >complete, because the space of possible mis-configurations and buggy software
> | >knows no bounds.
> | 
> | True, but the same can be said for firewalls, in that there are always new
> | attack mechanisms being developed to defeat firewalls; so in a sense they
> | are never complete either. Certification of firewalls is usually
> | assurance-based; that is, verified to levels of asuusrance - such as the
> | Common-Criteria evaluations. This means that basically the certification
> | procedure checks and confirms what the manufacturers claim it can can do -
> | a security target. Maybe it would be possible to set a similar security
> | target for intrusion detection software and scanner software too?
>
>       The platonic-ideal firewall resists new attacks.   I don't
> believe that the ideal scanner finds new things.  Thus, a firewall
> that does not block a new attack in the class of things it is designed 
> to watch is broken.  This is the result of a deny everything stance.
> In practice, firewalls will fall short of their goal.  The question to 
> ask is how far and how often?

IMHO the platonic-ideal scanner detects all the bugs its designed to
detect without false positives disregarding a finite set of
factors ( net. topology, firewalls in between, NAT, operating
systems, configuration variations, os versions and interactions
between  those factors).
In that sense i believe its harder to certify a scanner than a firewall
*unless* a scope and framework for the certification procedure is
predefined.

OTOH a firewall must enforce a policy no matter what..

Ivan Arce
SecLabs @ NAI