Re: Penetration testing via shrinkware

[Ted Doty <ted () iss net>]

At 11:19 PM 9/19/98 -0400, Paul D. Robertson wrote:

>                                   While it isn't 100% foolproof, there's 
> a lot to be learned from a B2 evaluation.  Security modeling, code 
> walk-throughs, secure development methodologies, they all have their 
> place if you're going to build assurance.  "After-the-fact" testing is 
> always _much_ more blind than during "construction" testing.  Just as 
> crystal boxes tend to be better than black boxes in that regard.  

ObDisclaimer: I survived two B1/2 evaluations (actually one Orange Book and
one ITSec), and I build a scanner.

The biggest problem in the evaluations was the quality of the evaluation
teams.  Teams with high caliber members, which operate together as a stable
team for extended periods might be effective.  What I saw was that you
could count on neither.

The upshot is that Orange Book methods can probably only be applied for
products which place a premium on reliability - for example, medical
applications.  These systems will always be more expensive if developed
under TCSec guidelines, and they will be upgraded with new features more
slowly.  This argues pretty strongly for less formal methods, such as peer
review, for most products.

I have never seen any data that suggest that formally-evaluated systems are
(much, if any) higher quality than non-evaluated systems.  If you make the
assumption that each of the Orange Book implementations of {security
modeling, code walk-throughs, secure development methodologies) will only
catch a portion of the defects, the industry may be better served by a less
formal/structured analysis combined with black box analysis.  By "better
served" I mean less expensive, easier to use products that provide roughly
equivalent levels of protection.

Then again, maybe not.  However, I won't be taking my scanner anywhere near
Orange Book until this is a *lot* more clear.  Your mileage, as always, may
vary.

- Ted

-----------------------------------------------------------------------
Ted Doty, Internet Security Systems          | Phone: +1 678 443-6000
6600 Peachtree Dunwoody Road, 300 Embassy Row | Fax:   +1 678 443-6479
Atlanta, GA 30328  USA                       | Web: http://www.iss.net
-----------------------------------------------------------------------
PGP key fingerprint: 362A EAC7 9E08 1689  FD0F E625 D525 E1BE