Re: Penetration testing via shrinkware

["Marcus J. Ranum" <mjr () nfr net>]

John McDermott wrote:
> [...] Just out of curiousity 
> does ICSA look at the source for certification?

Last I saw, the ICSA "certification" was based on a firewall's
ability to not have identified vulnerabilities in an ISS scan
and a couple of other "advanced" tests.

I was actually involved in the earliest phase of the ICSA (then
NCSA) firewall certification effort, and wound up burning a
few bridges over this issue. :) But if you look at their
documents you'll see that one of the other requirements is
a "functional profile" (my idea, has my copyright, too) :)
in which the vendor is expected to state their claims on paper
as to why they think the product is any good. It was an attempt
to move towards a minimal form of design review. Never went
anyplace:( the ensuing discussion went something like this:
mjr:    "Well, as long as you weren't planning on just running
        ISS against the product and blessing it, then I think
        this could be valuable!"
NCSA:   "Uh, uh..."
mjr:    "Oh, !@#(&!@#!" (exit stage left)

To my knowledge, ICSA never looks at source for anything.
They're really a consultancy disguised as a testing
agency, though, so they'd probably do it if you offered
them enough money. They charge $30,000 for the privilege
of having your firewall tested with ISS...

The only security evaluation I can think of that includes
code review is the NCSC (NSA) TPEP (orange book nonsense)
which is hideously slow and expensive. At least they get
some of it right:
        - first you look at the design documents
        - then you look at the code

It's actually pretty sad, virtually none of the vendors
make code accessible. TIS did when I was there (guess whose
idea that was?) and NFR does (guess where I work?) but
that's it. Earlier versions of some vendors' products
probably wouldn't have stood up to code review because
they contained ripped-off code from the firewall toolkit
(certain firewalls) or hacker toolkits (a certain scanner).

mjr.
--
Marcus J. Ranum, CEO, Network Flight Recorder, Inc.
work - http://www.nfr.net
home - http://www.clark.net/pub/mjr