Re: your mail -Reply

[Laris Benkis <lbenkis () bank-banque-canada ca>]

So,... don't keep us all in suspense.  While I find interesting your
assertion, and the reasoning behind it, that scanners are the wrong tool
to test firewalls, I am more interested in what the right tools are.  Who
makes these tools and what do they test?  If they don't exist currently,
and have to be home-grown, what specific tests should they perform? 

You make some very strong and informed arguments against what was
written in the article, and I don't necessarily think you are wrong in what
you said, but I think it is too easy to simply criticize their work without
providing an alternative methodology that would meet your criteria for an
acceptable test.
 
Laris

> > > <tqbf () pobox com> 06/30/98 04:01am >>>
> This is incorrect. The methodology also included an extensive
evaluation of
> firewall management and performance. 

This discussion is about whether NT Firewalls in general and MS Proxy
Server 2.0 in particular are secure. I don't care to discuss how much
faster or easier to manage any given firewall is. That isn't my business.

> And note that we did put all kinds of
> caveats telling readers that they *should* think critically about security
> test results.

A quote from the intro paragraphs of the article:

        Get a bunch of security gurus together and turn
        the talk to NT-based firewalls.
        Then sit back and enjoy the show. "Not ready for
        the enterprise," they'll sniff. "Not as
        manageable as Unix utilities." "Not secure." 

        Is that so? Data Communications and National
        Software Testing Laboratories Inc. (NSTL, Conshohocken, 
        Pa.) just wrapped up an exhaustive test of NT firewalls.
        And we've got one thing to say to the experts:
        "Bull." 

        How can we be so sure? We bombarded seven
        top-selling NT firewalls with nearly 300 forms of
        attack-without finding any significant security
        loopholes. (Past tests, including those of Unix
        products, turned up dozens of flaws.) What's
        more, these firewalls do an excellent job of
        locking down potential vulnerabilities in
        Windows NT itself. 

Now, I don't consider myself an expert, but I know one thing that
definitely IS bull, and that's the idea that you can assess the security
of a firewall software platform and compare it to other firewalls by
running a network scanner up against it. This is a strong introduction to
the article, and nowhere in it is a caveat about the validity of the
tests. There is a caveat later

        ...Our security tests do not in any
        way certify these firewalls as safe. This
        evaluation involved the world's foremost
        experts on these products-the vendors that
        built them-configuring their own software to
        withstand well-known attacks in a carefully
        controlled lab environment. THAT'S A VALID 
        WAY OF DEMONSTRATING THE FIREWALL CODE DOES
        WHAT IT'S SUPOOSED TO. But it's not the same 
        as saying these firewalls are absolutely
        secure...

[Emphasis mine]

Running a network scanner against a firewall is a valid test of the
reliability of the code? Exactly why do you think this? The output of a
network scanner is a series of automated attacks, designed to assess
the
vulnerability of an end-system. Network scanners do not emit attacks
that
are designed to evade firewalls (with a very few exceptions); your test
didn't attempt 300 different firewall attacks (as your introduction
claims), but rather 300 plain-vanilla clusters of network traffic that any
packet filter 3 years ago could have handled.

> From the article, it seems that this is the extent of your security
testing methodology, which is what I mean from now on when I use the
term
"methodology". If this is the case, then your claims about the security of
NT firewalls are based solely on the output of a network security
scanner
that was not designed to assess firewall security. In case you're
wondering, I don't know of ANY network security scanner that really is
designed to assess firewalls.

Your caveats about the validity of your methodology involve stating that
"in the real world, new attacks are discovered and firewalls are
misconfigured, unlike in our test environment" (my paraphrasing). In the
real world, many old firewall attacks were left untried by your
methodology, because you relied entirely on a scanner that doesn't
attempt
them. In the lab room, misconfigurations are irrelevant; you're testing
the software, not it's deployment. So what are you warning about here?

Your test methodology seems to boil down to "throw the most obvious
possible attacks at the firewall, and make sure the firewall blocks them."
For example, when assessing the ability of firewalls to block "malicious
code" (ActiveX and Java applets):

        To see if products could block bad code, we set
        up a Web server with pages containing
        ActiveX and Java applets. Vendors were
        asked to configure their firewalls to deny
        access to these applets. Five could...

This is akin to saying "my firewall can block telnet connections, because
when I run "telnet" to connect to a protected address, it doesn't work."
So what? Attacks against properly configured firewalls don't involve
running normal networking commands --- they involve creating
pathologically 
complex streams of network traffic designed to confuse the firewall.

Likewise, an informed attack against hosts behind a firewall that utilized
ActiveX probably wouldn't involve simply sticking an applet on a web
page;
rather, it would involve trying to fool the vulnerable browser into
running an ActiveX control as a result of a web page that looked nothing
like an ActiveX control. Finding problems like this is the essence of
firewall security testing. What you did here was not a firewall security
test; it was a sanity check.

Surprise, you found that the firewalls you tested were sane. One would
assume, as you point out in your article, that vendors would have the
good
sense to spend a few hundred dollars on a scanner to make sure that
their
firewall performed well in magazine security reviews. What does this
say
about the actual security of a firewall? Almost nothing.

For an article that starts out with the premise that the firewall
"experts" of full of BS about the security of NT firewalls, your factual
support is awfully weak. 

-----------------------------------------------------------------------------
Thomas H. Ptacek                           SNI Labs, Network Associates, Inc.
-----------------------------------------------------------------------------
http://www.pobox.com/~tqbf       "If you're so special, why aren't you
dead?"
                                        
DISCLAIMERS:

I work for Network Associates. I do not speak for them.

Network Associates produces CyberCop Scanner, a competitor of ISS's
scanner. I do not believe CyberCop Scanner would be any more
appropriate
as the basis for a firewall test than ISS's tool is.

Network Associates also produces Gauntlet, an application-gateway
firewall. I have nothing to do with Gauntlet, but you can color your
opinions of my opinions as you see fit.