Re: MISSI X31 results

[Alfred Huger <ahuger () silence secnet com>]

On Tue, 7 Oct 1997, Frederick M Avolio wrote:

> > I have to agree with you. The work I have seen from the X31 group far
> > outstrips that of other Firewall testing 'authorities' I have seen. I
> > admit that this surprised me given that the X31 team is gov't based and I
> > tend to hold a dim view of such agencies and computer security. 
>
>
> Is this to say, for example, that you think the X31 tests mean more than
> NCSA's or is worth more? They seem fairly similar, 

For me at least, the testing done by the X31 group is worth more than the
testing done by the NCSA. For a number of reasons, the primary being that
I can see precisely what was done to test the firewall. From reading the
NCSA literature I see their testing methodoligy is indeed similar.
However, what I read were outlines of testing semantics. What I saw in the
MISSI reports were very detailed reports of their procedures. 

I cannot seem to find the actual reports for each firewall tested by the
NCSA, yet I can see the entire procedure for each firewall with the X31
group. This allows me to see the shortcomings of a firewall as well as the
strong points. I prefer this over a carte blanche stamp of approval from
the NCSA.


> except that NCSA has
> provisions for routine and on-going testing (so that the test results
> aren't 18 months old, for example).

I put very little weight behind this. I have seen current production
firewalls with problems the NCSA should have found. When I say current, I
mean within the last few months. I have recently seen application level
firewalls vulnerable to both SYN flooding and TCP sequence prediction. 
Beyond this, I have seen router based firewalls which seem to improperly
block source routed packets as well as filtering devices which are trivial
to knock over (although they do FAIL CLOSE). *All* of these products were
NCSA certified. All of these problems should have been easy to find,
provided you were not using canned security checks. These particular flaws
needed some envelope pushing to be discovered, which IMO should have been
done by the NCSA.


/****************************************************************************
Alfred Huger                                    http://www.secnet.com/ballista
Project Director                                ahuger () secnet com
Secure Networks Inc. (SNI)
*****************************************************************************/