Re[2]: Penetration Tests

[Frank Willoughby <frankw () in net>]

Edward,

Testing firewalls is a very complex undertaking.  Being thorough is 
perhaps the most valuable asset you will have.

There are some basic kinds of questions which need to answered.  

o Are you trying to establish a firewall test lab or do you just want
   to find out which firewall is the best for your company?

   The answer to this question determines the type of testing that you
    will be performing.  Testing for a firewall test lab goes way beyond 
    what a someone will do for their company.  When someone is looking 
    for a firewall for their company, they will start a sifting process 
    which should take a couple of months (gathering data, verifying claims,
    etc.).  Much of the initial sifting process can be done on paper without
    having to take firewalls apart.  When you have a set of 3-5 firewalls 
    which meet your basic criteria, then start take the firewalls out for
    a test drive.  This will eliminate another one or two.  Then start the 
    testing process.

o "Joe at Company A uses brand X firewall.  He likes it and recommends it
   very highly.  I should use it too, right?  Maybe, maybe not.  First, 
   Joe's comments are hearsay.  Second, Joe's experience with firewalls
   may be limited.  Third, and most important: What works well for Joe,
   may be a complete disaster for your company.

   Every company has unique business and security requirements.  A firewall
   is an implementation of a security policy which is based on these 
   requirements.  Putting the NSA's security posture into a university will
   bankrupt the university very quickly.  Putting a university's security
   posture (of an open environment) into the NSA is a recipe for a national
   security disaster.  Determine in advance what you need and choose your
   firewall accordingly.  Choose wisely.

o A firewall is an implementation of a security policy.  Having the policy
   will help define the firewall's rules as well as deal with legal and
   non-compliance issues.

o How much time are you ready to spend testing?
   A "network scan" of a firewall for vulnerabilities can take as little 
    as 5-15 minutes using commonly available commercial products which 
    were mentioned in other's postings.

   A thorough firewall test takes 1-2 months (minimum).  It is extremely
    time-consuming to test a firewall and do it right.  (And we haven't
    even gotten to the report-writing)  8^(

   As in the testing of CPU chips, complete testing coverage isn't practical
   or even feasible.  You have to do the best you can in the time allowed.
   
o What is your methodology?
   Before you start testing, you should first map out your firewall test
   methodology.  If you are looking for a starting point, you might check
   out www.fortified.com which has a Free Firewall Evaluation Checklist.
   The Checklist is available via HTTP only.  While it is primarily 
   designed to help people who are evaluating firewalls, it may give 
   you an idea of some things you might want to test.

o What should I test for?
   o Vulnerabilities   Most people test for vulnerabilities.  If it passes 
       all of the tests, then it must be OK.  Right?  Not really.  Testing 
       of a firewall should be *very* comprehensive and go way beyond looking 
       for vulnerabilities.  A firewall's ability to pass vulnerability tests 
       may or may not be a good indicator of how robust the firewall really 
       is.  It could mean that the firewall has a very robust architecture 
       and it is not vulnerable against the attacks you tried.  It could also 
       mean that the firewall's architecture is not quite up to speed and
that 
       the  vendor is very fast in generating patches for their product. Both
       appear to produce the same results.  Looking at the firewall in
detail        will help determine what is really going on.
    
   o Functionality - does the firewall do the things it is supposed to do?
   o Gotchas - does the firewall do the things that it is not supposed to do?
   o Verification of claims - does the firewall really do all of the 
      things that the vendor says it can?  This different than the Gotchas
      or Functionality testing mentioned above
   o Documentation
   o How easy/difficult it is to configure the rules
   o Tech Support
   o History of the company
   o Etc., etc.

o What about firewall "certification"?
   Some organizations will wave a scanning tool across the firewall and
   "certify" it if it passes all of the tests.  One in particular comes
   to mind.  In this particular case, I am not aware of any firewalls 
   which failed to be certified.  Most of the "certified" firewalls would 
   have not made it past the initial sifting process of evaluating firewalls.
   This doesn't mean that they may be bad firewalls.  It only means that I 
   don't consider them robust enough to recommend or use for my purposes. 
   YMMV, of course.

   I've discovered problems in every firewall I ever tested.  So have 
   other professionals on this list.  Although most problems are minor, 
   some have been rather severe ("show-stoppers").  (Please don't bother 
   to ask which ones I have tested, or which ones have had problems.)

   There is no such thing as a perfect firewall.  Some are better than 
   others in different areas.  You really have to look at the whole 
   picture.  As vendors tend to leapfrog each other in terms of 
   technology, the test criteria get updated frequently.  

   Also, there is no "one size fits all" when it comes to firewalls.  
   What works for you may not necessarily work for someone else.

   My personal opinion is that firewall certification says nothing and 
   proves nothing.  It is a nice marketing tool and tends to make a lot 
   of money for those who are performing the testing.  Let me give you a 
   couple of examples:
   o Suppose you have a firewall which passes every test you can think of?
     What about the tests that you haven't thought of, (but the hackers will 
      or have)?
   o Hypothetically speaking, suppose you have a bullet-proof firewall 
      which is impervious to every possible vulnerability.  Unfortunately,
      when the firewall is installed, it should is installed incorrectly.
      Instead of being protected from the risks of the Internet, the company
      now has more exposures than before - perhaps enough to bankrupt the 
      company.
   o Who has failed, and for what problems?  If no firewalls ever failed 
      the testing, then how valid is the testing methodology really?
   o What about the legal liabilities if a "certified" firewall is 
      penetrated by an attacker?  If the tester is going to certify
      something, they should also be capable of backing up their claims
      that the product performs as it should.  What are the legal 
      liabilities for the tester if the firewall is known to be vulnerable 
      to certain types of attacks & the tester passes it anyway?

   For all of the reasons above and more, I'll never certify firewalls or 
   other security products.


I hope the above has been of some help to you.

Best Regards,


Frank
The opinions of the author of this mail may not necessarily be 
representative of the opinions of Fortifed Networks, Inc.

Fortified Networks, Inc. - http://www.fortified.com/
Expert (vendor-neutral) Computer and Network Security Consulting
Phone: (317) 573-0800     Fax: (317) 573-0817

Any sufficiently advanced bug is indistinguishable from a feature.
-- Rich Kulawiec