Firewall Wizards mailing list archives
Re[2]: Penetration Tests
From: Frank Willoughby <frankw () in net>
Date: Mon, 29 Sep 1997 07:33:55 -0500
Edward, Testing firewalls is a very complex undertaking. Being thorough is perhaps the most valuable asset you will have. There are some basic kinds of questions which need to answered. o Are you trying to establish a firewall test lab or do you just want to find out which firewall is the best for your company? The answer to this question determines the type of testing that you will be performing. Testing for a firewall test lab goes way beyond what a someone will do for their company. When someone is looking for a firewall for their company, they will start a sifting process which should take a couple of months (gathering data, verifying claims, etc.). Much of the initial sifting process can be done on paper without having to take firewalls apart. When you have a set of 3-5 firewalls which meet your basic criteria, then start take the firewalls out for a test drive. This will eliminate another one or two. Then start the testing process. o "Joe at Company A uses brand X firewall. He likes it and recommends it very highly. I should use it too, right? Maybe, maybe not. First, Joe's comments are hearsay. Second, Joe's experience with firewalls may be limited. Third, and most important: What works well for Joe, may be a complete disaster for your company. Every company has unique business and security requirements. A firewall is an implementation of a security policy which is based on these requirements. Putting the NSA's security posture into a university will bankrupt the university very quickly. Putting a university's security posture (of an open environment) into the NSA is a recipe for a national security disaster. Determine in advance what you need and choose your firewall accordingly. Choose wisely. o A firewall is an implementation of a security policy. Having the policy will help define the firewall's rules as well as deal with legal and non-compliance issues. o How much time are you ready to spend testing? A "network scan" of a firewall for vulnerabilities can take as little as 5-15 minutes using commonly available commercial products which were mentioned in other's postings. A thorough firewall test takes 1-2 months (minimum). It is extremely time-consuming to test a firewall and do it right. (And we haven't even gotten to the report-writing) 8^( As in the testing of CPU chips, complete testing coverage isn't practical or even feasible. You have to do the best you can in the time allowed. o What is your methodology? Before you start testing, you should first map out your firewall test methodology. If you are looking for a starting point, you might check out www.fortified.com which has a Free Firewall Evaluation Checklist. The Checklist is available via HTTP only. While it is primarily designed to help people who are evaluating firewalls, it may give you an idea of some things you might want to test. o What should I test for? o Vulnerabilities Most people test for vulnerabilities. If it passes all of the tests, then it must be OK. Right? Not really. Testing of a firewall should be *very* comprehensive and go way beyond looking for vulnerabilities. A firewall's ability to pass vulnerability tests may or may not be a good indicator of how robust the firewall really is. It could mean that the firewall has a very robust architecture and it is not vulnerable against the attacks you tried. It could also mean that the firewall's architecture is not quite up to speed and that the vendor is very fast in generating patches for their product. Both appear to produce the same results. Looking at the firewall in detail will help determine what is really going on. o Functionality - does the firewall do the things it is supposed to do? o Gotchas - does the firewall do the things that it is not supposed to do? o Verification of claims - does the firewall really do all of the things that the vendor says it can? This different than the Gotchas or Functionality testing mentioned above o Documentation o How easy/difficult it is to configure the rules o Tech Support o History of the company o Etc., etc. o What about firewall "certification"? Some organizations will wave a scanning tool across the firewall and "certify" it if it passes all of the tests. One in particular comes to mind. In this particular case, I am not aware of any firewalls which failed to be certified. Most of the "certified" firewalls would have not made it past the initial sifting process of evaluating firewalls. This doesn't mean that they may be bad firewalls. It only means that I don't consider them robust enough to recommend or use for my purposes. YMMV, of course. I've discovered problems in every firewall I ever tested. So have other professionals on this list. Although most problems are minor, some have been rather severe ("show-stoppers"). (Please don't bother to ask which ones I have tested, or which ones have had problems.) There is no such thing as a perfect firewall. Some are better than others in different areas. You really have to look at the whole picture. As vendors tend to leapfrog each other in terms of technology, the test criteria get updated frequently. Also, there is no "one size fits all" when it comes to firewalls. What works for you may not necessarily work for someone else. My personal opinion is that firewall certification says nothing and proves nothing. It is a nice marketing tool and tends to make a lot of money for those who are performing the testing. Let me give you a couple of examples: o Suppose you have a firewall which passes every test you can think of? What about the tests that you haven't thought of, (but the hackers will or have)? o Hypothetically speaking, suppose you have a bullet-proof firewall which is impervious to every possible vulnerability. Unfortunately, when the firewall is installed, it should is installed incorrectly. Instead of being protected from the risks of the Internet, the company now has more exposures than before - perhaps enough to bankrupt the company. o Who has failed, and for what problems? If no firewalls ever failed the testing, then how valid is the testing methodology really? o What about the legal liabilities if a "certified" firewall is penetrated by an attacker? If the tester is going to certify something, they should also be capable of backing up their claims that the product performs as it should. What are the legal liabilities for the tester if the firewall is known to be vulnerable to certain types of attacks & the tester passes it anyway? For all of the reasons above and more, I'll never certify firewalls or other security products. I hope the above has been of some help to you. Best Regards, Frank The opinions of the author of this mail may not necessarily be representative of the opinions of Fortifed Networks, Inc. Fortified Networks, Inc. - http://www.fortified.com/ Expert (vendor-neutral) Computer and Network Security Consulting Phone: (317) 573-0800 Fax: (317) 573-0817 Any sufficiently advanced bug is indistinguishable from a feature. -- Rich Kulawiec
Current thread:
- Re: Penetration Tests, (continued)
- Re: Penetration Tests Bennett Todd (Sep 26)
- Policy ? (was RE: Penetration Tests) Capt Jim Bailey - SSG/SINS - DSN 596-6106 (Sep 26)
- Re: Policy ? (was RE: Penetration Tests) Edward Cracknell (Sep 29)
- Re: Policy ? (was RE: Penetration Tests) Bennett Todd (Sep 29)
- Re: Policy ? (was RE: Penetration Tests) Paul D. Robertson (Sep 30)
- Policy ? (was RE: Penetration Tests) Capt Jim Bailey - SSG/SINS - DSN 596-6106 (Sep 26)
- Re: Penetration Tests Bennett Todd (Sep 26)
- Re: Penetration Tests Darren Reed (Sep 26)
- Re[2]: Penetration Tests Edward Cracknell (Sep 26)
- Re: Penetration Tests -= ArkanoiD =- (Sep 26)
- Re: Penetration Tests Chuck Kenyon (Sep 26)
- Re: Penetration tests Bill Kennedy (Sep 26)
- Re[2]: Penetration Tests Frank Willoughby (Sep 29)
- RE: Penetration Tests Gary Crumrine (Sep 29)
- RE: Penetration Tests Andreas Siegert (Sep 30)