Penetration Testing mailing list archives
Re: Evaluating Pen Testers
From: Andre Gironda <andreg () gmail com>
Date: Mon, 12 Apr 2010 23:49:51 -0500
On Thu, Apr 8, 2010 at 9:18 PM, Daniel Kennedy <danielkennedy74 () gmail com> wrote:
There ought to be a "who's who of penetration testers, especially with some of what I read about and hear at conferences when it comes to penetration testing, for many years now, and its not getting any better. That said, it wouldn't be easy to put together. A firm in the UK was testing pen testers for a while, but their approach left some questions to be answered.
Are you referring to CHECK? They are still verifying penetration-testing capability at the company-level. I strongly discourage anyone from building a list of individuals; Microsoft and others have tried this before and the ethical consequences of these actions is somewhat revolting (to at least myself). It would be impossible to keep a list current because people come and go all of the time (at least by the hour).
#Confusion Many customers, and many security testers, confuse what is a vulnerability scan with a penetration test. A scan for vulnerabilities
I do not like the words "manual" or "automated". Not all penetration-testing activity can be automated, but a lot can. Not all application scanning activity should be automated. Humans HAVE to be involved. Vulnerability scan activity should not be automated without humans, but Qualys QG has convinced people (read: managers) that it can. Penetration-testers MUST be allowed to choose their own toolchain and decide which parts to automate and which parts to leave with some manual intervention.
can be a recon activity in a pen test, its valuable information, but its not a pen test. Pen tests involve exploitation (usually a
Please read [PDF] http://www.securityacts.com/securityacts02.pdf [PDF] http://www.net-security.org/dl/insecure/INSECURE-Mag-25.pdf before continuing...
non-damaging one like opening a shell or dropping a text file) reached under some rules of engagement. This doesn't suggest one is better than the other, frankly its completely dependent on what the client is hoping to accomplish.
I think I disagree with you. Penetration-testing can certainly stop before exploitation, assuming that something is found that is believed or known to be capable of exploitation.
#Standards The OSSTMM is an interesting project but its miles from being a standard where you can eliminate people that don't follow its methodology. It would be akin to saying you only accept software from CMM level 5 companies - the model is thorough but smart people raise legitimate objections to it.
The art of drafting a proper RFQ to potential penetration-testing consultants is a WIP by OWASP as seen here -- http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers&discussionID=17288173&gid=36874&commentID=14543002 Almost all RFQ analysis is followed by Case Study analysis and extremely high-quality References before hiring an application security consulting company. I recommend retaining in-house penetration-testing talent. I suggest letting them follow any penetration-testing methodology/standards which they choose, as long as it gets results. Most will choose all standards and no standards at the same time. Penetration-testers are very talented at being in two places at once given their paradoxical natures. Allow them to do this.
#Certification Alongside things people present as standards are certification. They tell you something about the person, namely that they are willing to take the time/cost to prove some level of proficiency in an area, but there are a great many security luminaries without any. The CEH is gaining some traction, not sure if that's a good or bad thing yet.
I think it's a bad thing. What does China use to certify their penetration-testing talent? If somebody has written some new code that steals some stuff that nobody has stolen before, then that should be certification enough. In other words, the first question in the interview should be "Which BackTrack tool did you write or contribute to?" and the second question should be "When was the last time that you spoke at an OWASP local chapter meeting?"
#Nessus As you say, one who runs a scan and hands you a Nessus report is not doing much. However Nessus is a sophisticated tool for vulnerability scanning, has a professional license model, and compares favorably to more expensive options. So you can't eliminate someone for using Nessus, only for only using Nessus. #Open Source Tools The suggestion that using open source tools reveals some lack of sophistication or worthiness is silly. I would rather have someone capable of making contributions to the Metasploit project, someone who understands what they're running and can do hand testing, then some bozo who just points Core Impact at my environment and hits 'go'.
I have no idea what you're talking about, but as I said before -- let the penetration-testers choose their own toolchain. There are plenty of badasses at Core that were instantly disrespected by your remark. Also see "automation" vs. "manual" above. Finally, if you want to know the differences between tools, read this -- http://stackoverflow.com/questions/72166/penetration-testing-tools/74513#74513
#Legal Considerations You should consider that if something goes wrong with a company you are essentially sharing confidential information with, whether you will have protection under the law. That usually means dealing with a firm or person who is legitimately 'filed' (has a background you can check) and using someone in your firm's country or a country where your familiar and comfortable with the legal environment in place. Further you might be more comfortable with folks from certain backgrounds (educationally, professionally, whatever), so check out linkedin or something similar.
Every application security consulting company has insurance to cover themselves. Here's an RFQ/RFP hint: Make sure the ones that you hire have insurance.
#Reputation Most companies that can provide value in pen testing have at least some names that will show up when you Google. They've been quoted in some article, done some presentation or talk, and so forth.
Sometimes people use pseudonyms or like to keep a low profile for personal reasons, so be careful with this one. Please do not hire people based on Google, their blog, or some claim of "specialty" or "generality". You need a formal RFQ/RFP process If you really need a starting point, check this out -- http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2 Cheers, Andre ------------------------------------------------------------------------ This list is sponsored by: Information Assurance Certification Review Board Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified. http://www.iacertification.org ------------------------------------------------------------------------
Current thread:
- Evaluating Pen Testers Daniel Kennedy (Apr 12)
- Re: Evaluating Pen Testers Stephen Mullins (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 14)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 15)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 15)
- Re: Evaluating Pen Testers Andre Gironda (Apr 19)
- Re: Evaluating Pen Testers Nathan Sportsman (Apr 20)
- Re: Evaluating Pen Testers Pete Herzog (Apr 22)
- Re: Evaluating Pen Testers van van (Apr 22)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)