Re: Evaluating Pen Testers

[Andre Gironda <andreg () gmail com>]

On Thu, Apr 8, 2010 at 9:18 PM, Daniel Kennedy
<danielkennedy74 () gmail com> wrote:
> There ought to be a "who's who of penetration testers, especially with
> some of what I read about and hear at conferences when it comes to
> penetration testing, for many years now, and its not getting any
> better. That said, it wouldn't be easy to put together. A firm in the
> UK was testing pen testers for a while, but their approach left some
> questions to be answered.

Are you referring to CHECK? They are still verifying
penetration-testing capability at the company-level.

I strongly discourage anyone from building a list of individuals;
Microsoft and others have tried this before and the ethical
consequences of these actions is somewhat revolting (to at least
myself). It would be impossible to keep a list current because people
come and go all of the time (at least by the hour).

> #Confusion
> Many customers, and many security testers, confuse what is a
> vulnerability scan with a penetration test. A scan for vulnerabilities

I do not like the words "manual" or "automated". Not all
penetration-testing activity can be automated, but a lot can. Not all
application scanning activity should be automated. Humans HAVE to be
involved. Vulnerability scan activity should not be automated without
humans, but Qualys QG has convinced people (read: managers) that it
can.

Penetration-testers MUST be allowed to choose their own toolchain and
decide which parts to automate and which parts to leave with some
manual intervention.

> can be a recon activity in a pen test, its valuable information, but
> its not a pen test. Pen tests involve exploitation (usually a

Please read
[PDF] http://www.securityacts.com/securityacts02.pdf
[PDF] http://www.net-security.org/dl/insecure/INSECURE-Mag-25.pdf
before continuing...

> non-damaging one like opening a shell or dropping a text file) reached
> under some rules of engagement. This doesn't suggest one is better
> than the other, frankly its completely dependent on what the client is
> hoping to accomplish.

I think I disagree with you. Penetration-testing can certainly stop
before exploitation, assuming that something is found that is believed
or known to be capable of exploitation.

> #Standards
> The OSSTMM is an interesting project but its miles from being a
> standard where you can eliminate people that don't follow its
> methodology. It would be akin to saying you only accept software from
> CMM level 5 companies - the model is thorough but smart people raise
> legitimate objections to it.

The art of drafting a proper RFQ to potential penetration-testing
consultants is a WIP by OWASP as seen here --
http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers&discussionID=17288173&gid=36874&commentID=14543002
Almost all RFQ analysis is followed by Case Study analysis and
extremely high-quality References before hiring an application
security consulting company.

I recommend retaining in-house penetration-testing talent. I suggest
letting them follow any penetration-testing methodology/standards
which they choose, as long as it gets results. Most will choose all
standards and no standards at the same time. Penetration-testers are
very talented at being in two places at once given their paradoxical
natures. Allow them to do this.

> #Certification
> Alongside things people present as standards are certification. They
> tell you something about the person, namely that they are willing to
> take the time/cost to prove some level of proficiency in an area, but
> there are a great many security luminaries without any. The CEH is
> gaining some traction, not sure if that's a good or bad thing yet.

I think it's a bad thing. What does China use to certify their
penetration-testing talent?

If somebody has written some new code that steals some stuff that
nobody has stolen before, then that should be certification enough. In
other words, the first question in the interview should be "Which
BackTrack tool did you write or contribute to?" and the second
question should be "When was the last time that you spoke at an OWASP
local chapter meeting?"

> #Nessus
> As you say, one who runs a scan and hands you a Nessus report is not
> doing much. However Nessus is a sophisticated tool for vulnerability
> scanning, has a professional license model, and compares favorably to
> more expensive options. So you can't eliminate someone for using
> Nessus, only for only using Nessus.
> #Open Source Tools
> The suggestion that using open source tools reveals some lack of
> sophistication or worthiness is silly. I would rather have someone
> capable of making contributions to the Metasploit project, someone who
> understands what they're running and can do hand testing, then some
> bozo who just points Core Impact at my environment and hits 'go'.

I have no idea what you're talking about, but as I said before -- let
the penetration-testers choose their own toolchain.

There are plenty of badasses at Core that were instantly disrespected
by your remark.

Also see "automation" vs. "manual" above.

Finally, if you want to know the differences between tools, read this
-- http://stackoverflow.com/questions/72166/penetration-testing-tools/74513#74513

> #Legal Considerations
> You should consider that if something goes wrong with a company you
> are essentially sharing confidential information with, whether you
> will have protection under the law. That usually means dealing with a
> firm or person who is legitimately 'filed' (has a background you can
> check) and using someone in your firm's country or a country where
> your familiar and comfortable with the legal environment in place.
> Further you might be more comfortable with folks from certain
> backgrounds (educationally, professionally, whatever), so check out
> linkedin or something similar.

Every application security consulting company has insurance to cover
themselves. Here's an RFQ/RFP hint: Make sure the ones that you hire
have insurance.

> #Reputation
> Most companies that can provide value in pen testing have at least
> some names that will show up when you Google. They've been quoted in
> some article, done some presentation or talk, and so forth.

Sometimes people use pseudonyms or like to keep a low profile for
personal reasons, so be careful with this one.

Please do not hire people based on Google, their blog, or some claim
of "specialty" or "generality". You need a formal RFQ/RFP process

If you really need a starting point, check this out --
http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2

Cheers,
Andre

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------