Re: Evaluating Pen Testers

[Stephen Mullins <steve.mullins.work () gmail com>]

Blogs, "who's who" lists, bids, and yes, even your contribution to
this mailing list are all forms of advertising.

Seems to me that a "who's who" list would merely serve to allow
certain firms to rest on their laurels while they continue to have a
steady stream of high paid work because "they're the best."  Many like
to boast about their vulnerability research groups, their white
papers, and anything else they can point to in order to demonstrate
that they are "at the top of their game" still.  So let them continue
their blog posts, their white paper publishing, and their head line
making news releases.  This level of competition is healthy for the
industry.

Any sort of mandatory "certification", especially from the government,
would merely serve to reduce competition and drive up rates from the
reduced pool of "certified" groups.  Of course, that sort of
artificially induced monopoly is the best way to drive profits for a
company and it is obvious why anyone in any industry would like to
reduce their competition.

Steve Mullins

On Thu, Apr 8, 2010 at 10:18 PM, Daniel Kennedy
<danielkennedy74 () gmail com> wrote:
> There ought to be a "who's who of penetration testers, especially with
> some of what I read about and hear at conferences when it comes to
> penetration testing, for many years now, and its not getting any
> better. That said, it wouldn't be easy to put together. A firm in the
> UK was testing pen testers for a while, but their approach left some
> questions to be answered.
>
> #Confusion
> Many customers, and many security testers, confuse what is a
> vulnerability scan with a penetration test. A scan for vulnerabilities
> can be a recon activity in a pen test, its valuable information, but
> its not a pen test. Pen tests involve exploitation (usually a
> non-damaging one like opening a shell or dropping a text file) reached
> under some rules of engagement. This doesn't suggest one is better
> than the other, frankly its completely dependent on what the client is
> hoping to accomplish.
>
> #Standards
> The OSSTMM is an interesting project but its miles from being a
> standard where you can eliminate people that don't follow its
> methodology. It would be akin to saying you only accept software from
> CMM level 5 companies - the model is thorough but smart people raise
> legitimate objections to it.
>
> #Certification
> Alongside things people present as standards are certification. They
> tell you something about the person, namely that they are willing to
> take the time/cost to prove some level of proficiency in an area, but
> there are a great many security luminaries without any. The CEH is
> gaining some traction, not sure if that's a good or bad thing yet.
>
> #Nessus
> As you say, one who runs a scan and hands you a Nessus report is not
> doing much. However Nessus is a sophisticated tool for vulnerability
> scanning, has a professional license model, and compares favorably to
> more expensive options. So you can't eliminate someone for using
> Nessus, only for only using Nessus.
>
> #Open Source Tools
> The suggestion that using open source tools reveals some lack of
> sophistication or worthiness is silly. I would rather have someone
> capable of making contributions to the Metasploit project, someone who
> understands what they're running and can do hand testing, then some
> bozo who just points Core Impact at my environment and hits 'go'.
>
> #Legal Considerations
> You should consider that if something goes wrong with a company you
> are essentially sharing confidential information with, whether you
> will have protection under the law. That usually means dealing with a
> firm or person who is legitimately 'filed' (has a background you can
> check) and using someone in your firm's country or a country where
> your familiar and comfortable with the legal environment in place.
> Further you might be more comfortable with folks from certain
> backgrounds (educationally, professionally, whatever), so check out
> linkedin or something similar.
>
> #Reputation
> Most companies that can provide value in pen testing have at least
> some names that will show up when you Google. They've been quoted in
> some article, done some presentation or talk, and so forth.
>
> #Does Size Matter?
> I worked for a large transaction processing firm previously, and have
> had the benefit of seeing many pen test results from different firms.
> What I learned is that the large firms with no 'name' employees
> provide mostly nonsense, that the big ones with 'name guys' can be hit
> or miss (sometimes great, sometimes not), and that the small ones with
> a 'name guy' or two are most likely to be useful (best chance for
> great). By 'name guy' I just mean what I said earlier, someone you can
> Google and they show up having said something intelligent about
> security at some point and have a work background that makes sense for
> what they do. With large firms, as with many consultants, sometimes a
> well known intelligent guy only shows up to the first meeting by phone
> and will have nothing to do with your testing. With some, that well
> known guy is there to do his own research, write, and do presentations
> that bring attention to the company, not to do project work. Its
> something to be aware of and watch out for.
>
> Small firms where you can't get any data on anybody are usually either
> a disaster or a sign that your Google foo stinks.
>
> #Their Blog
> Tying into the previous paragraph, a lot of times I can tell the
> competency of an outfit by reading their blog. This is a good way to
> identify outfits you've maybe never heard of (can't know everybody),
> but that can provide superior results. Usually you can go through a
> few posts, see how they approach a security problem, see the quality
> with which they will be able to explain it in a report, and work from
> there.
>
> If people work in security but don't write about security or don't
> contribute to the industry, is it really their passion or just a job?
> I know which out of those two groups that I like to hire.
>
> #The Specialists
> Some security researchers are incredible at certain areas, but less
> useful as generalists which is what a firm is usually looking for with
> a pen test. In other words, if you need something specific (you want a
> product your manufacturing tested), paying a premium for "the best" at
> that thing makes sense. If you need a more general approach, you
> wouldn't go to a cardiologist for a physical, so having an RFID expert
> test your web architecture is also kind of strange. Many good firms
> have collaborative environments (ex: two guys with slightly different
> skill sets working on your stuff with two different angles).
>
> #Too Good to be True
> If a firm's bid is too low, its almost always a sign of incompetence
> or chicanery. Ask a lot of questions, because the economics of having
> a trained person work on your environment for some period of time are
> pretty constant, meaning bids of a few thousand are a rip off or a
> joke. If they seem good, just make sure you ask a lot of questions.
> You can get lucky, but the most in demand people are both what you
> need to be successful and not usually going around under bidding their
> value.
>
> #Subcontracting
> Make sure the firm you're dealing with is set to do most of the work.
> Its ok to bring experts in when you need them, but far too many firms
> are simply bidding on projects, taking a percentage, and hiring
> another entity to do the work. This is not useful to you, and what are
> you paying that percentage for.
>
> Cheers,
> Dan Kennedy
> Praetorian Security Group LLC
> http://www.praetoriansecuritygroup.com
>
> Twitter: @danielkennedy74
> Blog: http://www.praetorianprefect.com
>
> ------------------------------------------------------------------------
> This list is sponsored by: Information Assurance Certification Review Board
>
> Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
> and CEPT certs require a full practical examination in order to become certified.
>
> http://www.iacertification.org
> ------------------------------------------------------------------------

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------