RE: Evaluating pentesters

["Frye, Dan" <Dan.Frye () cedarcrestone com>]

Does anyone know if a "bakeoff" of pentest vendors has ever been done?
As an overly simplified example, think of an IDP bakeoff where they fire
a certain number of tests at different sensors. Whichever sensor records
the most attacks is "better" (remember this is a simplified view). If
you reverse it, basically let a certain number of pentest firms target a
test network then publish the results of who scored the highest (% vulns
existing vs % actually found). Grading is done by the firms submitting
reports and documentation of the flaws discovered in the test network.
You can tack on bonus points for good layout, etc, and probably put some
kind of scale with it against the OSSTMM to see who followed it, etc.

Or maybe have a yearly competition from the guys at ISECOM and publish
the results.

It sounds pretty simple in theory... just thinking out loud.

Daniel


-----Original Message-----
From: listbounce () securityfocus com [mailto:listbounce () securityfocus com]
On Behalf Of Rudra Kamal Sinha Roy
Sent: Thursday, March 11, 2010 12:50 AM
To: Tony Turner
Cc: pen-test () securityfocus com
Subject: Re: Evaluating pentesters

Hi Tony,

Have a look at this blog post : "5 Tips on Choosing Penetration
Testing Companies" :
http://www.ivizsecurity.com/blog/penetration-testing/how-to-choose-penet
ration-testing-companies/
 Go through it carefully and it will answer all your queries, I hope.
The tips which are highlighted in this post are:

Tip 1: Evaluate Technology Competence of Vendors
Tip 2: Focus on the vendor's real knowledge and not just on
certifications
Tip 3 Evaluate the company's trustworthiness and competence
Tip 4: Consider cost versus frequency maximum leverage
Tip 5: Seek penetration testers (Specialists) and not Generalists

Thanks!

Rudra Kamal Sinha Roy


On Tue, Mar 9, 2010 at 4:33 AM, Shohn Trojacek <trojacek () gmail com>
wrote:
> Tony,
>
> I'd say that similar to a job interview, you could ask them to tell
> "war stories" and then measure their hesitation and response time to
> detect BS. Of course, you don't want to mistake contemplation for
> hesitation, but this is generally an effective tool in any area. For
> example, you can call up a former employer and ask if they would hire
> that person again. The lack of a response can be more telling than an
> actual response at times.
>
> So essentially, the process is filter based on sample report,
> methodology, etc. This is basically like looking at someone's resume.
> Perhaps you could ask about certs, but then that may not mean anything
> either. Then once you have screened the Nessus repackagers, interview
> them placing an emphasis on war stories or perhaps describing a
> scenario and evaluating the thought process.
>
> If you issue an RFP and such, I imagine you could just bake this into
> the process depending upon your organization's contraints and such.
>
> Of course, I've found that often people request a "penetration test"
> and really want they want is exactly what you don't want.
>
> Often they just want a Nessus scan repackaged so that they can check
> whatever box they are required to. This comes back to defining what
> you want. I like to use the terms "creating management awareness of
> the depth of issues by demonstrating pro-longed and undetected access"
> in conjunction with breadth by perhaps requiring "cross-checks and
> verification of the results of scanning tools".
>
> Your mileage may vary and each situation is unique usually.
>
>
> Shohn
>
>
> On Fri, Mar 5, 2010 at 6:01 PM, Tony Turner <tony_l_turner () yahoo com>
wrote:
> > Is there some kind of "Who's Who" of penetration testing firms?
Right
> > now my primary methods for evaluating potential firms for pentest
> > engagements are requesting sanitized reports from past tests and
asking
> > questions about their methodology. Is there some resource online I
might
> > be able to use to locate quality testers? I've been burned in the
past
> > with some real bad ones.. I'm looking for
> > network/systems/application/web/wireless from a PCI focused firm.
Not so
> > much interested in physical security and social engineering tests at
> > this time but these services may be useful for future engagements.
Also
> > not interested in paying good money for someone else to just do a
> > Kismet/Gpsmap or Nessus scan for me and hand me the scan data.
Useful
> > tools of course, but I've met a few idiots who thought that was what
> > penetration testing was. I am in the SE United States.
> >
> > --
> >
> > Tony L Turner
> > CISSP, CISA, GPEN, GCIA, GSEC, VCP, ITIL-F
------------------------------------------------------------------------
> > This list is sponsored by: Information Assurance Certification
Review Board
> > Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require
a full practical examination in order to become certified.
> > http://www.iacertification.org
------------------------------------------------------------------------
> >
------------------------------------------------------------------------
> This list is sponsored by: Information Assurance Certification Review
Board
> Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require
a full practical examination in order to become certified.
> http://www.iacertification.org
------------------------------------------------------------------------
>

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review
Board

Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require
a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------


------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified.

http://www.iacertification.org
------------------------------------------------------------------------