Penetration Testing mailing list archives
RE: Evaluating pentesters
From: "Frye, Dan" <Dan.Frye () cedarcrestone com>
Date: Thu, 11 Mar 2010 17:31:02 -0500
Does anyone know if a "bakeoff" of pentest vendors has ever been done? As an overly simplified example, think of an IDP bakeoff where they fire a certain number of tests at different sensors. Whichever sensor records the most attacks is "better" (remember this is a simplified view). If you reverse it, basically let a certain number of pentest firms target a test network then publish the results of who scored the highest (% vulns existing vs % actually found). Grading is done by the firms submitting reports and documentation of the flaws discovered in the test network. You can tack on bonus points for good layout, etc, and probably put some kind of scale with it against the OSSTMM to see who followed it, etc. Or maybe have a yearly competition from the guys at ISECOM and publish the results. It sounds pretty simple in theory... just thinking out loud. Daniel -----Original Message----- From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Rudra Kamal Sinha Roy Sent: Thursday, March 11, 2010 12:50 AM To: Tony Turner Cc: pen-test () securityfocus com Subject: Re: Evaluating pentesters Hi Tony, Have a look at this blog post : "5 Tips on Choosing Penetration Testing Companies" : http://www.ivizsecurity.com/blog/penetration-testing/how-to-choose-penet ration-testing-companies/ Go through it carefully and it will answer all your queries, I hope. The tips which are highlighted in this post are: Tip 1: Evaluate Technology Competence of Vendors Tip 2: Focus on the vendor's real knowledge and not just on certifications Tip 3 Evaluate the company's trustworthiness and competence Tip 4: Consider cost versus frequency maximum leverage Tip 5: Seek penetration testers (Specialists) and not Generalists Thanks! Rudra Kamal Sinha Roy On Tue, Mar 9, 2010 at 4:33 AM, Shohn Trojacek <trojacek () gmail com> wrote:
Tony, I'd say that similar to a job interview, you could ask them to tell "war stories" and then measure their hesitation and response time to detect BS. Of course, you don't want to mistake contemplation for hesitation, but this is generally an effective tool in any area. For example, you can call up a former employer and ask if they would hire that person again. The lack of a response can be more telling than an actual response at times. So essentially, the process is filter based on sample report, methodology, etc. This is basically like looking at someone's resume. Perhaps you could ask about certs, but then that may not mean anything either. Then once you have screened the Nessus repackagers, interview them placing an emphasis on war stories or perhaps describing a scenario and evaluating the thought process. If you issue an RFP and such, I imagine you could just bake this into the process depending upon your organization's contraints and such. Of course, I've found that often people request a "penetration test" and really want they want is exactly what you don't want. Often they just want a Nessus scan repackaged so that they can check whatever box they are required to. This comes back to defining what you want. I like to use the terms "creating management awareness of the depth of issues by demonstrating pro-longed and undetected access" in conjunction with breadth by perhaps requiring "cross-checks and verification of the results of scanning tools". Your mileage may vary and each situation is unique usually. Shohn On Fri, Mar 5, 2010 at 6:01 PM, Tony Turner <tony_l_turner () yahoo com>
wrote:
Is there some kind of "Who's Who" of penetration testing firms?
Right
now my primary methods for evaluating potential firms for pentest engagements are requesting sanitized reports from past tests and
asking
questions about their methodology. Is there some resource online I
might
be able to use to locate quality testers? I've been burned in the
past
with some real bad ones.. I'm looking for network/systems/application/web/wireless from a PCI focused firm.
Not so
much interested in physical security and social engineering tests at this time but these services may be useful for future engagements.
Also
not interested in paying good money for someone else to just do a Kismet/Gpsmap or Nessus scan for me and hand me the scan data.
Useful
tools of course, but I've met a few idiots who thought that was what penetration testing was. I am in the SE United States. -- Tony L Turner CISSP, CISA, GPEN, GCIA, GSEC, VCP, ITIL-F
------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification
Review Board
Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified.
http://www.iacertification.org
------------------------------------------------------------------------
------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review
Board
Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified.
http://www.iacertification.org
------------------------------------------------------------------------
------------------------------------------------------------------------ This list is sponsored by: Information Assurance Certification Review Board Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified. http://www.iacertification.org ------------------------------------------------------------------------ ------------------------------------------------------------------------ This list is sponsored by: Information Assurance Certification Review Board Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified. http://www.iacertification.org ------------------------------------------------------------------------
Current thread:
- Evaluating pentesters Tony Turner (Mar 08)
- Re: Evaluating pentesters Tracy Reed (Mar 08)
- Re: Evaluating pentesters security curmudgeon (Mar 11)
- Re: Evaluating pentesters David Glosser (Mar 08)
- Re: Evaluating pentesters Andre Gironda (Mar 08)
- Re: Evaluating pentesters aceinyaface (Mar 09)
- Re: Evaluating pentesters Jason Ross (Mar 09)
- Re: Evaluating pentesters Brent Huston (Mar 11)
- Re: Evaluating pentesters Shohn Trojacek (Mar 09)
- Re: Evaluating pentesters Rudra Kamal Sinha Roy (Mar 11)
- RE: Evaluating pentesters Frye, Dan (Mar 11)
- RE: Evaluating pentesters security curmudgeon (Mar 15)
- Re: Evaluating pentesters Pete Herzog (Mar 17)
- RE: Evaluating pentesters Cor Rosielle (Mar 23)
- Re: Evaluating pentesters Rudra Kamal Sinha Roy (Mar 11)
- Re: Evaluating pentesters Tracy Reed (Mar 08)
- Message not available
- Fwd: Evaluating pentesters Daniel Hood (Mar 11)
- Re: Evaluating pentesters Mohamed Farid (Mar 11)
- <Possible follow-ups>
- Re: Evaluating pentesters ben . dexter (Mar 11)
- Re: Evaluating pentesters Daniel Clemens (Mar 11)