RE: Evaluating pentesters

["Cor Rosielle" <cor () outpost24 com>]

Competitions are fun, but the result does not tell you which pentest vendor
is the best. In Dan's simplified example he suggests 'the most attacks is
"better"', but for pentesting I would suggest that penetrating with the
least attacks gives a winner. But again, this does not tell you which vendor
provide the best pentesters, it just tells which is best in winning a
competition (or playing a game). Further, as a customer you don't have any
guarantees that the testers who participated in the contest will be assigned
to your project. So in my opinion such a rating is not worth much.

The reversed approach is far from ideal too. I think it is a fundamentally
wrong approach for a pentester to only search for known vulnerabilities. Of
course testers include this low hanging fruit in a test, but you can use a
vulnerability scanner to do the majority of the work (you won't be surprised
if I tell you that Outscan is the best). The right approach is when a tester
interacts with the target and gains good knowledge about the target (like
how it responds to requests, how it interacts with its environment etc.).
Once that knowledge is there, attempts can be made to use/misuse/abuse that
knowledge to construct an attack. And of course the tester could use a known
exploit, but good testers are capable of constructing their own attacks (and
not only use those found at milw0rm and the like).

Also the analyst/auditor should not focus on known vulnerabilities only in
the analysis phase. Protecting against known vulnerabilities gives
protection about attacks from the past. A good analyst can give
recommendations to protect against future attacks as well.

If you need a proper pentest done, then make sure you demand the assessment
is done by qualified testers and analysts. If the tester/analyst hold some
certifications, you know they must have at least some knowledge about what
(s)he's doing. The ISECOM certifications are good ones. They are based on
the OSSTMM and therefore have the big advantage you're able to objectively
compare (future) test results with previous tests. It can further assist you
in making decisions when you have a limited budget and need to choose
between different security solutions. By calculating the expected RAV-scores
you can predict the effect of a security solution before you spend any money
and then choose the one with the most effective cost/benefit ratio.
You can verify if someone has an ISECOM certification at
http://www.isecom.org/verify_people/ (people are only found on this list
after they gave permission; if the name is on the list, you are sure they
really hold that certification). 

Cor Rosielle - Lab106 - Amsterdam


> -----Original Message-----
> From: listbounce () securityfocus com
> [mailto:listbounce () securityfocus com] On Behalf Of Frye, Dan
> Sent: donderdag 11 maart 2010 23:31
> To: pen-test () securityfocus com
> Subject: RE: Evaluating pentesters
>
> Does anyone know if a "bakeoff" of pentest vendors has ever been done?
> As an overly simplified example, think of an IDP bakeoff where they
> fire
> a certain number of tests at different sensors. Whichever sensor
> records
> the most attacks is "better" (remember this is a simplified view). If
> you reverse it, basically let a certain number of pentest firms target
> a
> test network then publish the results of who scored the highest (%
> vulns
> existing vs % actually found). Grading is done by the firms submitting
> reports and documentation of the flaws discovered in the test network.
> You can tack on bonus points for good layout, etc, and probably put
> some
> kind of scale with it against the OSSTMM to see who followed it, etc.
>
> Or maybe have a yearly competition from the guys at ISECOM and publish
> the results.
>
> It sounds pretty simple in theory... just thinking out loud.
>
> Daniel
...snip...


------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------