Re: Evaluating Pen Testers

[Pete Herzog <lists () isecom org>]

Hi,

The problem with pen testing is that as a model it used to fit well
into the cyber security realm because it was a way to prove problems
in this nascent industry where many people failed to understand that
what you can't see can hurt you. So, as I had heard once said, pen
testing is akin to pushing a child out of the way of an oncoming car
(sorry Andre, I know you dislike security metaphors but this ties in
nicely with my point). The right thing to do starts with how roads are
made to enforce safety and then the child is educated, etc. but when
things are new, people sometimes need someone, like a pen tester, to
make the dangers glaringly obvious to them. This need has lessened. So
pen testing is still needed but much less so these days as people ARE
aware there are dangers but they just can't care or have other ways of
justifying them. Making problems glaringly obvious to them usually
doesn't help change them though. Mostly because they don't see how
what they do may also cause harm to those around them mostly because
we are all interconnected.

Today, pen testing seems to have evolved into a way of teaching
clients they have problems. I say this as a negative thing. Logically
we know an ineffective way to teach kids street safety is to line them
up and drive at them, forcing them to dodge out of the way. Therein
lies the problem.

If a penetration test only tells the client where the holes are and
how they were abused, nothing is learned. At best, useless processes
like patching and perhaps some coding fixes take place but this lasts
only until the next pen tester or next 0day comes along. So if you
really want to evaluate pen testers, look at what they provide you.
That's the biggest thing the OSSTMM 3 has changed for pen testing- the
product. It's not about following a methodology to show you were
thorough. The methodology part is just a way that is suggested you can
use to deliver the right product which is being able to clearly state
the interactive points discovered within a specific scope and the
effectiveness of all controls for each of those interactions.
Delivering a product like that does 2 things: it gives a client a
specific accounting of operations from the tested vector and it allows
them to address that which is not controlled correctly. Why this is so
different is because now it's not about your hack-fu being smarter or
stronger but about it letting you be deeper. So even a pen tester with
less skills can still provide an equally good pen test if the product
which they deliver is actionable and thoroughly accounts for the
operations of all interactions. So a client has the chance to learn
about how their lack of understanding or caring effects so much more
than what they probably think and how to address uncontrolled
interactions in a way that allows them to step out of the endless
vuln/patch role and into building up security.

So for the pen testers who don't evolve to support new strategies
based on new security research it will become as glaringly obvious as
the pen tests they hawk that they shouldn't be hired.

Just an observation.

Sincerely,
-pete.


> On Thu, Apr 15, 2010 at 10:02 PM, Andre Gironda <andreg () gmail com> wrote:
> > On Thu, Apr 15, 2010 at 2:49 PM, Daniel Kennedy
> > <danielkennedy74 () gmail com> wrote:
> > > At this point you're resorting to ad hominem attacks or misdirections
> > At some point either one of us should have de-escalated any arguments
> > that were thought to be attacks. I never thought I was attacking you,
> > but certainly you seem to have taken it that way. Or, we could have
> > kept it off the list. I'm going to delete anything in this thread that
> > I feel that I have to defend, accept your corrections, and make some
> > of my own. Hopefully the list gets something out of all of this.
> >
> > > I've been a member of both at various times, but I wasn't talking
> > > about myself I was talking about evaluating a potential penetration
> > > testing partner. I was suggesting that not every capable tester is
> > > necessarily also a member of OWASP.
> > I think I've met one penetration-tester in my whole life who was a
> > member of OWASP. Most of the time, it's companies. This particular
> > individual is his own company, so the lines blur. Note that Gotham
> > Digital Science is both an OWASP member and CHECK certified (this is
> > an example for your reference and my amusement).
> >
> > > > Why? Isn't an alert box or !exploitable output (especially peer/tool> reviewed) enough for you?
> > > In web application security testing, both of those things are proof of
> > > exploitation, falling under what would count as supplying a proof of
> > > exploitation.
> > Hooray! I agree, but there are many people out there who do not.
> >
> > > > Somehow I'm sure we don't disagree on much. You just seem to be new and I just feel like I'm over-educating you 
> > > > for free.
> > > A search on linkedin, Google, or whatever takes about two seconds and
> > > saves you from making asinine statements like this one. But if I need
> > > more help I'll be sure to give Gotham Digital Science a call.
> > I'm really not the expert that I'm making myself out to be, but I am a
> > B.S. detector. I'm not saying that anything you've said so far is
> > B.S., but I'm certainly trying to push you to fall on your face if
> > that's what you want to do. You haven't fallen yet, but you have made
> > some mistakes.
> >
> > Such as the fact that your attempts at guessing who I am have failed.
> >
> > > > Background check companies (you definitely want one that is listed on napbs.com) are notoriously expensive
> > > Not really, but if you've never been involved in having background
> > > checks done for a company, I could see why someone might think that.
> > Hah, well I've only been the one that causes the expense of background
> > checks to go up, but I am familiar with the process. I'd explain, but
> > no.
> >
> > > > Define "usually"? Most people just don't want to be bothered with industry punditry.
> > > Your leadership seems to understand the value of such punditry:
> > > http://www.darkreading.com/security/attacks/showArticle.jhtml?articleID=212400450.
> > > Most of the company and researcher names you dropped below also have a
> > > history of understanding basic marketing.
> > Yeah. I don't work for Brian and never have. This might be bad, but oh
> > well. I never said that I did -- you just assumed. Sorry to all
> > involved.
> >
> > > > Doesn't sound like you read it to me,
> > > Forrester corporate memberships are a great thing.
> > Yeah. I learned about that paper from my past. I delete old files that
> > I shouldn't have access to anymore, so I couldn't even tell you what
> > the paper said. Was Gotham Digital Science in there, but Praetorian
> > not? Oh no! I'm so sorry -- I've heard good things about Nathan
> > Sportsman, and certainly I'll have nice things to say about you from
> > now on as well (you did beat me pretty fair and square in this little
> > argument/discussion even though you guessed a whole bunch of things
> > about me wrong. I have to look up to that).
> >
> > Oh and just as a final jab, I hope you keep your CEH skills current.
> > (that's for the mailing-lists enjoyment, btw)

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------