Re: Evaluating Pen Testers

[Andre Gironda <andreg () gmail com>]

On Tue, Apr 13, 2010 at 1:59 PM, Daniel Kennedy
<danielkennedy74 () gmail com> wrote:
> > Are you referring to CHECK? They are still verifying
> Don't recall, it was a presentation years ago at a conference. Doesn't
It would be nice if the CHECK people would respond here about what
they offer and why it's worth anyone's time.

> I sympathize with the security consumer when trying to find someone
> competent to perform a test.
I don't necessarily sympathize. There are better activities to
perform, such as threat-modeling or code-assisted app assessments.
Pen-testing is not everything that everyone makes it out to be.

> > I do not like the words "manual" or "automated". Not all
> Whether you like the terms or not is not really material. A good
> penetration test likely has some automated tasks for time savings
> (these are time boxed tests) and some hand, or manual, or custom
> testing, whatever you'd like to call it. That said there are some
> penetration testers out there who use no well known tools and are at
> the top of the game.
> We are in complete agreement that fully automated vulnerability
> scanning is not effective and that human involvement (using
> knowledgeable humans) is a key component in a successful vulnerability
> management program.
Define "well-known" tool. I think everybody uses Burp Suite Pro,
unless there are specific circumstances that require using UHooker or
Echo Mirage before using Burp Suite Pro. Certain people like Jared
DeMott or Charlie Miller gravitate towards EFS or the Pedram Amini
PaiMei tool. Microsofties gravitate towards IOActive, Leviathan
Security, and Casaba Security tools. The security blogging community
gravitates towards Matasano and Gotham Digital Science tools (which
rely a lot on Burp, btw). And, yes, Immunity Security and Core
Security guys like their toolchains built into their commercial
products. The AttackResearch/Offensive-Security guys are totally into
Metasploit (although there was a recent blog post about Burp, which
shows up often).

> If it makes sense, then yes. With an in house team, there are all
> kinds of company policies affecting the type of software that can be
> used. But if your point is that a knowledgeable person must be
> equipped with adequate tools that the person requests, then sure.
If an internal penetration-testing team can't walk over to the
exceptions-management team and make an exception, then there is some
sort of breakdown in that particular InfoSec/Risk-Mgmt department. Or
it's a government agency that doesn't hire state-supported random
kids.

> Not taking reading assignments that aren't linked as a reference to a point
If you're not an addicted self-learner, then you will probably fail as
a penetration-tester, or even finding a good one.

Why wouldn't you just take my word for it?

> So, to me, I expect to see results in a penetration test that show, at
> a high level, what was attempted, what is believed to be exploitable,
> and what was exploited, with exploitation within the penetration
> test's time frame being the end goal of the testing team.
Yeah asking for that stuff usually requires money. It's probably
better to find the developer and ask him to show you where the code is
obviously secure then to hire a penetration-tester at 150-300 US
dollars per hour per person to exploit a target that could take hours,
days, or weeks to write an exploit that is now unusable unless you
plan on selling it or using it to an adversarial advantage. Another
reason why penetration-testing is flawed.

> What you're proposing could be interpreted as being handed a report of
> possible vulnerabilities (an incomplete one at that since you're
> stopping testing at something 'believed to be capable of
> exploitation'. That's probably useful information, but for me not
> useful enough to warrant spending money on a penetration test over
> having someone do a vulnerability scan which will show me all possible
> or believed routes of exploitation.
What? Ok, look, man. You can pay for whatever you want to pay for --
it's your money. I'll just say that I don't agree with your approach.

> Actual exploitation, which involves finding a vulnerability or
> chaining vulnerabilities together, in a custom environment, to achieve
> a proof that a system can be exploited, is difficult and what I'm
> looking to have be attempted in a penetration test.
Why? Isn't an alert box or !exploitable output (especially peer/tool
reviewed) enough for you? Isn't an obvious lack of input validation
combined with improper coding practices enough to say -- let's make
this obviously secure in the code instead of spending time on
penetration-testing?

> > Almost all RFQ analysis is followed by Case Study analysis and
> > extremely high-quality References before hiring an application
> > security consulting company.
>
> To assume that all security companies/consultants are hired after a
I didn't assume anything. You seemed to have assumed that I assumed
something. Re-read what I wrote.

> In house is valuable because you retain available talent and generally
> can spend more time testing more things. That said it does not have
> the economy of scale of hiring outside consulting help, and many

Robert Auger recently said, "Many consultants don't seem to understand
practical business risk management (or often aren't around long enough
to get good at these activities) and instead are used to providing
generic advice for solving a problem with little understanding on how
to accomplish this in the real world (at both a technical level and
business ). An advantage of doing appsec full time is the ability to
develop real solutions and see how they can be improved based on real
world experience rather than educated guesses".

> companies (especially in this economy) are not of the shape and scale
> to justify maintaining a full time penetration testing team. Having a
You don't seem like an economist or a business-level decision-maker to me.

> single resource runs the risk of having that resource leave at any
> time (no coverage overlap), and robs the penetration testing team of
> the benefit of collaborating during testing (most testers are not
They could always collaborate with the consultants on deck.

> experts in every system or type of system they encounter). Having a
> resource that does other things, but sometimes tries to do penetration
> tests, leaves you with a party not fully committed or immersed in the
> infosec industry doing your testing.
Actually the best penetration-testers come from other fields. Inciting
others inside the organization to take up the pen-testing torch is a
very wise move. And, boom, you've got even more collaboration.

> > > #Certification
> > > there are a great many security luminaries without any. The CEH is
> > > gaining some traction, not sure if that's a good or bad thing yet.
> > I think it's a bad thing. What does China use to certify their
> > penetration-testing talent?
> Not sure why what China is doing or not doing is important. But I
Oops. I think I forgot to mention that the US DoD is now requiring the
CEH for specific roles.

> > other words, the first question in the interview should be "Which
> > BackTrack tool did you write or contribute to?" and the second
> > question should be "When was the last time that you spoke at an OWASP
> > local chapter meeting?"
> What is they're not a member of OWASP, or just don't want to speak at
You don't have to be a member of OWASP. There are no dues and no fees.
OWASP is the opposite of ISSA and ISACA. You don't require a
background check to come to meetings like InfraGuard. You just show
up. You just ask the chapter leader if you can present at a future
meeting. You just send the OWASP Board an email and start your own
chapter if you don't have one, or if yours seems dead. It's actually
easier than what I'm describing.

> OWASP meetings. OWASP is a great outfit, but not everyone is a member.
OWASP does need more sponsorship and money in the form of memberships.
They honestly do. But that has nothing to do with this conversation.

> What if its not an application penetration test? What if they don't
> use BackTrack (which is a great tool as an aside)?
If they don't  like BackTrack, then I hope they have another answer
that would respect the purpose of that question.

A bad answer might be "What's BackTrack?", unless it's followed up
with "Oh yeah, I don't use that garbage; I built my own pen-test OS
platform"

> Further you're making my point below for me, that a company or person
> with verifiable talent is a better hire then one without.
Somehow I'm sure we don't disagree on much. You just seem to be new
and I just feel like I'm over-educating you for free.

> I don't think I said anything about anyone at Core Impact. Core Impact
> is a tool that has reached a point in maturity where even a fairly
> non-technical person can know an IP address or range, run a scan, run
> a set of vulnerabilities based on that scan, and install the Core
> Impact backdoor on the target which would meet the definition of most
> penetration tests, but which is probably not worth paying someone to
> do. Its moronic to think such a statement was an insult.
No, Core is much more than the RPT module -- I don't think you
understand that. Recently, Core added support to drop into Metasploit.
Wait until all of the web application scanners add that same sort of
drop-in support for Burp. Or Metasploitburpuby.

What I'm trying to say is that there are plenty of people who work for
Core and do penetration-testing (or write Impact). Just because they
use Impact doesn't mean that they only use the RPT module. Get over it
already -- you were being presumptuous and I called you on it.

> In the hands of an experienced person, Core Impact is a powerful tool
> and one that can be a help during a penetration test. So is
> Metasploit. The point is that if I'm paying the money to bring someone
> in, I want that experienced person. The point I made above is that I'd
You want Ivan Arce and HD Moore? I think they already have day jobs
that keep them busy...

> rather have an experienced person with Metasploit then someone with no
> experience using Core Impact. You can write it the other way too, I'd
You forgot that I told you that you should let people run their own tools.

> rather have an experienced person with Core Impact then an
> inexperienced person with Metasploit.
Ok, I think you're starting to understand now!

> This all leads to not using "the person lists Metasploit as a tool" as
> a way to eliminate candidate companies or persons for doing your
> penetration testing.
The best way to eliminate someone from your list of candidates is to
not know them personally or what they are capable of. If you don't
know anybody -- go to a local OWASP chapter meeting, or perhaps a
CitySec event (e.g. ChiSec), or maybe a Hackers Anonymous (e.g. AHA).
Or go to a cheaper, regional conference such as Toorcon/Toorcamp,
Shmoocon/SOURCE, or a SecurityBSides event.

Also -- be an addicted self-learner and post stuff to mailing-lists,
read blogs/twitter, and make friends and influence people by reading
books.

> Insurance, especially with limitations in coverage, may protect the
> security company in cases of legal liability but provides a small
> amount of protection to the hiring company. In most cases, if a
> penetration tester went rogue with information from a penetration
> test, the resulting reputation damage and bad publicity would be of
> greater value then the insurance settlement.
When insurance fails, litigation is quick to follow... BTW IANAL

> So I stand by checking people out, both from a legal protection
> standpoint, but also because you want the engagement to be successful
> in your environment and therefore should check out the backgrounds of
> the people involved with the test. I don't view possessing insurance
> as an end all indicator of anything.
I suggest criminalsearches.com (it's free and it works). Also good to
do an SSN check -- http://www.ssa.gov/employer/ssnv.htm
Verify their business license, do those case studies, and verify a
reference if you really want to do more. Track their parcel addresses
back and make sure you know where they live/work and zoom in on it
from Google Maps if you are really paranoid.

Background check companies (you definitely want one that is listed on
napbs.com) are notoriously expensive and difficult to deal with -- so
best of luck with that strategy. Perhaps it's best to build your own
background check system. Even LexisNexis and ChoicePoint are usually a
total failure.

> With respect to their personal wishes, one would immediately ask why
> they want to keep a low profile. Assuming there is nothing untoward
Maybe they are too busy working to be talking on mailing-lists?

> there, those folks should understand that there abilities have to be
> known to someone in order for a demand to be there for them. Even
Known, yes. By Google? No!

> folks with pseudonyms usually leave a trail to find them, they just
> don't want to be identified trivially and sent nonsense correspondence
> by people who don't understand the information security industry.
Define "usually"? Most people just don't want to be bothered with
industry punditry.

> In reality, the decision to hire one company or individual over
> another is based on a range of factors (that could include an RFQ or
> RFP) some more legitimate factors than others.  But if I wanted to
> hire someone, and one candidate had something like this online:
> And the next guy had no information I could verify, then I would
> probably look more favorably on the skills of the first candidate.
Isn't the Internet great? I think we agree on these points ;>

> > http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2
>
> Good example of how RFP processes can be rife with document templates
> filled with boilerplate language.
Doesn't sound like you read it to me, but it's not free information
for probably a damn good reason.

Cheers,
Andre

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified.

http://www.iacertification.org
------------------------------------------------------------------------