Re: Verify Your Security Provider -- The truth behind manual testing.

["Adriel T. Desautels" <ad_lists () netragard com>]

This should be fun/interesting, I'll embed my comments below.

On Jul 18, 2009, at 8:51 PM, Dotzero wrote:

> I'm going to respond to Adriels comments from a client side  
> perspective.
>
> On Thu, Jul 16, 2009 at 11:36 AM, Adriel T.
> Desautels<ad_lists () netragard com> wrote:
> > A recent blog entry that we thought "some" of you pen-testers might  
> > find
> > interesting.  Feel free to leave comments on the blog:
> >
> > Direct URL:
> > http://snosoft.blogspot.com/2009/07/truth-behind-manual-testing.html
> >
> >
> > Verify Your Security Provider -- The truth behind manual testing.
> > Something that I’ve been preaching for a while is that automated
> > vulnerability scanners do not produce quality results and as such  
> > shouldn’t
> > be relied on for penetration tests or vulnerability assessments.  
> > I’ve been
> > telling people that they should look for a security company that  
> > offers
> > manual testing, not just automated scans. The price points for  
> > quality work
> > will be significantly higher, but in the end the value is much  
> > greater.
> > After all the cost in damages of a single successful compromise is  
> > far
> > greater than the cost of the best possible security services.
>
> Adriel is correct that there is a wide variation in the nature and
> quality of services offered in the pentesting space. Perhaps the
> larger issue is that a significant portion of people on the client
> side are not in a position to a) understand exactly what they are (or
> should be) asking for and b) to evaluate the proposals from vendors.

I've often wondered if there would be a market for a company or group  
that
did nothing but evaluate proposals for vendors.  It is after all not  
always about
the ferrari, it is about offering a service that meets the customers  
requirements.

> > I’ve noticed that there are a bunch of vendors who claim to be  
> > performing
> > manual testing. But when I dig into their methodologies their  
> > manual testing
> > isn’t real manual testing at all, its just vetting of automated  
> > scanner
> > results or testing based on the results. In other words they test  
> > on what
> > the automated scanner reports and don’t do any real manual  
> > discovery. I’m
> > not saying that tools like nessus (an automated scanner) don’t have  
> > their
> > place, I’m just saying that they aren’t going to protect you from  
> > the bad
> > guys. If you want to be protected from the threat, you need to be  
> > tested at
> > a level that is a few notches higher than the threat that you are  
> > likely to
> > face in the real world.
>
> The threat in the real world is not a constant and in fact is a
> constantly moving target. Pentesting (whether manual or automated) is
> a snapshot in time. The risk associated with the fluctuating threat
> level is also constantly changing as the target environment changes.

That is absolutely accurate.  As a result of the evolving threat it is  
important
that the security vendor be active in keeping tabs on the threat.   
Additionally,
it is the evolving threat that requires businesses to perform  
vulnerability
assessments and/or penetration tests more than once a year.  My  
suggestion
is that quarterly vulnerability assessments be done and annual  
penetration
tests.

> > This is akin to how the Department of Defense tests the armor on  
> > its tanks,
> > and I’ve probably mentioned this before somewhere on the blog. But,  
> > we don’t
> > test our tanks against fire from bb guns and .22 caliber pistols.  
> > If we did
> > that they wouldn’t be very effective in war.We test the tanks  
> > against a
> > threat that is a few levels higher in intensity than what they are  
> > likely to
> > face in the real world. As a result, the tank can withstand most  
> > threats and
> > is a very effective weapon. Doing anything less isn’t going to  
> > protect you
> > when the threat tries to align with your risks; you’ll end up being  
> > an
> > expensive casualty of war.
> >
> > So why do some security companies test at this lesser level? Its  
> > simple
> > really, they are in the business of making money and care more  
> > about that
> > then they do about actually protecting their customer’s  
> > infrastructure.
>
> It might be that the customer has decided that the "lesser level" is
> what they want. Even in military purchasing tradeoffs are made. A
> country may choose a less heavily armored vehicle because they believe
> they can get greater mobility and range. It is in fact not a binary
> choice. The purchaser may choose a mix.

Well, I can't argue with you about the customer's needs and desires.   
My point
was more that security companies often test at a lesser level than  
what they
market to their customers.  It is almost as if some are selling a  
false sense of
security to their customers.  If they do sell a service that tests at  
a lesser level
than the "real world threat" then they should make it clear to their  
customer that
they are testing at this lesser level, shouldn't they? Isn't that type  
of honesty ethical?


> > Additionally, there is a market for that sort of low quality  
> > testing. There
> > are some businesses that don’t actually care about their security  
> > posture;
> > they just care about passing the test so that they can put a check  
> > in their
> > compliancy box. Then there are other businesses that unknowingly  
> > get taken
> > advantage by of vendors because they don’t know the difference  
> > between high
> > quality and low quality services.
> >
> > So what is the difference between high quality and low quality?  
> > From a high
> > level perspective it’s the difference between real manual research  
> > based
> > security testing or not. Once hackers have access, they can do  
> > anything to
> > your data from steal it, to install back door technology in your  
> > product's
> > source code. Its happened before, and its going to happen again.
>
> I'm going to have to disagree with you Adriel. "Access" is not a
> generic. The real questions are access to what? Access of what nature?
> and access for how long?.

I think that I'll partially agree with your disagreement because I  
didn't clearly
define "Access".  When i say "Access" I am talking about code  
execution /
command execution. Once that level of access is attained its usually,  
but not
always just a matter of time till Distributed Metastasis happens with  
great
success.

> > When a company tells you that they perform manual testing hold  
> > their feet to
> > the fire. You can do the following things to verify it:
> >
> >
> >        • Dig into their methodology and ask them specific questions  
> > about
> > how they perform their testing. (See our white papers on how to do  
> > that).
> >        • Don’t swallow jargon and terms that sound cool and don’t  
> > mean
> > anything, use Wikipedia to look up the terms and make sure that  
> > they make
> > sense.
>
> I'm not sure that I would consider wikipedia authoritative. Take the
> term "pivot" this is commonly used in pentesting/security yet if you
> search on wikipedia for it you will not find it as relates to
> pentesting.

I agree that the sites that I provided are not "authoritative" but  
they can provide
insight into a vendor's capabilities.  What other tools do customers  
have to use
to verify their security vendor?

> >        • Ask them for the names of their security experts and then  
> > use tools
> > like Google, LinkedIn, Facebook and PIPL to do research on those  
> > experts. If
> > nothing comes up then chances are their experts aren’t experts at  
> > all.
>
> Knowing some highly qualified people that do not show up in searches
> such as you describe, I'm not sure that I would agree with you. I'm
> thinking of folks that have backgrounds with 3 letter agencies or use
> handles/nom de plumes (if you will) to distinguish their personal
> activities from their activities as employees or representatives of
> organizations. I've used dotzero (various ISPs/mail accounts) for over
> 20 years. It is a simple and easy way to make clear that what I write
> or post is personal.

Remember, I'm not talking about individuals, I'm talking about  
businesses as a
whole.  Most businesses that have teams that perform research do  
publish their
research.  That said, you are right in that there are individuals who  
are talented
but don't have any "public" exposure as a matter of speaking. That  
said, I wouldn't
consider someone talented just because they worked at a 3 letter  
agency.  In fact,
we have members of our team that come from some of those agencies,  
most of what
they learned they didn't learn working public sector. Instead, they  
were hired because
of what they learned on their own.

> >        • Search vulnerability databases like milw0rm,  
> > securityfocus,sirtfr,
> > secunia, packetstormsecurity, etc. for the vendor’s name to see if  
> > they have
> > research capabilities. If you don’t get anything in return then  
> > chances are
> > that they don’t have research capabilities. If that’s the case then  
> > how do
> > you expect them to perform quality manual testing? Chances are that  
> > they
> > won’t be able to.
> >
> > Remember you’re putting the integrity of your business and its  
> > respective
> > name into their hands.
>
> I agree with you that there are significant issues but it's not clear
> to me that it is as clear cut as you make it out to be. I'm currently
> evaluating proposals for a pentest engagement and I figure I will have
> about 110+ hours into it by the time we make a decision. This doesn't
> include the hours for the other folks who will need to sign off on the
> decision. The rough break out of my time is:
>
> 8 hours to prepare the RFP (working from and modifying a previous  
> version)
> 8 hours deciding which vendors to invite to participate in the RFP  
> process
> 4 hours dealing with the NDA that potential vendors were required to
> sign (does not include time of our contract coordinator)
> 20 hours answering questions from vendors
> 80 hours evaluating proposals

I think that the most difficult part of your job is going to be  
proposal evaluation.
You'll need to find vendors that don't just have a big hat, but that  
also have a
lot of good healthy cattle.  :)

> So how many companies are going to spend 3+ person weeks of working
> time (what kind of impact does that have on the organization assuming
> the person is qualified and not simply an administrative drone going
> through the motions?) just to select a vendor? Price is certainly a
> factor but there so many other things to look at.
>
> How should the average client compare various alphabet soup
> combinations that follow peoples names? How does a CISSP compare to a
> CEH or a GIAC certification? Some people look mighty impressive on
> paper (or the internet) but are not worth the air they breath if you
> were to end up hiring them.

I absolutely agree with you and understand your frustrations.

> My personal take is that the overall situation will get worse before
> it is likely to get better. There simply aren't enough
> qualified/experienced security people to go around.... and let's be
> honest, IT security does not come cheap (although I agree a breach is
> potentially much more expensive)
> Even though we may not agree 100% I appreciate your perspective and
> your thought provoking posts.

Thank you and it was my pleasure.  I knew when I published that post  
that
most people would have a problem with my idea of using the various sites
that I mentioned to vet providers.  I have some of my own issues with  
it as well,
but I'm not sure what else to recommend.  I'd like to help people find  
providers
that meet their needs and that are honest about their service  
offerings.  I hate
seeing people buy a service only to find out later that it wasn't  
worth the paper
that it was printed on.

That is to say, some people buy penetration tests and get nothing more  
than
nessus scans in return but in the end they don't know it because they  
are not
the experts.  It is the job of the ethical provider to educate their  
customers so
that they know what they are getting, and what they aren't.  If a  
customer just
wants a scan, then fine, scan them and charge them fair value.  If a  
customer
wants a penetration test, figure out how intense and deliver it if you  
have the
required talent.  But don't say that you are delivering a penetration  
test that's
manually intensive and produce a report that is based on nessus  
scans.  That
is where my frustration with this industry comes into play. That  
happens too
often and that is selling someone a false sense of security.

Anyway, I am supposed to be on vacation right now... so I really  
should get
off the computer... my wife will kill me. :)

> ------------------------------------------------------------------------
> This list is sponsored by: Information Assurance Certification  
> Review Board
>
> Prove to peers and potential employers without a doubt that you can  
> actually do a proper penetration test. IACRB CPT and CEPT certs  
> require a full practical examination in order to become certified.
>
> http://www.iacertification.org
> ------------------------------------------------------------------------



        Adriel T. Desautels
        ad_lists () netragard com
        --------------------------------------

        Subscribe to our blog
        http://snosoft.blogspot.com


------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified.

http://www.iacertification.org
------------------------------------------------------------------------