Re: [PEN-TEST] Evaluating Auditors Abilities

[Deri Jones <Deri.Jones () NTA-MONITOR COM>]

At 10:56 07/09/00 -0700, Mark wrote:
> 1. Management got ripped off by what you describe

Yup!  We've heard stories like this so often - poor testing at inflated
prices
from 'big name' suppliers...  It's the norm, not the exception, I would say
:<(

> 2. Many auditing firms unless they present their credentials will just run
> the typical commercially available toolsuite plus a couple of hobbled
> together tools, produce a nice report and never validate the results

Yup

> ...snipped
> 9. No there are no certifications or Industry Groups that monitor or
> endorse the auditors.

Actually this is not quite true - in Europe, the UK Government seems to be
unique in having a certification scheme for testers - maned by the govt
body 'Communications Electronics Security Group' (www.cesg.gov.uk) - the
scheme is called the CHECK scheme.

We were one of the founding members back in January 1999.
(Here at NTA Monitor we do more pen testing than any other company in
Europe (eg over 520 test assignments in 1999, over 200 clients on current
quarterly or monthly test contracts - from all 5 continents).)

Now whether the CHECK scheme is a high enough quality standard- humm?...  It
was quite a debate in the set-up phase - just how high to set the bar.  We
think it's too low...but we're just one voice...

But, using a CHECK member *does* provide some level of assurance.

But the *most* important thing you can do
-----------------------------------------
 - is ask the vendor for a *list* of customers - if they can't show they've
done work for say 30 or 50 companies - then have they really got the
experience?  Once you've got the list, you can select the references *you*
want to speak to - not just call the 2 or 3 names the vendor offers  - it's
easy to have 2 happt customers!

One thing that differentiaites us from the 'Big 5's of this, is we have
customer lists on our web site and in our literature - and we invite
new prospects to choice their choice of who they want to call.

Maybe that's why we are picking up a rush of new clients who are dropping
the Big 5..., and the like :<)

Deri Jones
NTA Monitor


> ANyone with enough money and political saavy can
> open up shop (whether you are name or not), invest some money in a fancy
> web site, claim to have all the vulnerabilities and exploits, and provide
> cruddy service, but are backed by large VC..
>
> /mark
>
> At 12:46 AM 9/7/00 -0400, Derrick wrote:
> > Dear Pen-Testers,
> >
> >         Recently I underwent something that had me thinking about
> > Security Auditing
> > companies and others (Big accounting firms that offer a side service of
> > auditing). Management decided that we needed to be audited by an outside
> > firm, which I am in full favor of. The problem came about in what an
> > un-named auditor did. Firewalls tend to cause false positives in some tests
> > and other anomalies that many auditors may not be aware of. So they
> > performed this audit which we did pick up and were aware of. What happened
> > next is what baffles me. The auditors did not understand the results that
> > nmap and other tools gave them. Near the end of the business day they
> > contact management proclaiming they have found numerous security issues and
> > even some backdoors in our network. After a long couple of days of testing
> > we found none of these issues were correct, and we then spent many hours and
> > several meetings explaining that the firm hired didn't seem to know what
> > they were doing. Management made the default comment of "We are paying them
> > a lot so they must be right, fix these problems". After several days of
> > explaining why they results were wrong and verifying the network we came out
> > to show that the auditors did in fact improperly interpret the results.
> >         The end result is management walks away wondering if they got
> > ripped off or
> > if we were just trying to cover problems. It also caused a lot of overtime
> > and extra work for us to explain and prove the network to management. So the
> > end questions are these.
> >
> > How can companies decide which auditors really do a decent job and are worth
> > their value ?
> > Are there any certifications or Industry groups out there or on the horizon
> > that will evaluate and endorse auditors ?
> > What is the best approach from a Network Admin position to counter end
> > results delivered by auditors if they seem to be in error ?
> > Has anyone else been through this, and is destined to get worse before
> > getting better ?
> >
> > Thanks for any thoughts or comments,
> > Derrick