Re: [PEN-TEST] Evaluating Auditors Abilities

[Domenico De Vitto <dom () DEVITTO DEMON CO UK>]

Easy, use at least TWO companies, that's what we do.

If you let them know that there is competing auditing company, it means that
they'll be thorough.  If you get 3 companies, then if one is crap, the
other two will give you evidence to the contrary.  In practice, companies
use a couple of 'known good' companies on a schedule, and bring in a 3rd
just
to check if the other two are still the best two.

Another tack, is to say that you think they are a fraud, in short.

Ask them to use the information they have to read c:\secret.txt on the
server with the big holes. For an extra touch, get the boss to type in the
secret text himself.

He'll be amazed, and expecting them to succeed, when they 'fail' they'll
look daft.  If they refuse to co-operate just suggest that you look for a
better company, and the test be "test this server and read c:\secret.txt".

When the next company, and the one after that, fails, mail the boss to
remind him that his secret is still safe, so you guess the first company was
crap.


Dom

-----Original Message-----
From: Penetration Testers [mailto:PEN-TEST () SECURITYFOCUS COM]On Behalf
Of Derrick
Sent: 07 September 2000 05:46
To: PEN-TEST () SECURITYFOCUS COM
Subject: [PEN-TEST] Evaluating Auditors Abilities


Dear Pen-Testers,

        Recently I underwent something that had me thinking about Security Auditing
companies and others (Big accounting firms that offer a side service of
auditing). Management decided that we needed to be audited by an outside
firm, which I am in full favor of. The problem came about in what an
un-named auditor did. Firewalls tend to cause false positives in some tests
and other anomalies that many auditors may not be aware of. So they
performed this audit which we did pick up and were aware of. What happened
next is what baffles me. The auditors did not understand the results that
nmap and other tools gave them. Near the end of the business day they
contact management proclaiming they have found numerous security issues and
even some backdoors in our network. After a long couple of days of testing
we found none of these issues were correct, and we then spent many hours and
several meetings explaining that the firm hired didn't seem to know what
they were doing. Management made the default comment of "We are paying them
a lot so they must be right, fix these problems". After several days of
explaining why they results were wrong and verifying the network we came out
to show that the auditors did in fact improperly interpret the results.
        The end result is management walks away wondering if they got ripped off or
if we were just trying to cover problems. It also caused a lot of overtime
and extra work for us to explain and prove the network to management. So the
end questions are these.

How can companies decide which auditors really do a decent job and are worth
their value ?
Are there any certifications or Industry groups out there or on the horizon
that will evaluate and endorse auditors ?
What is the best approach from a Network Admin position to counter end
results delivered by auditors if they seem to be in error ?
Has anyone else been through this, and is destined to get worse before
getting better ?

Thanks for any thoughts or comments,
Derrick