Penetration Testing mailing list archives
Re: [PEN-TEST] Evaluating Auditors Abilities
From: "Khan, Mansoor" <Mansoor.Khan () INVESTORSGROUP COM>
Date: Fri, 8 Sep 2000 09:08:33 -0500
I have seen some interesting comments shared on this topic and I agree with most of them. The issue of how to evaluate auditors abilities/competency level is an interesting one. In fact this is not only true for auditors but also for other professionals who are willing to provide their professional services to your organization. We have already seen responses on how we should go about hiring some one for pene testing or other engagements of this sort e.g. performing reference checks, reviewing their credentials (resume etc.), their clientele, etc. One thing that I would like to point out is that auditors or other security consultants are professionally required (though not mandatory) to share their findings with their auditees before reporting on them. This step is considered to be a best practice. Being a professional auditor myself, I know we have always done this with our auditees. In some cases we also issue draft reports and issue it only to the concerned individuals for discussions. There is a huge misconception that the more findings you have the better auditor you are. During the last two years, I have had the experience of hiring three different consulting firms for assignments relating to security. The selection was made on the reputation of the firms, the professional abilities of the individuals in their team, and reference checks. Two of these three firms are still in top 5 accounting firms. The thrid one is a small security consulting firm out in Alberta, Canada. One of the top five accounting firm that I had to deal with was BAD, to the extent that we had to tell them about their weaknesses and the root cause of their findings. My impression was that it is not the firm that was bad but the individuals working on the team. However, my experience with the other two team was not bad at all. The individuals working for their teams were very professional and knew about all the security issues and were aware of all the false positives that these pene testing tools spit out. The reason I am sharing my experience with all of you is that you need to do your best to ensure that you hire the right person but there are no guarantees. The only thing you can do to minimize the situation that you had to encounter is to have "exit meetings" with your auditors/security consultants and provide them with your feed back. I hope this helps. PS: David: There is nothing wrong with hiring auditors for security engagements. What matters is the individual that you are hiring should have the appropriate knowledge and experience. MK
---------- From: Derrick[SMTP:Derrick () ANEI COM] Sent: Wednesday, September 06, 2000 11:46 PM To: PEN-TEST () SECURITYFOCUS COM Subject: Evaluating Auditors Abilities Dear Pen-Testers, Recently I underwent something that had me thinking about Security Auditing companies and others (Big accounting firms that offer a side service of auditing). Management decided that we needed to be audited by an outside firm, which I am in full favor of. The problem came about in what an un-named auditor did. Firewalls tend to cause false positives in some tests and other anomalies that many auditors may not be aware of. So they performed this audit which we did pick up and were aware of. What happened next is what baffles me. The auditors did not understand the results that nmap and other tools gave them. Near the end of the business day they contact management proclaiming they have found numerous security issues and even some backdoors in our network. After a long couple of days of testing we found none of these issues were correct, and we then spent many hours and several meetings explaining that the firm hired didn't seem to know what they were doing. Management made the default comment of "We are paying them a lot so they must be right, fix these problems". After several days of explaining why they results were wrong and verifying the network we came out to show that the auditors did in fact improperly interpret the results. The end result is management walks away wondering if they got ripped off or if we were just trying to cover problems. It also caused a lot of overtime and extra work for us to explain and prove the network to management. So the end questions are these. How can companies decide which auditors really do a decent job and are worth their value ? Are there any certifications or Industry groups out there or on the horizon that will evaluate and endorse auditors ? What is the best approach from a Network Admin position to counter end results delivered by auditors if they seem to be in error ? Has anyone else been through this, and is destined to get worse before getting better ? Thanks for any thoughts or comments, Derrick
Current thread:
- Re: [PEN-TEST] Evaluating Auditors Abilities Emeigh, Mike (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities topher hughes (Sep 08)
- <Possible follow-ups>
- Re: [PEN-TEST] Evaluating Auditors Abilities Tansey, Don (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Benjamin P. Grubin (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Kuss, Kenneth (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Edward Slusarski (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities David Hopkins (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Khan, Mansoor (Sep 08)
- Re: [PEN-TEST] Evaluating Auditors Abilities Meritt, Jim (Sep 08)
- Re: [PEN-TEST] Evaluating Auditors Abilities Dunker, Noah (Sep 08)
- Re: [PEN-TEST] Evaluating Auditors Abilities Gallicchio, Florindo (2282) (Sep 08)
- Re: [PEN-TEST] Evaluating Auditors Abilities Hill, Mark (Sep 08)