Re: [PEN-TEST] Cost of Penetration Testing

[Alfred Huger <ah () SECURITYFOCUS COM>]

On Tue, 12 Sep 2000, Christopher M. Bergeron wrote:

> The cost of the test would be dependent on the skills of the tester.
> In my opinion, the overhead cost for such a test is relatively low
> (for commercial scanners, free scanners, etc).  I also tend to think
> that you get what you pay for (please don't flame, I know that there
> are a lot of overcharging, commercial scan only type pen-tester
> companies out there).  The cost the company will charge you will vary
> depending on many factors:  If they have a programming staff to write
> custom scan-type software;  If they have "professional" (aka, not
> cheap) pen-testers on staff; and if they deal with larger clients or
> smaller clients, etc...  If banking is your livelyhood (and
> considering what the public perception of your bank would be if it
> were ever hacked) I would probably elect to have multiple pen-tests
> performed by different companies.  Each company may approach it
> entirely differently and the more you test the better off you'll be.
> Of course, you'll have to do the cost/benefit analysis yourself
> (unless you can easily afford 1000+ pen-tests, har har).
>
> Please understand that this is just my opinion on the subject, and I'm
> relatively certain that you'll receive many other points of view from
> this list...

 Well, I will break with tradition here and talk about the 'cost' of an
audit. It irks me that consultants treat this subject like a holy grail of
sorts. Everyone jealously guarding their rates like shiny treasures.

 We have a quarterly audit arrangement here, meaning we get audited from
top to bottom once a quarter. The timing was layed out this way because of
our requirement for fairly regular auditing. Unfortunately because of who
we are and what we do there are *alot* of people out there determined to
twist our doorknobs. Furthermore alot of the people at our front door are
not script kiddies. We end up with some pretty sophisticated people
lurking outside our house.

 So, having said all this we layed out the plan for our ongoing audits to
be supplemented with our own in-house work (keeping up w/ BUGTRAQ et al.).
The audit covers the following in terms of locations:

 1. Our website and all of it's connected bits. File servers, audio/video
servers, routers redundant systems etc.

 2. Our operations site in Canada, our Business development site in
California. This work included again, all machines available to an outside
user. Firewalls, DNS servers et al.

 In terms of the 'audit' work included:

 1. Source review - OpenSource Products. All the open source products we
deploy Internet side were reviewed for vulnerabilities. Meaning, our
auditors poured through the source line by line and searched for holes in
our software.

 2. Source Review - In-house Products. We have built a number of products
in house which are not yet on the market. They were reviewed in the same
manner as the as Open Source packages. Line by line looking for the
mistakes we had made.

 3. Blackbox review. We run several proprietary software packages Internet
side and these were reviewed for vulnerabilities. Not with a commercial
scanner or a freeware scanner but with individual test plans per package.

 4. Systems review. This was the standard auditing our boxes for *known*
vulnerabilities. This was done with a collection of free packages
complimented by a single commercial scanner. To be frank, this was the
least of our concerns given that we stay pretty much up to date with
vulns. If we did not, we would have mno oney to pay for an audit :>

 5. Internet policy review. Before the engagement I sat down with the team
lead and described what a user on our site should be able to do. We
defined very clearly acceptable use limits. With that information the team
we hired vetted our site(s) and defined how close we came to reaching
those standards.

 In terms of penetration testing:

 The work in the audit portion of the engagement set the groundwork for an
actual penetration test. The scope of this was simple. Break into whatever
you can in all of the above sites and tell us what you can access
internally and what you can steal. There was alot of conversation around
this particular point but that is the essence of the directive.

 Now, having said all this, cost was not the most important factor for me.
It was skill and finding auditors capable of doing this type of depth
intensive audit. This is *nightmarishly* difficult. As you all know there
are alot of security consulting houses out there. In my opinion the vast
majority of them are incapable of doing an audit outside of running
commercial and freeware scanners.

However, I did find and settle on one and the price tag when it was all
said done was $80,000 and in my opinion worth every single cent. In fact
it's pretty cheap given the amount of time and labour that went into this
engagement.

-al


Alfred Huger
VP of Engineering
SecurityFocus.com