RE: The economics of testing

[Craig Wright <Craig.Wright () bdo com au>]

30% slice for pentesting was a bit high
It is - the 30% is just for internal audit against pen testing, not monitoring and the like as well.

"an ecomonic-neutral way to assign weighted value to the various aspects of a security program to correlate to end-term 
real-world security"

This is the issue, you can not do this. The only way to make a valid security metric is using an economic model. The 
time and effort adding one control is time and effort taken from another.

The issue is people try to take isolated measurements. This is wrong. This is not a valid means of taking metrics. Any 
time people choose the metrics, they measure perception and not reality. There are many quantitative methods for making 
decisions on metrics. This can be PCI, CART, LDAs right up to random forest ensembles, but there are ways.

And the scary side note... Marketing is WAY more advanced than IT at this.

I started in engineering. Though many in IT call themselves engineers, it is not something that many understand. Along 
these lines, there is a REALLY good engineering handbook, "Systems engineering and analysis" by B. Blanchard & W. 
Fabrycky. It is now in the 4th Ed. (2006) but was first published in 1980. It relates general engineering and not just 
IT - though it can be applied.

I have (amongst other things) a 3-4 year plan to replicate their effort in a way that is aligned to IT and in 
particular security and operational control.

The issue for most is that being engineering, there is lots of calculus.

Regards,
Craig Wright GSE LLM


Craig Wright
Manager, Risk Advisory Services

Direct : +61 2 9286 5497
Craig.Wright () bdo com au
+61 417 683 914

BDO Kendalls (NSW-VIC) Pty. Ltd.
Level 19, 2 Market Street Sydney NSW 2000
GPO BOX 2551 Sydney NSW 2001
Fax +61 2 9993 9497
http://www.bdo.com.au/

The information in this email and any attachments is confidential. If you are not the named addressee you must not 
read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
this message in error, please notify the sender by return email, destroy all copies and delete it from your system.

Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
bdo com au.

BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
under Professional Standards Legislation.
-----Original Message-----

From: Erin Carroll [mailto:amoeba () amoebazone com]
Sent: Saturday, 7 June 2008 2:11 AM
To: Craig Wright; 'Adriel Desautels'
Cc: 'Scott Race'; security-basics () lists securityfocus com
Subject: RE: The economics of testing

Craig, thanks for the response. Some comments below.

"Economics can not be ignored. We live in a world with limits. To speculate
on a world without economic constraints is on one where there is no shortage
of anything. All people have anything they want any time.

This is fantasy. You may as well ask "what if dragons where real?" Who
cares? I have too many real world things to consider to bother with
fantasy."

I'm not ignoring the economics of this mental exercise. What I was trying to
drive at was an ecomonic-neutral way to assign weighted value to the various
aspects of a security program to correlate to end-term real-world security.
i.e. which areas of a security program deliver the best security end result,
pentest? VA? Policy & process? Code audit? Etc. Obviously the economics will
skew what aspect gets the most impetus in practice and which aspect provides
the best ROI. But if all aspects are not equal in value to achieving a
secure infrastructure sometimes a higher investment will return the better
result for overall security.

If anything, I thought your 30% slice for pentesting was a bit high. Pen
testing is definitely valuable but of all the more secure infrastructures
I've seen have one thing in common; effective monitoring (logs, access,
traffic). If you don't know and review constantly what is happening in your
network/systems there is no way to know what is good or bad and how to
respond appropriately.


--
Erin Carroll
Moderator, SecurityFocus pen-test mailing list
amoeba () amoebazone com
"Do Not Taunt Happy-Fun Ball"