The economics of testing

[Craig Wright <Craig.Wright () bdo com au>]

"But what if they (economics) were ignored?"

Economics can not be ignored. We live in a world with limits. To speculate on a world without economic constraints is 
on one where there is no shortage of anything. All people have anything they want any time.

This is fantasy. You may as well ask "what if dragons where real?" Who cares? I have too many real world things to 
consider to bother with fantasy.

Finance is simply micro economics. We are bound by limits. The Universe has limits on the speed we can travel. We have 
only so much energy per person on the earth. We have only so many materials. When we get into space and start mining 
Jupitor, this will increase, but supply and demand will bring costs back in line. Everything has a limit.

Time is also a limiting factor, we all have set limits to life. Mind you, the genetic possibilities may be larger, but 
this is sci-fi as yet. Time is money is stated for a reason.

A factor of finance is time. This is where the concept of the time value of money comes into play. When assessing risk 
and possible costs, at least an NPV and IRR calculation needs to be factored.

Pen testing is limited economically. Companies can either go for more low cost testing that rarely finds anything at 
one extreme to an infrequent test by highly skilled individuals at the other. This can range thus from $100 per hour 
people, to $370-600 for the top people. These find more, but less frequently and the number of people who can do this 
are limited.

Limits pose constraints.

Risk should be a quantitative function - in many organisations it is required to be (such as BASELII) quantitatively 
defined even for IT. This means calculus. Either risk is optimised at a inflection point that is a maxima/minima or the 
function is compounded by saddle points. Quantitative does not equal assigning numbers - this is a perception exercise. 
Risk needs to be scientifically calculated within defined confidence levels.

If we take a pen test team with 10 members all working on a large site with 500 hosts, we give them 5 days per host 
(large budget here). The test time is the entire working year. Either a sample is taken or the test takes a year.

This means systems are tested at best yearly.

A full test generally takes more than 5 days for a system so I am being conservative. I do not want to look at "but I 
broke x in 10 minutes" etc.

Security and risk like finance have a time factor. How long it takes for a consultant to come in and test and how 
frequently they do this is important.

Testing is a detective control and a validation. Validation is only effective in 2 ways:
        if the control has a failure - to find and rectify
        to ensure that a control is in place.

Testing is not generally done scientifically. We seek to prove a negative. In Pen testing this can be simple as the 
standard are frequently low enough already. Testing a broken model (a system with poor controls) will lead to 
discoveries. The problem is that it can not state that a system is secure.

A pen test can at best find and exploit a flaw. A worst it can make speculations. In any event, it does not test the 
full compliment of control failures.

The alternatives need to be in place. An infrequent detective control with no preventative controls is less than 
useless. A pen test is only effective in pointing out holes and control flaws. Even with zero days, there is defence in 
depth to be considered.

A good control framework will detect and stop even most zero days far more than it stops many other threats. Good 
logging and monitoring at the host and network level with competent people is a more effective control than pen testing.

I will discover a breach with a combination of integrity checks that run live and database triggers faster than I will 
using pen testing.

Monitoring and baselining network traffic is more cost effective if done correctly than testing.

Yes testing and audit have their place. Their place is to ensure that other controls exist.

Regards,
Craig Wright GSE-Compliance


Craig Wright
Manager, Risk Advisory Services

Direct : +61 2 9286 5497
Craig.Wright () bdo com au
+61 417 683 914

BDO Kendalls (NSW-VIC) Pty. Ltd.
Level 19, 2 Market Street Sydney NSW 2000
GPO BOX 2551 Sydney NSW 2001
Fax +61 2 9993 9497
http://www.bdo.com.au/

The information in this email and any attachments is confidential. If you are not the named addressee you must not 
read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
this message in error, please notify the sender by return email, destroy all copies and delete it from your system.

Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
bdo com au.

BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
under Professional Standards Legislation.