RE: Getting the value of an asset and the probability of a risk to it

[Craig Wright <Craig.Wright () bdo com au>]

"So as with most things, the more data there is available, the greater
degrees of accuracy can be sifted from that data."
Correct, a larger volume of data narrows the confidence bounds and increases the accuracy of the result.

"'future' data and factors appear to be magically produced."
Arthur C. Clarke - see law 3
The 3 "laws" of prediction:
1       When a distinguished but elderly scientist states that something is possible, he is almost certainly right.
2       When he states that something is impossible, he is very probably wrong. The only way of discovering the limits 
of the possible is to venture a little way past them into the impossible.
3       Any sufficiently advanced technology is indistinguishable from magic.

"So the one of the important things is to consider and be aware of the variables that are being added over a period of 
time that forms the basis of the historical data set?
Yes, the process for selecting variable in the model should be based on those variables that produce a statistically 
significant effect against the model correlation. There are methods (PCA for instance) for selecting these. Better are 
methods such as GEE.

"Not only that, be aware of the number of variables present in any measurements taken?"
Yes - also look at confounding factors and whether effects are independent or not.


Regards,
Craig


Craig Wright
Manager, Risk Advisory Services

Direct : +61 2 9286 5497
Craig.Wright () bdo com au
+61 417 683 914

BDO Kendalls (NSW-VIC) Pty. Ltd.
Level 19, 2 Market Street Sydney NSW 2000
GPO BOX 2551 Sydney NSW 2001
Fax +61 2 9993 9497
http://www.bdo.com.au/

The information in this email and any attachments is confidential. If you are not the named addressee you must not 
read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
this message in error, please notify the sender by return email, destroy all copies and delete it from your system.

Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
bdo com au.

BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
under Professional Standards Legislation.
-----Original Message-----

From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Murda Mcloud
Sent: Tuesday, 20 May 2008 9:00 AM
To: security-basics () securityfocus com
Subject: RE: Getting the value of an asset and the probability of a risk to it

Craig, I'm trying hard to get my head around this(as much as I possibly can
it's been almost 17 years since I did stats at Uni and I may have been
asleep for one of those lectures):

> > I am sorry, how did you make this up? I see no basis in reality. I also
> > see that you have factored this into a single dimension. Wrong. You need
> > all the data. For a start:
> >        Type of industry
> >        Location
> >        Traffic volume and patterns
> >        Router and firewall rulesets

So as with most things, the more data there is available, the greater
degrees of accuracy can be sifted from that data.
Not only that, is this a more 3 dimensional approach to that 'high school'
model? Ie, there are several ways of processing the data-sets and not just
in a linear fashion. So the one of the important things is to consider and
be aware of the variables that are being added over a period of time that
forms the basis of the historical data set? Not only that, be aware of the
number of variables present in any measurements taken?

The thing that no doubt seems counter intuitive to many, is that the
'future' data and factors appear to be magically produced. At least, that's
how I read Jon's post and have heard similar arguments from others. This is
not to say that the methods you mention do not produce accurate estimates.
Even that sounds counter intuitive-but it isn't, I guess ;-)



> > -----Original Message-----
> > From: listbounce () securityfocus com [mailto:listbounce () securityfocus com]
> > On Behalf Of Craig Wright
> > Sent: Sunday, May 18, 2008 10:25 AM
> > To: Jon.Kibler () aset com; Sergio Castro; security-
> > basics () securityfocus com; Rivestp () metro ca
> > Subject: RE: Getting the value of an asset and the probability of a risk
> > to it
> >
> >
> > In theory and practice.
> >
> > How do you model historical data? Well the answer is multivariate means.
> > Longitudinal data analysis.
> >
> > Jon, your whole basis of what you are considering probability theory is
> > starting from a flawed foundation and is being built without substance.
> > The chance based theory you are basing your determination is from the
> > 17th century (literally). Although we still teach this in high school, it
> > is not the basis of a modern curriculum in statistics.
> >
> > The how is the same how used in heteroscadestic financial modelling,
> > biostatistics and similar disciplines.
> >
> > You are starting with a qualitative assumption. You have in your own mind
> > decided on the risk factors. As I specified - you can not do this. This
> > is another flaw in understanding quantitative methods. You need to use a
> > dimensional reduction technique and allow the data itself to determine
> > the correlative factors.
> >
> > "For example: Historically, the chances of a Windows box on a secure
> > network getting rooted were less than 1 in 100,000."
> >
> > I am sorry, how did you make this up? I see no basis in reality. I also
> > see that you have factored this into a single dimension. Wrong. You need
> > all the data. For a start:
> >        Type of industry
> >        Location
> >        Traffic volume and patterns
> >        Router and firewall rulesets
> >                (and it is easy to feed these into a correlation engine)
> >        ...
> >
> > As for factoring video card bios root kits, I have done this for many
> > years.
> >
> > I did a paper on the use of ARIMA (autoregressive integrated moving
> > average) methods for the prediction of malware a couple years ago.
> > Although people such as yourself scoff at this type of modelling, my
> > predicted model is still accurate after 2 years (based on a 95%
> > confidence).
> >
> > " Except perhaps for risks associated with Mother Nature. And with
> > climate change"
> >
> > Please Jon, are you kidding. IT risk is simple compared to weather
> > modelling. The dimensionality in IT risk comes at most in the order of
> > 60-100 factors. Weather modelling comes in the 10's of thousands.
> >
> > The problem with this type of attitude is that you see this as hocus
> > pocus just as you do not understand it. Yet the maths is the same in many
> > cases as that which allows a GPS to not drift the minutes a day that
> > relativity theory dictates it must due to velocity differentials to
> > earth. It is the same that allows your phone to work.
> >
> > You can not make the dimensional reduction to a windows host has a 1 in x
> > chance of being compromised.
> >
> > You need to model EACH host.
> >        Workstations in network A,
> >        Servers on DMZ with config A,
> >        Servers on DMZ with config A that are patched a week later,
> >        Workstations on a hub
> >        Workstations on a switch
> >        Workstations on the same network as a win 95 box
> >        ...
> >
> > As I stated, this type of modelling is not cheap. Doing is not hard, it
> > just requires more maths than most have. In fact I have the problem of
> > getting staff for this reason. I had a grad, Zac, a year ago. He left as
> > one of the investment banks offered him 150% of my salary. Now he models
> > hedge funds.
> >
> > Most end up doing BI (Business Intelligence) modelling for banks and
> > telcos to predict client churn. Same maths, but IT people with maths are
> > rare. I am not talking B.Sc. I mean a good post grad research math
> > degree.
> >
> > In Australia, we produce less than 250 of these per year. Of those, in
> > any field of IT there is about 5% - and most of this goes to
> > bioinformatics.
> >
> > So is there a great volume of quant snakeoil. Answer as yes and you are
> > correct. The issue is that few can do the maths to see if it works.
> >
> > How do you tell what is real. Well look at track records. Those who are
> > willing to publish their models and who have a track record over the
> > years and can be validated etc are more likely to keep doing this. Those
> > who refuse to publish their models and algorithms as they are
> > "proprietary" are basically snake oil sales organisations.
> >
> > As for future aspects, my models take EVERYTHING into account and I let a
> > dimensional reduction method choose those factors that have a
> > statistically significant effect remain.
> >
> > As an example, I am already factoring the impact of 3d printing
> > technology on IP (intellectual property) protection.
> >
> > "how do you base risk on historical data"
> > Again, you are thinking high school stats. I have pointed out a few
> > methods. LDA and other methods are used for missing data projections.
> > These have been around for 15 years or so now and have proven themselves.
> > I have been teaching these to my data analytic team as even in
> > University, most do not learn them. Just as most people do not know grad
> > level statistics, does not make it magic.
> >
> > Multivariate data analysis using Bayesian techniques accounts for the
> > gaps in data. What you get is a range and confidence interval. As an
> > example, a calculation would provide something of the type (based on real
> > data):
> >
> >        System                                  Expected Risk at 95% CI
> >        Windows host A (patching daily) $3,521 - $4,210
> >        Windows host A (patching weekly)        $5,422 - $6,585
> >        Windows host A (patching monthly)       $13,895 - $15,510
> >
> >        System                                  Expected Risk at 99% CI
> >        Windows host A (patching daily) $3219 - $4512
> >        Windows host A (patching weekly)        $5002 - $6905
> >        Windows host A (patching monthly)       $20275 - $22130
> >
> > The trade off is that the more accurate the confidence level, the wider
> > the range. What this then allows is a determination of the benefits.
> >
> > For instance, if the Windows host A cost estimate at a 95% CI is set
> > daily at $35 (+/- $2.50) we have a years cost range for daily patching of
> > ($11862.50, $13687.50). SO we are 95% confident that patching the system
> > on a daily basis will cost us between $11,862.50 and $13687.50.
> >
> > The calculated costs of patching weekly are ($4,225, $5,362.50)
> > The calculated costs of patching monthly are ($1,482.20, $1,596.21)
> >
> > So looking at the expected benefits:
> >        System                                  Cost of patching (CI =
> > 95%)
> >        Windows host A (patching daily) $16,640.50      (+/- $1,257.00)
> >        Windows host A (patching weekly)        $10,797.25      (+/-
> > $1,150.25)
> >        Windows host A (patching monthly)       $16,241.71      (+/-
> > $864.50)
> >
> > So we see that the additional effort to patch the system for this
> > organisation daily is a cost. That doing this less than monthly is a
> > cost. So the best (lowest cost) strategy is to patch weekly.
> >
> > The results where statistically significant at the alpha=5% level for a
> > determination that the effort to patch daily would cost more than it
> > saved. Equally, the cost "savings" of patching the system on a monthly
> > basis added additional risk.
> >
> > If the client had wanted to pay more we could have modelled this to the
> > inflection point and determined the exact benefits, but the model was not
> > significantly better than the simple model in any event and did not
> > justify the cost addition.
> >
> > (So Matt and others at iDefense, MacAfee, the Certs etc, this is what I
> > do with that zero day data.)
> >
> > Regards,
> > Dr Craig Wright (GSE-Compliance)
> >
> > PS 3d printers (or rapid prototypers are available now if you have enough
> > money)
> >
> >
> > Craig Wright
> > Manager, Risk Advisory Services
> >
> > Direct : +61 2 9286 5497
> > Craig.Wright () bdo com au
> > +61 417 683 914
> >
> > BDO Kendalls (NSW-VIC) Pty. Ltd.
> > Level 19, 2 Market Street Sydney NSW 2000
> > GPO BOX 2551 Sydney NSW 2001
> > Fax +61 2 9993 9497
> > http://www.bdo.com.au/
> >
> > The information in this email and any attachments is confidential. If you
> > are not the named addressee you must not read, print, copy, distribute,
> > or use in any way this transmission or any information it contains. If
> > you have received this message in error, please notify the sender by
> > return email, destroy all copies and delete it from your system.
> >
> > Any views expressed in this message are those of the individual sender
> > and not necessarily endorsed by BDO Kendalls. You may not rely on this
> > message as advice unless subsequently confirmed by fax or letter signed
> > by a Partner or Director of BDO Kendalls. It is your responsibility to
> > scan this communication and any files attached for computer viruses and
> > other defects. BDO Kendalls does not accept liability for any loss or
> > damage however caused which may result from this communication or any
> > files attached. A full version of the BDO Kendalls disclaimer, and our
> > Privacy statement, can be found on the BDO Kendalls website at
> > http://www.bdo.com.au/ or by emailing mailto:administrator () bdo com au.
> >
> > BDO Kendalls is a national association of separate partnerships and
> > entities. Liability limited by a scheme approved under Professional
> > Standards Legislation.
> > -----Original Message-----
> >
> > From: listbounce () securityfocus com [mailto:listbounce () securityfocus com]
> > On Behalf Of Jon Kibler
> > Sent: Saturday, 17 May 2008 12:01 PM
> > To: Sergio Castro; security-basics () securityfocus com; Rivestp () metro ca
> > Subject: Re: Getting the value of an asset and the probability of a risk
> > to it
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Sergio Castro wrote:
> > > Hi Philippe,
> > >
> > > The only true way of doing a quantitative risk assessment on an asset
> > is using statistics.
> > <SNIP!>
> >
> > In theory, yes.
> >
> > In reality, it just doesn't work that way.
> >
> > For example: Historically, the chances of a Windows box on a secure
> > network getting rooted were less than 1 in 100,000. But if you use that
> > as a basis for computing future risk, I would argue that the historical
> > data has absolutely zero to do with reality today or in the future.
> >
> > I would suspect that within the next 12 to 24 months, the chances of a
> > Windows box on a secure network getting rooted are about 1 in 1,000. So,
> > if you use statistics based on historical data, your risk assessment is
> > off by two orders of magnitude! (These numbers are for illustrative
> > purposes only! I just created these numbers by AE, but they are probably
> > within an order of magnitude of being correct.)
> >
> > So, when projecting risk for the next 5 years, from where do you get the
> > data to form your statistical basis for risk?
> >
> > Another example: A couple of years ago I heard Gadi Evron talk about
> > hardware rootkits (in BIOS, Video NRAM, NICs, Routers, etc.). Most
> > people laughed at the idea. And now, what is the big anticipated talk at
> > EusecWest? IOS Rootkits.
> >
> > Again, how do you base risk on historical data, or do any type of risk
> > modeling when historical data is not applicable today and no one has a
> > reasonable guess for the future? To use statistics, it has to be based
> > on data. When historical data is not representative of current / future
> > risk, it is not a valid basis for forming statistical projections -- of
> > risk, or anything else for that matter.
> >
> > As I said previously, it is essentially impossible in today's I.T.
> > security environment to do quantitative risk assessment that stands any
> > chance of passing the laugh test.
> >
> > Except perhaps for risks associated with Mother Nature. And with climate
> > change, who knows how accurate those data will be?
> >
> > Jon Kibler
> > - --
> > Jon R. Kibler
> > Chief Technical Officer
> > Advanced Systems Engineering Technology, Inc.
> > Charleston, SC  USA
> > o: 843-849-8214
> > c: 843-224-2494
> > s: 843-564-4224
> >
> > My PGP Fingerprint is:
> > BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253
> >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.8 (Darwin)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> >
> > iEYEARECAAYFAkguPEgACgkQUVxQRc85QlPoMwCfdNcxLlGPl9s5PtJImaEuNHXl
> > FPEAoKCKXtcpYCFdXUM5Z4MSchxGR1Wm
> > =p9hK
> > -----END PGP SIGNATURE-----
> >
> >
> >
> >
> > ==================================================
> > Filtered by: TRUSTEM.COM's Email Filtering Service
> > http://www.trustem.com/
> > No Spam. No Viruses. Just Good Clean Email.