Security Basics mailing list archives
RE: Getting the value of an asset and the probability of a risk to it
From: "Murda Mcloud" <murdamcloud () bigpond com>
Date: Tue, 20 May 2008 09:00:16 +1000
Craig, I'm trying hard to get my head around this(as much as I possibly can it's been almost 17 years since I did stats at Uni and I may have been asleep for one of those lectures):
I am sorry, how did you make this up? I see no basis in reality. I also see that you have factored this into a single dimension. Wrong. You need all the data. For a start: Type of industry Location Traffic volume and patterns Router and firewall rulesets
So as with most things, the more data there is available, the greater degrees of accuracy can be sifted from that data. Not only that, is this a more 3 dimensional approach to that 'high school' model? Ie, there are several ways of processing the data-sets and not just in a linear fashion. So the one of the important things is to consider and be aware of the variables that are being added over a period of time that forms the basis of the historical data set? Not only that, be aware of the number of variables present in any measurements taken? The thing that no doubt seems counter intuitive to many, is that the 'future' data and factors appear to be magically produced. At least, that's how I read Jon's post and have heard similar arguments from others. This is not to say that the methods you mention do not produce accurate estimates. Even that sounds counter intuitive-but it isn't, I guess ;-)
-----Original Message----- From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Craig Wright Sent: Sunday, May 18, 2008 10:25 AM To: Jon.Kibler () aset com; Sergio Castro; security- basics () securityfocus com; Rivestp () metro ca Subject: RE: Getting the value of an asset and the probability of a risk to it In theory and practice. How do you model historical data? Well the answer is multivariate means. Longitudinal data analysis. Jon, your whole basis of what you are considering probability theory is starting from a flawed foundation and is being built without substance. The chance based theory you are basing your determination is from the 17th century (literally). Although we still teach this in high school, it is not the basis of a modern curriculum in statistics. The how is the same how used in heteroscadestic financial modelling, biostatistics and similar disciplines. You are starting with a qualitative assumption. You have in your own mind decided on the risk factors. As I specified - you can not do this. This is another flaw in understanding quantitative methods. You need to use a dimensional reduction technique and allow the data itself to determine the correlative factors. "For example: Historically, the chances of a Windows box on a secure network getting rooted were less than 1 in 100,000." I am sorry, how did you make this up? I see no basis in reality. I also see that you have factored this into a single dimension. Wrong. You need all the data. For a start: Type of industry Location Traffic volume and patterns Router and firewall rulesets (and it is easy to feed these into a correlation engine) ... As for factoring video card bios root kits, I have done this for many years. I did a paper on the use of ARIMA (autoregressive integrated moving average) methods for the prediction of malware a couple years ago. Although people such as yourself scoff at this type of modelling, my predicted model is still accurate after 2 years (based on a 95% confidence). " Except perhaps for risks associated with Mother Nature. And with climate change" Please Jon, are you kidding. IT risk is simple compared to weather modelling. The dimensionality in IT risk comes at most in the order of 60-100 factors. Weather modelling comes in the 10's of thousands. The problem with this type of attitude is that you see this as hocus pocus just as you do not understand it. Yet the maths is the same in many cases as that which allows a GPS to not drift the minutes a day that relativity theory dictates it must due to velocity differentials to earth. It is the same that allows your phone to work. You can not make the dimensional reduction to a windows host has a 1 in x chance of being compromised. You need to model EACH host. Workstations in network A, Servers on DMZ with config A, Servers on DMZ with config A that are patched a week later, Workstations on a hub Workstations on a switch Workstations on the same network as a win 95 box ... As I stated, this type of modelling is not cheap. Doing is not hard, it just requires more maths than most have. In fact I have the problem of getting staff for this reason. I had a grad, Zac, a year ago. He left as one of the investment banks offered him 150% of my salary. Now he models hedge funds. Most end up doing BI (Business Intelligence) modelling for banks and telcos to predict client churn. Same maths, but IT people with maths are rare. I am not talking B.Sc. I mean a good post grad research math degree. In Australia, we produce less than 250 of these per year. Of those, in any field of IT there is about 5% - and most of this goes to bioinformatics. So is there a great volume of quant snakeoil. Answer as yes and you are correct. The issue is that few can do the maths to see if it works. How do you tell what is real. Well look at track records. Those who are willing to publish their models and who have a track record over the years and can be validated etc are more likely to keep doing this. Those who refuse to publish their models and algorithms as they are "proprietary" are basically snake oil sales organisations. As for future aspects, my models take EVERYTHING into account and I let a dimensional reduction method choose those factors that have a statistically significant effect remain. As an example, I am already factoring the impact of 3d printing technology on IP (intellectual property) protection. "how do you base risk on historical data" Again, you are thinking high school stats. I have pointed out a few methods. LDA and other methods are used for missing data projections. These have been around for 15 years or so now and have proven themselves. I have been teaching these to my data analytic team as even in University, most do not learn them. Just as most people do not know grad level statistics, does not make it magic. Multivariate data analysis using Bayesian techniques accounts for the gaps in data. What you get is a range and confidence interval. As an example, a calculation would provide something of the type (based on real data): System Expected Risk at 95% CI Windows host A (patching daily) $3,521 - $4,210 Windows host A (patching weekly) $5,422 - $6,585 Windows host A (patching monthly) $13,895 - $15,510 System Expected Risk at 99% CI Windows host A (patching daily) $3219 - $4512 Windows host A (patching weekly) $5002 - $6905 Windows host A (patching monthly) $20275 - $22130 The trade off is that the more accurate the confidence level, the wider the range. What this then allows is a determination of the benefits. For instance, if the Windows host A cost estimate at a 95% CI is set daily at $35 (+/- $2.50) we have a years cost range for daily patching of ($11862.50, $13687.50). SO we are 95% confident that patching the system on a daily basis will cost us between $11,862.50 and $13687.50. The calculated costs of patching weekly are ($4,225, $5,362.50) The calculated costs of patching monthly are ($1,482.20, $1,596.21) So looking at the expected benefits: System Cost of patching (CI = 95%) Windows host A (patching daily) $16,640.50 (+/- $1,257.00) Windows host A (patching weekly) $10,797.25 (+/- $1,150.25) Windows host A (patching monthly) $16,241.71 (+/- $864.50) So we see that the additional effort to patch the system for this organisation daily is a cost. That doing this less than monthly is a cost. So the best (lowest cost) strategy is to patch weekly. The results where statistically significant at the alpha=5% level for a determination that the effort to patch daily would cost more than it saved. Equally, the cost "savings" of patching the system on a monthly basis added additional risk. If the client had wanted to pay more we could have modelled this to the inflection point and determined the exact benefits, but the model was not significantly better than the simple model in any event and did not justify the cost addition. (So Matt and others at iDefense, MacAfee, the Certs etc, this is what I do with that zero day data.) Regards, Dr Craig Wright (GSE-Compliance) PS 3d printers (or rapid prototypers are available now if you have enough money) Craig Wright Manager, Risk Advisory Services Direct : +61 2 9286 5497 Craig.Wright () bdo com au +61 417 683 914 BDO Kendalls (NSW-VIC) Pty. Ltd. Level 19, 2 Market Street Sydney NSW 2000 GPO BOX 2551 Sydney NSW 2001 Fax +61 2 9993 9497 http://www.bdo.com.au/ The information in this email and any attachments is confidential. If you are not the named addressee you must not read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received this message in error, please notify the sender by return email, destroy all copies and delete it from your system. Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () bdo com au. BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved under Professional Standards Legislation. -----Original Message----- From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Jon Kibler Sent: Saturday, 17 May 2008 12:01 PM To: Sergio Castro; security-basics () securityfocus com; Rivestp () metro ca Subject: Re: Getting the value of an asset and the probability of a risk to it -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sergio Castro wrote:Hi Philippe, The only true way of doing a quantitative risk assessment on an assetis using statistics. <SNIP!> In theory, yes. In reality, it just doesn't work that way. For example: Historically, the chances of a Windows box on a secure network getting rooted were less than 1 in 100,000. But if you use that as a basis for computing future risk, I would argue that the historical data has absolutely zero to do with reality today or in the future. I would suspect that within the next 12 to 24 months, the chances of a Windows box on a secure network getting rooted are about 1 in 1,000. So, if you use statistics based on historical data, your risk assessment is off by two orders of magnitude! (These numbers are for illustrative purposes only! I just created these numbers by AE, but they are probably within an order of magnitude of being correct.) So, when projecting risk for the next 5 years, from where do you get the data to form your statistical basis for risk? Another example: A couple of years ago I heard Gadi Evron talk about hardware rootkits (in BIOS, Video NRAM, NICs, Routers, etc.). Most people laughed at the idea. And now, what is the big anticipated talk at EusecWest? IOS Rootkits. Again, how do you base risk on historical data, or do any type of risk modeling when historical data is not applicable today and no one has a reasonable guess for the future? To use statistics, it has to be based on data. When historical data is not representative of current / future risk, it is not a valid basis for forming statistical projections -- of risk, or anything else for that matter. As I said previously, it is essentially impossible in today's I.T. security environment to do quantitative risk assessment that stands any chance of passing the laugh test. Except perhaps for risks associated with Mother Nature. And with climate change, who knows how accurate those data will be? Jon Kibler - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkguPEgACgkQUVxQRc85QlPoMwCfdNcxLlGPl9s5PtJImaEuNHXl FPEAoKCKXtcpYCFdXUM5Z4MSchxGR1Wm =p9hK -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
Current thread:
- Basic Computer Security Advice Needed Mark Goodridge (May 16)
- RE: Basic Computer Security Advice Needed Sergio Castro (May 16)
- Getting the value of an asset and the probability of a risk to it Rivest, Philippe (May 16)
- RE: Getting the value of an asset and the probability of a risk to it Sergio Castro (May 16)
- Re: Getting the value of an asset and the probability of a risk to it Jon Kibler (May 17)
- RE: Getting the value of an asset and the probability of a risk to it Craig Wright (May 18)
- RE: Getting the value of an asset and the probability of a risk to it Murda Mcloud (May 20)
- RE: Getting the value of an asset and the probability of a risk to it Craig Wright (May 21)
- Getting the value of an asset and the probability of a risk to it Rivest, Philippe (May 16)
- RE: Getting the value of an asset and the probability of a risk to it Craig Wright (May 18)
- RE: Getting the value of an asset and the probability of a risk to it Rivest, Philippe (May 20)
- RE: Getting the value of an asset and the probability of a risk to it Craig Wright (May 21)
- RE: Basic Computer Security Advice Needed Sergio Castro (May 16)
- Re: Getting the value of an asset and the probability of a risk to it Jon Kibler (May 16)
- Re: Basic Computer Security Advice Needed Gustavo V G C Rios (May 18)
- Re: Basic Computer Security Advice Needed Gleb Paharenko (May 20)
- <Possible follow-ups>
- Re: Basic Computer Security Advice Needed rah . wollongong (May 19)