Re: Experience with Risk Assessment tools, such as RiskWatch?

["Mclaughlin, Kevin L (mclaugkl)" <mclaugkl () UCMAIL UC EDU>]

I can only second what Jim, James and Brad have to say.  In my
experiences in using a variety of the Risk tools and methodologies
available I keep coming back to what my friend Tom Peltier (author of
Information Security Risk Analysis) keeps saying - "at the end of the
day Risk Analysis is a qualitative event" based on the experience of the
manager doing the analysis. 

-Kevin


Kevin L. McLaughlin
CISM, CISSP, PMP, ITIL Master Certified
Director, Information Security
University of Cincinnati
513-556-9177 (w)
513-703-3211 (m)
mclaugkl () ucmail uc edu
 
  
 
   
CONFIDENTIALITY NOTICE: This e-mail message and its content is
confidential, intended solely for the addressee, and may be legally
privileged. Access to this message and its content by any individual or
entity other than those identified in this message is unauthorized. If
you are not the intended recipient, any disclosure, copying or
distribution of this e-mail may be unlawful. Any action taken or omitted
due to the content of this message is prohibited and may be unlawful.
 
-----Original Message-----
From: Jim Dillon [mailto:Jim.Dillon () CUSYS EDU] 
Sent: Wednesday, November 29, 2006 1:07 PM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] Experience with Risk Assessment tools, such as
RiskWatch?

I'll push Brad's comments a little farther.

Having tried a number of methods and having reviewed many more, I'm
personally unimpressed by any of the quantitative methods I've
encountered.  In order for a quantitative assessment to work for a
purely IT oriented assessment, you need to be able to accurately account
for the portion of cost/expense for each process, for the value
contributed to each process (and value becomes a judgment) for every
aspect of the IT infrastructure.  There are those that propose theories
that would argue against this perspective, but having tried to use
various methods, I've found most quantitative efforts to end up in a
"tweaking" session as common sense and qualification based on less
measurable influences take over. ("I know that this is not a greater
risk than that...")  In truth, it is hard to pull out an aspect of risk
(say IT as a component of process) without considering the whole
contribution (environmental, human, process, requirements/ restrictions,
etc.)

If quantitative solutions were enough, we wouldn't need managers.
Managers are used due to the uniquely human ability to observe, intuit,
and absorb the millions of different influences on a process.  And these
same managers are held responsible to determine an acceptable vs. an
unacceptable level of risk given what they know about the possibility
for mitigating risk.  A manager can account for the subtle clues of
employee behavior, most models can't.  Thus I'm negatively inclined to
think a purely quantitative process, computerized or not will be very
successful.  On the other hand, a process providing organization, forms,
ranking for qualified inputs could be of some value.

Most of us refer to risk as some combination of opportunity vs.
consequence, that is "how likely" and "what impact" are the primary
questions.  While some will argue this is simply a component of "how
likely", I tend to think more obscure measures such as prevalence (how
prevalent is this particular factor) are not simply a likelihood
measure, but something less tangible - a measure of cultural
"imbeddedness" to make up a word, that requires the flexibility of mind,
and the responsibility of decision, to resolve.  I find no comfortable
quantification that lends itself to a checklist or numerical measure
here.

Sorry - too long a rant and I'm not addressing your concerns about the
software.  I guess the bottom line for me is for you to evaluate the
effectiveness of the package at gathering and communicating qualitative
information consistently - I think this is the most important attribute
of a risk assessment, I do not trust largely mechanical models. 

Willing to be persuaded,

Jim

*****************************************
Jim Dillon, CISA, CISSP
IT Audit Manager, CU Internal Audit
jim.dillon () cusys edu
303-492-9734
*****************************************
 

-----Original Message-----
From: Brad Judy [mailto:Brad.Judy () COLORADO EDU] 
Sent: Wednesday, November 29, 2006 9:25 AM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] Experience with Risk Assessment tools, such as
RiskWatch?

Implementing an internal risk assessment framework/process is a
significant portion of my duties.  You can see the risk
assessment/management framework we've created here:
http://www.colorado.edu/its/security/itriskmanagement/

We aren't using any third-party software like RiskWatch, so I can't
comment on that part.  

The big initial steps in coming up with any risk assessment is defining
the scope and depth.  These two items, along with the resources at your
disposal, will help set the boundaries for your risk assessments.  Once
you get into performing risk assessments, I think the most critical
aspect is keeping the context of the business, IT environment and true
risks in mind.  Choosing how to address risks is a business decision,
not an IT one - IT merely creates opportunities for risk and risk
mitigation.  

Try not to fall too much into 'checklist' mentality for risk assessment
(this is one thing that worries me about using a software application).
While there are some widely applicable items that can be checklist in
nature, each situation has its own unique twists.  In the end, it's
about how much real risk exists, how much can be mitigated/eliminated
without unrealistic costs (financial or disruptive) and how much risk
can be accepted.  

Finally, quantitative analysis can be tricky.  One example was a school
on one of these lists that noted terrorist attack as a priority for
process improvement because they had no existing plans to handle it.
This bubbled to the top because they used a quantitative method where
the probability of the event was not accurately estimated.  The reality
is that the probability of such an event is extremely low for most
schools, but some may have special situations (like they have an
on-campus nuclear reactor that may be a target).  

Read up on the risk assessment links on our page - we linked to our
source material which is publicly available (some Burton/Gartner
articles were also used - look those up if you have a subscription).

Brad Judy

IT Security Office
Information Technology Services
University of Colorado at Boulder
 

-----Original Message-----
From: James Moore [mailto:jhmiso () RIT EDU] 
Sent: Wednesday, November 29, 2006 8:13 AM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: [SECURITY] Experience with Risk Assessment tools, such as
RiskWatch?

Does anyone have experience with RiskWatch or other tools that help to
quantitatively define risks in information systems?  I have looked at
RiskWatch, and they seem pretty good, but I just haven't found any other
similar tools for comparison.  Any experiences in implementing internal
risk assessments?  RiskWatch also has an ISO 17799 base, so experiences
(and tools) for leveraging that standard in a higher ed environment are
also welcome.
 
Thanks,
 
Jim