Re: Octave

[George Capehart <capegeo () opengroup org>]

Dave Aitel wrote:

<snip>

Apologies in advance for pulling the topic away from OCTAVE specifically
to problems with risk assessments and the risk management process in
general . . .

> """
> There are many approaches for evaluating information security risk. At
> the heart of any approach is an assessment, or evaluation. This slide
> defines two common approaches: a tool-based analysis and
> workshop-based analysis.
> The tool-based analysis normally requires someone to input information
> about the organization?s assets, threats, and infrastructure
> characteristics into a software-based analysis tool. The tool takes
> the information and performs a risk analysis, often based on
> proprietary mathematical algorithms. There are usually no restrictions
> on who enters the information into the process (often it is a small
> group of people) or on how they collect the required information. The
> interaction and number of people required by this type of analysis is
> small.This approach can be quick (after the initial information is
> entered into the tool), but it relies on only a few perspectives. The
> organization is also placing trust in proprietary analysis algorithms
> that might not be well understood by the organization?s personnel.

Nor is it very likely that the tool even uses the appropriate metrics or
is even sensitive to the appropriate dimensions.  The first phase of a
true risk assessment should be to identify the aspects of the entity
that need to be protected, and then understand the threats to those
aspects and the vulnerabilities to those threats.  Any third-party
cookbook tool will cover the "common" cases, but will miss the
idiosyncratic cases . . . which are frequently aspects that are
strategic differentiators . . . and therefore the ones that need
protecting the most.

> A workshop-based analysis requires the participation of many people to
> build an understanding of assets, threats, and characteristics of the
> infrastructure. A small group of people (an analysis team) leads the
> process and gathers information using interviews or workshops. The
> analysis team reviews and analyzes the information that has been
> gathered and creates mitigation plans. Decision-support tools can be
> used to assist the analysis team, but the analysis team is responsible
> for making all decisions. This approach involves many staff members in
> the organization and can be time intensive. However, the people in the
> organization make the decisions and understand why the decisions have
> been made.
> OCTAVE is a workshop-based approach.
> """
>
> We did a number of these at @stake, and I personally didn't find them
> to be of value. Workshops have a number of built in problems:
> o People lie to you. Often, people won't know the answers at all, but
> will still pretend to to look good. In many cases you will get
> conflicting information simply because people don't really know what
> they're talking about. You can spend forever tracking down the truth
> here. What this means is that at the end of the process you don't have
> hard evidence and you don't know how reliable your results are.
> o Workshops are hugely expensive for what they produce. You're trying
> to get a meeting with the CSO, CISO, CEO, various levels of
> management, and the actual technical staff. This involves a huge
> amount of effort even for a small organization, and is typically going
> to be not worth it. The loss of productivity is mind boggling when you
> add it up.
> o Workshops draw weak conclusions. I'm not sure why this is, but my
> experience with them tells me that overall, we didn't end up telling
> people anything they didn't know. A good process will, sometimes at
> least, produce results that surprise you. Workshops never will.
> Perhaps consensus based brainstorming is not a replacement for
> leadership or individual knowledge.

In other words, workshops rarely involve the individuals in the
organization whose job it is to manage risk.  And it's been my
experience that outside the financial services industry, there are few
organizations which have a formal risk management process, and even in
financial services, the formal risk management process rarely includes
information security risk.

> So to sum up: I feel that OCTAVE and things like it are a huge waste
> of time. This might not be the answer you were hoping for, but it's my
> opinion based on having done things like it and having read the
> materials presented on the website.

Much like the Certification and Accreditation Process.  The idea is
great: theoretically, it forces management to understand the risks and
formally (in writing) sign off the controls being implemented and accept
the residual risk.  In practice it's turning out to be a waste of time
and money because it's frequently implemented by people who don't
understand the risk management process, but who are very good at
creating punchlists . . .