ITL Bulletin for August 2003

[InfoSec News <isn () c4i org>]

Forwarded from: Elizabeth Lennon <elizabeth.lennon () nist gov>

IT SECURITY METRICS
Elizabeth B. Lennon, Editor
Information Technology Laboratory
National Institute of Standards and Technology

Introduction
IT security metrics provide a practical approach to measuring
information security. Evaluating security at the system level, IT
security metrics are tools that facilitate decision making and
accountability through collection, analysis, and reporting of relevant
performance data. Based on IT security performance goals and
objectives, IT security metrics are quantifiable, feasible to measure,
and repeatable. They provide relevant trends over time and are useful
in tracking performance and directing resources to initiate
performance improvement actions.

This ITL Bulletin summarizes the recently published NIST Special
Publication (SP) 800-55, Security Metrics Guide for Information
Technology Systems, by Marianne Swanson, Nadya Bartol, John Sabato,
Joan Hash, and Laurie Graffo. NIST SP 800-55 provides guidance for IT
managers and security professionals at all levels, inside and outside
of government. The document describes the development and
implementation of an IT security metrics program and provides examples
of metrics based on the critical elements and security controls and
techniques contained in NIST SP 800-26, Security Self-Assessment Guide
for Information Technology Systems. Both documents are available at
http://csrc.nist.gov/publications/nistpubs/index.html.

Why Measure IT Security?
Regulatory, financial, and organizational reasons drive the
requirement to measure IT security performance. For federal agencies,
a number of existing laws, rules, and regulations cite IT performance
measurement in general, and IT security performance measurement in
particular, as a requirement.  These laws include the Clinger-Cohen
Act, Government Performance and Results Act (GPRA), Government
Paperwork Elimination Act (GPEA), and Federal Information Security
Management Act (FISMA). In the financial arena, organizations that
measure successes and failures of past and current security
investments can use metrics to justify and direct future security
investments. From an organizational point of view, metrics improve
accountability to stakeholders, ensure an appropriate level of mission
support, determine IT security program effectiveness, and improve
customer confidence.

The Metrics Development Process
The IT security metrics development process consists of two 
major activities:

* Identification and definition of the current IT security 
  program; and

* Development and selection of specific metrics to measure 
  implementation, efficiency, effectiveness, and the impact 
  of the security controls.

The process steps need not be sequential. Rather, the process provides
a framework for thinking about metrics and facilitates the
identification of metrics to be developed for each system. The type of
metric depends on where the system is within its life cycle and the
maturity of the IT system security program. The framework facilitates
tailoring metrics to a specific organization and to the different
stakeholder groups in each organization.

Identify Stakeholders and Interests. Anyone within an organization is
an IT security stakeholder, though some functions have a greater stake
than others: CIO, program manager/system owner, security program
manager, resource manager, and training/human resources personnel.  
Metrics-related roles and responsibilities are dispersed throughout an
organization. Each stakeholder needs a set of metrics that provides a
view of the organization's IT security performance within their needs,
for a total of no more than 5-10 metrics per stakeholder. Many IT
security metrics can be created to measure each aspect of the
organization's IT security. Selecting the most critical elements of
the organization's IT security program during metrics prioritization
will make the program manageable and successful.

Define Goals and Objectives. IT security performance goals and
objectives are expressed in the form of high-level policies and
requirements in many laws, regulations, policies, and guidance that
describe the dimensions of an effective IT security program. These
include the Clinger-Cohen Act, Presidential Decision Directives,
Federal Information Security Management Act (FISMA), OMB Circular
A-130, Appendix III, and NIST Federal Information Processing Standards
(FIPS) and Special Publications. IT security performance goals
identify the desired results of system security program
implementation, while IT security performance objectives enable the
accomplishment of goals.  IT security metrics monitor the
accomplishment of goals and objectives.

Review Current IT Security Policies, Guidance, and Procedures.
Organizations must describe control objectives and techniques that
lead to accomplishing performance goals and objectives. Resources
include the organization's policies and procedures, the Federal Agency
Security Practices Website (http://csrc.nist.gov/fasp), and NIST SP
800-26, Self-Assessment Guide for IT Systems
(http://csrc.nist.gov/publications), which provides many control
objectives and techniques for IT systems.

Review the System Security Program Implementation.  Organizations must
ensure that processes and procedures are in place, existing
capabilities are documented, areas for improvement are noted, existing
metrics are identified, and existing data sources are available that
can be used to derive metrics data. These may be documented in the
following sources (and others): system security plans, OMB Plan of
Actions and Milestones reports, the latest GAO and IG findings,
tracking of security-related activities, and risk assessments and
penetration testing results.

Establish Level of Implementation. The focus of the metrics program
depends on the IT security program maturity within an organization.
Most organizations are new to measuring IT security with performance
metrics. They will begin by measuring the implementation level of
established security standards, policies, and procedures.

Quantify Program Results. As an organization's security program
implementation increases and performance data becomes readily
available, metrics will focus on program efficiency and effectiveness.
Examples include the timeliness of security service delivery and
operational results experienced by security program implementation.

Assess Business/Mission Impact. Business impact can be measured
through correlation analysis once an organization's processes are
self-regenerating and measurement data gathering is transparent.
Examples include business value gained or lost, or an acceptable loss
estimate.

Metrics Development and Selection
The selection of metrics is critical to the success of the program.
Selected metrics must use data that can realistically be obtained from
existing processes and data repositories, and must measure processes
that already exist and are relatively stable. Use output from standard
security activities to quantify IT security performance.  Potential
sources include, but are not limited to, incident handling reports,
testing results, network management logs and records, audit logs,
network and system billing records, configuration management,
contingency planning, training records, and certification and
accreditation.  (NIST SP 800-55, Appendix A, provides sample security
metrics.) When selecting data sources, keep in mind that IT security
metrics data collection must be as automated and non-intrusive as
possible.

The universe of possible metrics, based on existing policies and
procedures, will be quite large. Metrics must be prioritized to ensure
that the final set selected for initial implementation facilitates
improvement of high-priority security control implementation (as
defined by an audit or risk assessment). Based on current priorities,
use no more than 10-20 metrics at a time. This ensures that an IT
security metrics program will be manageable.

Selected metrics should be useful and relevant. Not all data are
useful, and collecting irrelevant data could cause stakeholders to
lose confidence in the IT security metrics approach. To ensure the
acceptable quality of data, standardize data collection methods and
data repositories.  Define standard data-reporting formats for events
throughout the organization, and store reports in a data repository.

Once metrics are selected, obtain organizational acceptance. Validate
metrics with the organization's stakeholders at headquarters and in
the field. Metrics should also be vetted through appropriate approval
channels. Lastly, phase out old metrics and phase in new metrics when
performance targets are reached or requirements change.

Metrics Program Implementation
The iterative process of implementation consists of six phases, which,
when fully executed, will ensure continuous use of IT security metrics
for security control performance monitoring and improvement.

Prepare for Data Collection. Key activities of this first phase
include identifying, defining, developing, and selecting the IT
security metrics. After the metrics have been identified, specific
implementation steps should be defined on how to collect, analyze, and
report the metrics.  These steps should be documented in the Metrics
Program Implementation Plan.

Collect Data and Analyze Results. Phase 2 of the process involves
collecting metrics data, consolidating collected data in the
prescribed format conducive to data analysis and reporting (e.g., a
database or spreadsheet), analyze data and identify gaps between
actual and desired performance, and discover areas needing
improvement.

Identify Corrective Actions. Phase 3 involves the development of a
plan to close the performance gaps identified in Phase 2.
Organizations must determine a range of corrective actions, select the
most appropriate corrective actions, and prioritize corrective actions
based on overall risk mitigation goals.

Develop Business Case. Phase 4 addresses the budgeting cycle required
for obtaining resources required for implementing remediation actions
identified in Phase 3.  The steps involved in developing a business
case are based on industry practices and mandated guidance. Steps
include developing a cost model, performing a sensitivity analysis,
developing the business case, and preparing a budget submission. The
results of the prior three phases will be included in the business
case as supporting evidence.

Obtain Resources. Phase 5 includes allocating the budget, prioritizing
available resources, and assigning resources.

Apply Corrective Actions. Phase 6 of the process involves implementing
corrective actions in technical, management, and operational areas of
security controls. After corrective actions are applied, the cycle
completes itself and restarts with a subsequent data collection and
analysis. Iterative data collection, analysis, and reporting will
track progress of corrective actions, measure improvement, and
identify areas for further improvement. The iterative nature of the
cycle ensures that the progress is monitored and the corrective
actions are affecting system security control implementation in an
intended way.

Conclusion
In summary, IT security metrics provide a practical approach to
measuring the effectiveness of an IT security program within
organizations, large and small. The results of a robust IT metrics
program provide useful data for organizations to allocate information
security resources and prepare performance-related reports. NIST SP
800-55 and related documents are available at our website
http://csrc.nist.gov/publications.


Disclaimer
Any mention of commercial products or reference to commercial
organizations is for information only; it does not imply
recommendation or endorsement by NIST nor does it imply that the
products mentioned are necessarily the best available for the purpose.



-
ISN is currently hosted by Attrition.org

To unsubscribe email majordomo () attrition org with 'unsubscribe isn'
in the BODY of the mail.