Industry Definitions... possible? was Re: Security Audit

[Don Bailey <baileydl () mitre org>]

All,

I've just caught up on this thread and its been very interesting to
read, especially since we just recently started a discussion on the
osstmm discussion list with regard to defining the following terms:
security test, security assessment, security audit, penetration test,
vulnerability assessment.  Personally, I feel that these terms are used
incorrectly most of the time, and in attempt to capture their nuances I
made a crack shot at definitions.  Note that my attempt at definitions
were without the benefit of several weeks of pen-test list
discussion--which I sorely regret not previously absorbing.

Granted, there's plenty of room to simply argue that this is all just an
exercise in semantics and my definitions posted to the osstmm discussion
list may just cloud the issue (that's not my intention I assure
you)--they may be considered way off in fact or flat out wrong.  I'm
willing to hear those opinions (even flames), in the hopes it will
better shape the osstmm.  Following is the relevant post which I feel
speaks to a search for the correct type of testing at the correct time.

Sincerely,

Don

-------- Original Message --------
Subject: Definitions Re: [OSSTMM-Discussion] new tests and stuff
Date: Thu, 13 Sep 2001 14:03:53 -0400
From: Don Bailey <baileydl () mitre org>
Organization: The MITRE Corporation
To: pete () ideahamster org
CC: OSSTMM Discussion
<ospentest-discussion () lists sourceforge net>,OSSTMM News
<ospentest-news () lists sourceforge net>
References: <FEENJDOHIDLOEIFPBFCMMEAICEAA.pete () ideahamster org>

pete wrote:
>
<snip side thread on steganography>
> I would also like to clarify some definitions for the industry.  Perhaps

I've lost some sleep over this already, so I'll start taking a crack at
it... hang on!

> security test
> security assessment
> security audit

First of all, I see these three as supersets of the other two.  Also, as
these three are listed, I see them as nearly the same but progressively
more intense with varying degrees of relevance with regard to policy,
officiality, etc.  Therefore, it IS important to define each seperately
and not just clump them together with some phrase such as "Oh...it's all
the same thing.", which is what I hear too often.  

Security Test:
A security test would be a routine and general test of an organization's
network security mechanisms, from outside in, to obtain a basic and
generally accurate idea of how well the organization has implemented
said security mechanisms.  No prior warning is given to any employees. 
The test may be performed "in house" with existing networking
personnel.  Results from a security test generally would be used to make
functional network "tweaks" to remedy any unexpected problems discovered
and bring the network back to spec.

Security Assessment:
A security assessment is an intensified security test in scope and
effort, the purpose of which is to obtain an advanced and very accurate
idea of how well the organization has implemented network security
mechanisms and to some degree policy (such as spot tests of password
strengths or acceptable allowed services).  No prior warning is given to
non-critical employees.  Outside technical assistance may be necessary
to handle the workload and should be seriously considered.  The results
of a security assessment may be surprising or unexpected and would be
used to make significant changes to both network implementations and
policy.

Security Audit:
A security audit would be an extreme security test, definitely handled
by an outside and impartial source, that performs a ground up, and
outside in, audit of the organization's network security mechanisms and
all pages of security policy.  Audit implies finding non-compliance with
policy.  All employees are informed well ahead of time in order to meet
compliance with policy.  They are interviewed by auditers with regard to
their knowledge of policy and their personal level of compliance.  The
results of a security audit should NOT be surprising, SHOULD only
validate existing implementations with regard to known policy, and
violations are to be taken very seriously, to include possible
termination of employment of individuals in direct violation of policy
or responsible for sections found in direct violation of policy. 
Extraneous results that are not covered by existing policy should be
addressed individually and considered for future policy changes and
security assessments.   

Note that these definitions may incorporate the use of the word
"network" and I apologise if I focused on that a bit more than I should
have, but that's my realm, so I'm biased.  Perhaps these definitions can
be sanitized by the removal of the word "network" to be made more
general yet accurate and for specific "network" related actions we can
say "network security test", "network security assessment", and "network
security audit".  These latter definitions extremely limit scope...
which may actually be preferred in some instances.

> penetration test

Penetration Test:
A penetration test is a no-holds barred, outside to inside, get in any
way and as many ways as you can, test of an organizations physical,
network, and human facets of security.  This is Red Teaming. 
Non-critical employees may or may not be notified in advance of a
pen-test.  A Blue Team may or may not be involved for active defense.  A
White Team may or may not be involved to referee the event.  Rules of
engagement are defined ahead of time for at least the Red Team and by
individuals authorizing the activity.  Specific exclusions may be
introduced to the pen-test, such as no testing of physical or human
facets of the organization's security (i.e. no lock-picking and no
social engineering).  Variations on this model of testing may be
"capture the flag" in which the Red Team has but one type of document to
find or alter--a proof of concept crack attempt, if you will--in order
to be successful or call and end to the event.  Also "cry uncle" is
common, in which an onslaught of near-destructive activity to an
organisation is so great as to test the limits of the Blue Team's
response capabilities, demonstrate for CEOs or corporate-types how bad
it could get, until someone in authority simply says, "that's enough...
we get the point."  Some or all of these variations may occur during a
pen-test.  Results of a pen-test are almost always surprising and
unfair, but are significant in helping to reshape policy, highlight the
significance of consistently present security flaws, discover previously
unknown weaknesses, as well as testing the resolve and ingenuity of Blue
Team members in the organization.  Pen-test is commonly but incorrectly
used synonymously with "security test", "security audit", "vulnerability
assessment", et al.  It is closely related to a "security assessment"
but a penetration test is a very different and distinct variation of the
security assessment and should be recognized as such.  

> vulnerability assessment

Vulnerabiltiy Assessment:
A vulnerability assessment is very regulated, controlled, cooperative,
and documented evaluation of an organization's network security posture
from both outside-in and inside-out, for the purpose of defining or
greatly enhancing security policy, and determining the need or removal
of security products / implementations.  Non-critical employees are not
included in a vulnerability assessment.  "Defense-in-Depth" will be the
phrase du jour during this event, and all types of corporate individuals
will be included to participate in discussions with regard to the
neccessity of security versus the need for functionality and
productivity.  The techies are often involved with or responsible for
product evaluation, providing their results to analysts that write
recommendation reports.  Corporate politics are involved and many
sub-organizations may prove defensive or outraged with regard to noted
deficiencies or recommended security implementations.  Outside technical
assistance with evaluating the organization's security posture is
recommended, but be wary of consultants that bring their product to the
table as THE solution to the results of a vulnerability assessment. 
Tasks may include offline testing or evaluation of existing or
anticipated security products, network profiling to determine critical
assets or subnets, independent code review of implemented scripts and
software, and security budget analysis.  A vulnerability assessment may
take months to nearly a year to complete, and the focus should be on
completing an unbiased and scientific evaluation of the organization's
security posture with regard to its current and near-future models of
real-world operation.  Results of a vulnerability assessment should be
the definition or ehancement of an organization's security policy, and
implementation plan for new security products & measures to mitigate
defined vulnerabilities and/or the removal of ineffective products &
measures.  A vulnerability assessment is commonly combined, brought on
by, or followed by a security assessment but does not require such an
activity in order to be conducted, complete, or successful. 

Crap.  That's my first shot at defining these.  There's a good bit of
the wording I'm still not comfortable with, but I think I got my general
ideas on the table.  Have fun hacking / editing this.  I hope this
begins some fruitful dialogue with regard to locking down these
definitions.
--end-- 

H C wrote:
>
<snip>
> fruition without being detected.  Security consulting
> firms should have this as their goal, as well, with
> respect to their clients.  This being said, what has
> been referred to as a "blind pen test" quickly drops
> out of the picture all together as a method of
> reaching this goal.  A vulnerability assessment of the
> overall infrastructure examines the configurations of
> hosts within that infrastructure, the relationship
> between the hosts, and the processes and procedures
> used by the admins.  The assessment gets into every
> nook and cranny and peeks into the deep, dark corners.
>  Verification testing (ie, "full disclosure pen test")
> can be done once recommended changes have been put in
> place.
>
> Attempting to break in blindly using no more
> information than a domain name is not something that
> can be completed in a week or two for larger
> infrastructures, and leaves many items unchecked.
> However, a "blind pen test" can be used at a later
> date to test the effectiveness of detection, as well
> as incident response procedures.  At that point,
> conducting such a test with full knowledge of the
> infrastructure would definitely be very beneficial.
>
> Thanks for your time.  Thoughts/comments appreciated.

--
Don Bailey
Senior INFOSEC Engineer/Scientist
Secure Information Technology
The MITRE Corporation


----------------------------------------------------------------------------
This list is provided by the SecurityFocus Security Intelligence Alert (SIA)
Service. For more information on SecurityFocus' SIA service which
automatically alerts you to the latest security vulnerabilities please see:
https://alerts.securityfocus.com/