Re: Current state of Anomaly-based Intrusion Detection

[Adam Powers <apowers () lancope com>]

Well said. A few additions...

The "anomaly detection" technology that you find in successful products such
as those offered by Arbor and Lancope rely less on packet payload and more
on behavior-based "flow analysis". This denotes a heavy reliance on
statistics, learned traffic thresholds, and pattern recognition.

But why is flow analysis important to anomaly detection research and
development?

Well, NetFlow is a good source of flow data. NetFlow doesn't provide packet
payload, only metadata about the conversation (sorta like a phone bill). If
you're gonna derive security value from NetFlow data, you have to use
statistics, thresholds, and patterns; not payload.

So why is NetFlow important?

Simply put: flow-based anomaly detection is practical, affordable, and
"lightweight". Instead of deploying a physical piece of hardware to each and
every remote location, switch closet, and datacenter cabinet, turn on
NetFlow exports from your Cisco router/switch. Presto. Instant flow-based
anomaly detection technology anywhere you have NetFlow capable
infrastructure.




On 2/28/05 11:35 AM, "Jose Nazario" <jose () monkey org> wrote:

> there are several methods that can all be called anomaly detection
> techniques, you named only a statistical method.
>
> statistical based methods: you mentioned hardcoded threshold values (ie
> 200 MBps) and also learned average values for traffic. its a bit more
> complicated than that, and more fine grained with respect to services and
> endpoints, but you get the general idea. basically what you're doing is
> monitoring traffic rates, either in bulk or per service and/or endpoint,
> and alerting when some value is overshot, either once or for a sustained
> period of time. statistical methods rely on a strong baseline of traffic
> to accurately alert. characterization, such as "it's all TCP SYNs that are
> responsible for this upsurge in bandwidth usage", can also be performed.
>
> the second kind that i'll list here is specification based, and again you
> mentioned it briefly. this can include protocol specifications (ie "a
> valid SMTP greeting is no longer than NN bytes long"), such as what is
> done with some products. traffic is monitored, examined by the application
> or protocol that is in use, and the data is compared against a
> specification. if, in this example, an SMTP greeting longer than NN bytes
> is passed and alert is thrown. this requires detailed understanding of the
> network protocols and application protocols in use, their standards, and
> their implementations. not a trivial thing to do.
>
> the third kind would be relational, which is what a few companies are
> doing. in this scenario what you do is you examine inter-host
> relationships (ie "host A is an SMTP server to hosts B, C and D") and when
> that relationship is violated (ie "host A is suddenly a web server for
> host E") an alert is thrown.
>
> the fourth kind would be behavioral, where some metric of host or network
> behaviors is modeled and constantly examined. these behaviors can include
> an application's file usage, a hosts network usage, or the like.
>
> the examine i gave above for a specification based anomaly detection
> system can be hardcoded, as i discussed, or even learned, using a tool
> like PI to group the observations. this learning can be unsupervised (ie
> wholly trusted from the training period's observations) or supervised
> (where some editing of the observed data is done to ensure
> trustworthiness). and finally, this learning period can be a one time deal
> or continuous, allowing for dynamic network behaviors and the normal
> change over time. in this case, alerting can be done because the model was
> violated or some statistical confidence measure can come into play, as
> well (ie 3 observations, std deviation of 10%, but you overshot traffic
> rates by 12%, would you alert?).
>
> is this ready for prime time? sure, it's been in real-world use for years
> now. arbor networks' peakflow DoS and SP systems have been doing this for
> several years, using traffic rates over time to detect and characterize
> attacks. this system is a mix of learned or profiled traffic rates per
> service and network block endpoint as well as some informed decisions (ie
> SYN packet rates). and peakflow X is a relational based anomaly detection
> system that's been seeing real world deployments (see some recent news
> reports for example customers). i'd say it's been seeing real world
> deployments and success. arbor networks is one of several companies
> finding success in this field.
>
> AD systems are a significantly more complex and widely available system
> than you seem to have acknowledged. go digging around and you'll see there
> are some real systems out there seeing real use, protecting real networks.
>
> notes and links:
>
> PI: http://www.baselineresearch.net/PI/
>
> ________
> jose nazario, ph.d.   jose () monkey org
> http://monkey.org/~jose/   http://infosecdaily.net/
>
> --------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it with real-world attacks from
> CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
> to learn more.
> --------------------------------------------------------------------------


-- 

Adam  Powers
Director of Technology
Lancope, Inc.
c. 678.725.1028
f. 770.225.6501
e. apowers () lancope com

StealthWatch by Lancope - Security Through Network Intelligence



--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------