RE: Random IDS Thoughts [WAS: Re: IDS thoughts]

["Mike Lyman" <mlyman () west-point org>]

> Hint: data mining techniques, anyone ? There's a great book 
> by J. Mena on
> the topic, which I warmly recommend.

I don't think I've posted here before so to set this up, I've been
running and building the IDS systems on a global network for about
three to four years. 60,000+ employees and contingent staff, 300,000+
systems on the network and Internet egress and ingress in over two
dozen locations around the world. Data overload is an understatement
for what we face.

The value of data mining on IDS data was first demonstrated to us by
folks in our research group who had wanted to do a project on our IDS
pilot data. They showed us stuff we'd have never seen even with
today's consoles on the commercial IDS systems we use. Since that time
we have more and more mining the data and twisting it this way and
that. The single most common skill we put on job requirements is the
ability to run SQL queries and that is a high priority on our training
schedules.

Through developing differenent views of all the data available to us
and constant analysis, we've been able to create reliable alerts with
few false positives from our commercial systems. With home grown log
collection, we've been able to craft low noise, high signal alerting
IDS systems from normal high noise event logging. All of it is finely
tuned for our environment instead of generic enviroments that the IDS
venders have to try to shoot for. It is no where near pefected yet but
it is far more managable that what we used to have even though we now
have considerably more data sources.

If you are not looking into data mining techniques, you are missing a
great way to use your data and reducing the data overload.

Mike Lyman
CISSP
mlyman () west-point org
pgp keyid 0xD7BBADAD 


-------------------------------------------------------------------------------
INTRUSION PREVENTION: READY FOR PRIME TIME?

IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities
- including intrusion identification, relevancy, direction, impact and analysis
- enabling a path to prevention.

Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at:
http://www.securityfocus.com/IntruVert-focus-ids2
-------------------------------------------------------------------------------