Re: Intrusion Detection Evaluation Datasets

["Stuart Staniford" <sstaniford () FireEye com>]

On Mar 12, 2009, at 8:40 AM, Zow Terry Brugger wrote:

> I see a lot of people saying (correctly) that advanced (non-signature
> based) NIDS can't be researched until we have good evaluation
> datasets, and I see a lot of people ignoring them and doing it anyway.
> Is anyone (else) actually working on fixing the data problem?

There's a number of things about the framing of this discussion that  
are bugging me (I come at this from the perspective of having spent  
quite a bit of time on both the research and the commercial sides of  
the field).

For one, the nature of the intrusion detection problem is very  
dynamic.  Ten+ years ago, the biggest problem was interactive  
attacks.  Five years ago, the biggest headache for organizations was  
automated random scanning worms.  Today, RS worms have become much  
less of a big deal, and most of the action is attacks on clients  
primarily via the web, and the resulting remote control of systems via  
bots.  These are very different problems requiring pretty different  
approaches.  And in another five years, I'm sure the main problem will  
be something else again.  So the main nuisances on the wire keep  
changing, and any dataset is necessarily going to get stale very  
quickly.  In particular, quite a lot of staleness will happen between  
the start of a hypothetical graduate student starting and finishing a  
thesis.

Secondly, I think there's an assumption lurking implicitly in the  
search for datasets that the appropriate focus for research is the  
inference algorithm.  Much like the machine learning community does -  
get a fixed data set, and then try all kinds of inference algorithms  
to see what works best.  For our problem set, I don't think that's a  
great way of doing things.  For us, the main focus is "What are the  
bad guys doing now?" and "What features do we need to detect what they  
are now doing".  Usually, if you have good features with high  
discrimination, most algorithms can be tweaked to do ok.  If you don't  
have good features, no inference algorithm will save you.  And if you  
have good features today, they'll be a lot less useful in a couple of  
years and new ones will have to be invented.

I think there's a lot of contribution that researchers can continue to  
make in this field.  But you can't think of it that you are  
discovering timeless principles or something - this is much too  
applied a field.  It's about figuring out what's happening on the wire  
*now*, and what can be done about it.

So forget looking for a dataset.  Look for a wire.  Do whatever it  
takes to get your institution let you sniff the egress link - it's  
just about guaranteed to have plenty of attacks on it. Build, or  
adapt, some software to look at the packets with respect to some  
problem that interests you and that seems like a currently rising  
challenge.  Spend a lot of time manually poring over the packets to  
figure out what is going on, and label your own data.  You need to get  
your hands dirty.  If you look at the most influential highly cited  
researchers (Todd Heberlein, Vern Paxson, etc, etc) their influential  
contributions were always driven off actually trying to detect attacks  
on real networks.  In the end, intrusion detection is about detecting  
intrusions, just like the name says.  Any amount of theoretical or  
algorithmic sophistication is a waste of time unless it directly  
contributes to that goal, and no amount of sophistication will be very  
exciting if it only improves the detection of five-year-old attacks  
(this is not to say that technical sophistication is not required for  
current problems - I believe it is).

I think the problem of producing regular timely datasets that can be  
safely published is probably just about intractable, even if one of  
the funding agencies were to step up to try and fill the shoes DARPA  
long ago left behind.  Synthetic datasets would not be that  
interesting, and since most attacks are now inside packet content, the  
challenge of reliably anonymizing the data while not affecting the  
traffic materially would be just about impossible (what algorithm is  
going to sanitize every single web developer's cookie format, for  
example?  How could one be sure that obfuscated javascript didn't  
contain any personal information?).

Stuart Staniford.