Re: Intrusion Detection Evaluation Datasets

[Stefano Zanero <s.zanero () securenetwork it>]

> http://www.icir.org/enterprise-tracing/devil-ccr-jan06.pdf

Still, we focus on header data, not on full traces.

> An overwhelming majority of network based IDSs use only spatial
> information present in packet headers.

"spatial" information ? if you mean "IP addresses", then
1) your statement is definitely not true and
2) such IDSs "work" only because of the artifacts in the evaluation datasets

> Moreover, you can find details of the endpoint worm propagation
> dataset in the following papers:

The dataset is similarly limited (only connection data) and moreover is
developed from a set of machines which are not established as
representative of real world traffic.

(I read only the peer reviewed paper)

> @Stefano: You have probably missed this point. Semi-automated
> procedures still require manual intervention, however, it will help
> to reduce its magnitude significantly. 

If you are reducing the magnitude, you are skipping attacks in the data
you are labelling, and therefore you are overestimating detection rates
(and potenzially false positive rate) in the systems you evaluate
afterwards.

The more you reduce the data, the less accurate your estimates.

Best,
SZ