Re: Intrusion Prevention

["Raistlin" <raistlin () gioco net>]

> It claims to have a 100% accuracy , no false positives.

It's really simple to build a system with no false positives. Just leave it
unplugged. It generates no false positives, since all the positives (none)
are true positives.

Unluckily, this doesn't say a word about the performance of the system, does
it ? :-)

If correct positive assignments are A, false positives are B and false
negatives are C, accuracy is A+D/A+B+C+D, precision is A/A+B and recall is
A/A+C (in document retrieval terms; i'm not aware of an established IDS
terminology, but the concepts are similar on the whole). A 100% accuracy has
no meaning whatsoever. The absence of false positives means a 100%
precision, but we cannot pretend marketing people to read the Communications
of the ACM, can we ? :-)

What you really want is a high signal-to-noise ratio (many true positives
among the positives), so a high precision, that's right, but also a high
recall (many of the  attacks must be detected). You can plot precision vs.
recall in a ROC curve. They have done it that way in biology and medicine
for years, and the graph usually shows an inverse proportionality. 100%
precision means a very, very low recall, if any (unless you have designed
the perfect intrusion detection system, and I'd challenge that even on
theoretical grounds ;).

A high precision, per se, means absolutely nothing. A nonexistent IDS is
totally precise: it never generates a false alert. It never generates an
alert, also :)

Stefano "Raistlin" Zanero
System Administrator Gioco.Net
public PGP key block at http://gioco.net/pgpkeys