RE: Tuning false positives

[Omar Herrera <oherrera () prodigy net mx>]

Hi Sam,

It might sound very obvious but only a system flagging every activity as
potentially malicious will catch every attack, along with an infinite number
of false positives. Also, a system without rules won't report any false
positives and it won't detect any attack as well. These are the 2 extremes,
and somewhere between these 2 limits is your IDS/IPS configuration.

Now think about this simple: an http connection from your internal network
to the Internet. Is it malicious? From the IDS/IPS point of view, as long at
this does not match any predefined malicious pattern, the traffic is OK. But
maybe you realize that this is your mail server, and in your corporation,
this mail server isn't supposed to establish http connections.

Visibility problems prevent blacklist based IDS/IPS from catching many
attacks, and they come (at least) in two flavors: organization and location
visibility problems. Use of network services at unauthorized times
(according to organization policy) is an example of a problem of
organizational visibility. A network based IDS/IPS being unable to tell if a
certain web connection was established by an authorized application is a
visibility problem related to its location (an hIDS/hIPS might be able to
solve this particular problem). 

We can solve these problems by telling the IDS/IPS what is acceptable (i.e.
creating customized white lists) and what is not. This is the reason why
tuning helps and why it is necessary. Of course, you might have a particular
configuration causing a predefined rule reporting a false positive, or a
rule alerting you of real attacks, which however you know are not a real
threat because you don't have any of the O.S. or applications the affect. 

The reason why IDS/IPS don't give useful information under these
circumstances is always the same: they lack the capability to acquire and
process all information that is specific to our business environment, which
is also necessary to perform reasonably well.

But vendors can't possibly anticipate all possible environments and
organizational policies, therefore many of them rely solely on black lists
(known bad behavior). Some have tried to solve this problem with behavior
analysis/adaptation, but as you might have already experienced, in some
environments the level of false positives might still be very high because
of frequent changes and complexity of the environment. For instance, these
systems all have to estimate an acceptable behavior by a certain time, and
updates are always recalculated between time frames. There is always the
risk that legitimate activity is ignored because it was not identified
inside these time frames or that malicious activity identified within the
time frame is regarded as valid.

Therefore, no matter what many vendors say, you shouldn't expect any IDS/IPS
to perform reasonably well without tuning. This of course is also the reason
why many organizations don't like IDS/IPS, it is definitely time consuming.
Also, you might want to create some white lists (many IDS/IPS allow them) to
complement your black list tuning. This will help you enforce your security
policies more clearly and probably yield more useful information than many
of the predefined rules. Be aware however, that even if the number of
authorized activities within your organizations is smaller that the number
of potential malicious activity that black list try to identify, this can
still be a huge number.

While tuning I would recommend to disable all rules for which you have
identified an unacceptably high number of false positives. If you know what
the problem is and you really thing the rule reports and important issue,
either fix the rule to avoid problems in your environment or simply create a
replacement rule. If you can't tell from all the reports which ones are real
and which ones are not the rule is not useful (remember the extremes of
malicious activity detection).  Then, with your security policies in hand
(or some clear idea of what might be serious malicious activity in the
context of your organization) try to create some white/black list based
rules. 

Don't think your vendor is rude by telling it is your problem... in the end
it is, some information for tuning can only be provided by you :-). Sure,
many vendors fail to clearly inform people about these requirements (time,
training and resources) because, obviously, it makes the IDS/IPS less
attractive.

In the end, yes, tuning is always required for an IDS/IPS to be useful.
Perfection cannot be achieved but there is always room for improvement. 

That's my opinion.

Regards,

Omar Herrera

> -----Original Message-----
> From: Sam Heshbon 
>
> My company is testing a few intrusion detection & prevention products. On
> the first few hours/days
> after deployment the machines alert on ten of thousands of events, which
> is way too much for us to
> ever go through, most of which are false alarms.
>
> The vendor's solution is tuning the systems, which means shutting down
> signatures, detection
> mechanisms, omitting defragmentation tests and so on. These tunings do
> reduce dramatically the
> number of alerts, but it seems most of the detection capabilities have
> been shut off too, so
> things are
> nice and quite but we've no idea what's really going on in our network
> apart from catching the
> trivial threats such as old worms, which don't get false alarms.
> Has anyone encountered this situation? Anyone got a solution?



------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------