Re: Announcement: Alert Verification for Snort

[Randy Taylor <gnu () charm net>]

Ok, I'll give this a go. Like Sam, I used to work for an
IDS vendor (*chuckle*), and did some work in this area during
my sentence there.
However, I've not been "in the industry" or even paying much
attention to it for awhile, so with that in mind, please observe
as I try to walk the wire without falling into the pit below...

From Marty's first post:

>1) Detect, Attack Present, Vulnerable:  True Positive
>2) Detect, Attack Present, Not Vulnerable: Nontextual (i.e. detect
>requiring contextual data to resolve)
>3) Detect, No Attack, [vuln|not vuln]: false positive
>4) No Detect, Attack Present, Vulnerable: False Negative
>5) No Detect, Attack Present, Not Vulnerable: ?
>6) No Detect, No Attack, [vuln|not vuln]: Don't care (true negative?)

I agree with Case 1.

I grok the terminology of Case 2 but in my days as an analyst (pre-IDS 
vendor),
I'd consider Case 2 something I'd very much want to know about. The presence
of an attack, regardless of its lack of  impact against the monitored 
network, is still
valid information and usable.

I agree with Cases 3, 4, and 6.

Using my logic (addled as it may be) from Case 2, Case 5 is also a False 
Negative.
It's information I wanted to know about, but wasn't told of.


At 09:57 PM 10/23/2003, Marty wrote:

>On Oct 23, 2003, at 6:53 AM, Sam f. Stover wrote:


>>On Wednesday, October 22, 2003, at 11:22  PM, Martin Roesch wrote:

>>In case 2 the "nontextual" isn't a false positive but I think that most 
people are calling it an FP these days.  I >>*personally* think that's a 
misconception.  What we have in that case is a *real attack* that your IDS 
is detecting >>exactly as it was asked to.  Just because it doesn't have 
the additional information about the context or relevance >>of the event 
isn't a problem with the IDS, it's a side effect of the way that NIDS have 
been built for the past 10 years.

>In the not too distant past I would have agreed with this - but I think 
as IDS implementations grew, the way people >describe FPs has changed.  I 
think today's IDS *needs* to know "the additional information about the 
context and >relevance" - because the event you are referring to is what 
I'll call an "effective FP".  Effective because any time I >spend trying to 
track down an IIS attack on an apache box is wasted effort.  I completely 
understand your point >Marty, because an attack did occur, and the IDS did 
log it.  However, if it is going to log it, then I want it to tell me >that 
the severity of the attack is lessened because it didn't succeed.  Even 
better, I want to see the 404 or 403 error, >so I can show my boss why I 
didn't even bother to look into it.

>>I agree and we're building new technology at Sourcefire to address it, 
and I know other companies are either >>building new stuff or adapting old 
stuff (e.g. vuln scanners) to get the same sort of data.  I was just 
pointing out that >>systems like Snort as it currently exists don't have 
this notion built into them so saying that alert verification >>reduces 
false positives when it's actually addressing nontextuals is something that 
I want to discuss, especially >>since it's my code that's producing the 
"false postives". :)

Oh ok, Sam already expressed my Case 2 - oops. *a growl emanates from the 
pit below...*


>I want my IDS to differentiate between an IIS attack on my apache box and 
an IIS attack on an IIS box.  I don't really >care how it does it.
>The two main methods, as I see it, are passive fingerprinting or 
integration with another tool like a vuln scanner.  >Both have their 
drawbacks w/ relation to different environments - which could probably fuel 
a complete thread.

>>Yup.

>The IDS landscape has changed.  Ten years ago, the type of event 
mentioned was probably not considered a FP.  >But at that time, IDS was an 
infant and people weren't dealing with events on the scale of millions per 
day like they >are today.  Current-day NIDS need to evolve to solve the 
problems that current-day users are facing.  IMHO 10 years >ago, NIDS 
administrators could afford to be a bit more interested in all kinds of 
attacks.  IDS was a new and exciting >technology.  I think it's lost some 
of it's glamour since then and people have to use it as just another 
tool.  And the >people I talk to don't have the time nor resources to run 
down half of the "real" attacks, much less look into attacks >that will 
never succeed.

>>Yes.  Separating the wheat from the chaff is becoming increasingly 
important in IDS as we all know, I'll be >>interested to see how the 
different techniques and approaches people are using to address this 
problem actually >>work in production.

I had the chance to see some tools being used to address this topic
a few months ago, working on a ton of live data.
Andrew Hall's post discusses it in adequate detail, and boiling it out,
it comes down to associating events from as many sources as may be
available into its correct aggregated identity. Current SIMs can do that,
but they are only as good as their rulebase and the validity of the data
being fed them. Still, they've come a long way from the last time
I had seen one.

Pretty much everything in the data feed chain can false, however, so
you need a SIM with enough smarts to be able to say something along
the lines of, "I didn't get everything I expected to see to satisfy this 
condition,
but I got enough good stuff to go ahead and throw a flag, just to be safe."
In other words, a variation on my Case 2 logic above. With the "Devil" in
that sentence being the word "good". Computers can be incredibly bad
at identifying "good".

But I did take a shot at the problem during my incarceration in IDS-vendordom,
and came up short. Collaborating with a fellow detainee, we found that "good"
could be defined relative to one source of data - an IDS in particular - 
but only
in the context of the validity of the IDS' own output. Where I failed was in
not provisioning for other relevant data sources - trending, VA, asset 
assessment,
etc. It was a good idea, just not enough of it, and not thought through 
well enough.
I can chalk some of that up to living inside the "glass bubble" effect Sam 
describes.
The rest is mostly likely one of the definitions of Wu Li from Gary Zukav's 
book.

I don't expect an IDS to be a one-stop shop for all relevant data and I doubt
any permutation of it ever will be. SIM seems to be the right idea, but it's
not mature yet. It wasn't that long ago we were saying the same thing about 
IDS,
so I'm not overly concerned about SIMs - they're very much worth using
right now, just as IDS' were then. One way or the other, I'm sure we'll
get it all nailed down eventually...which will happen just before the 
Universe goes
foom and forces us to start over again from scratch...



    -Marty


Randy
-----
"Nor does it do anything to make lemons bigger or encourage owls to explode."
  --- MartinG on /. ---



---------------------------------------------------------------------------
Network with over 10,000 of the brightest minds in information security
at the largest, most highly-anticipated industry event of the year.
Don't miss RSA Conference 2004! Choose from over 200 class sessions and
see demos from more than 250 industry vendors. If your job touches
security, you need to be here. Learn more or register at
http://www.securityfocus.com/sponsor/RSA_focus-ids_031023 
and use priority code SF4.
---------------------------------------------------------------------------