RE: Help in evaluating Inline IDS/IPS solution

["Golomb, Gary" <GGolomb () enterasys com>]

<Insert vendor warning here. Also, my views, not my employer's, yadda,
yadda...>

>       Do IDS vendors really test the signature against the vulnerable
> applications, hardware platform of the application and version of 
> application before
> releasing the signature? 

Absolutely!!! 

There's actually more testing involved than what you just described.
Examples include typical CERT advisories and vendor patch advisories -
especially for Microsoft products! They'll include some information on
what product is affected, what type of vulnerability it is (overflow,
input validation, etc.), and if we're lucky some hints on the vulnerable
sections of the application/code. Since signatures or other detection
algorithms cannot be developed based only on this information, we need
to determine exactly how to exploit the vulnerability and use our work
to base detection requirement on.  

Once you have a working signature/algorithm, you still need to do
further testing. When your signature will trigger every time you want it
to, you still have to test if it will trigger when you *don't* want it
to. There's been quite a bit of talk about the different theories for
reducing these types of alerts in a corollary thread (whether you call
them false positives, non-security events, or otherwise). [As a side
note, there's been lots of great talk about how these theories will
reduce false positives, but I haven't seen anyone discuss how they've
seen them impact false negatives in testing. A lot of those techniques
work great if your attacker is hacking based on what "Hacking Exposed"
tells them what to do, or running some VA scanner, but...] Anyways, this
kind of testing can include running the signature in a beta environment
for a short period of time before release, or additional research into
related protocols affected by the vulnerability in question.
Surprisingly, this is how new vulnerabilities can be found too!

It should also be noted that testing doesn't end there. Once a signature
is released, we need to keep an eye on all the exploits that
subsequently surface for the vulnerability. This is important for
several reasons. One is because someone may find a completely new and
different way to exploit a vulnerability that wasn't accounted for in
prior research. While this is not common at all, it does happen. Also,
we need to be conscious of how exploits incorporate "post-exploit"
activity into the code. Some of the most effective IDS techniques
available are to not look for the attack, but to look for the
compromise. 9 times out of 10, this can be done without relying on
statistical/strict anomaly detection routines, or related technologies.
(Don't take that as saying these technologies aren't useful - they are
VERY useful, especially when layered together with other types. The
point is, there is more than one way to skin the same cat.) There are
also other *very* interesting things that can be gleaned from closely
watching exploits. Read: NOT basing signatures for vulnerabilities
entirely off of specific exploits (yes, it happens too frequently), but
actually trending changes in exploit development. 


>       From sensor technology perspective, I find that all the vendors
> seems to be having similar capabilities. But, I am trying to see the 
> continued support on new attacks and vulnerabilities found. One vendor

> claims that they have 5 dedicated analysts looking at the
vulnerabilities 
> and updating signatures (if needed). Another vendors claims that they
have > more than 20 analysts doing this job. Can this be considered in
my eval? 
> Is it that other vendor exaggerating the number of resources 
> they have for this job.

Hahaha! I LOVE this one. ;) Funny how the vendor(s) that claim to have
umpteen signature developers/researchers will also bring them into
pre-sales meetings at the drop of a hat. If these guys have so much time
to spend on sales calls, then what are they really doing - or not doing
- for the organization? 

Don't get me wrong, we all savor every chance we get to go on sales
calls - really. It's a great opportunity to ensure that you are keeping
your head in the game, and really are in touch with customers. Sitting
in a lab, it's easy to understand the _threats_ networks are facing, but
you need to get out of the labs to understand the _problems_ networks
are dealing with.

I can guess who the vendor is that told you 5 people, and I would say
that is absolutely acceptable. Granted, LOTS of vulnerabilities are
announced every day, but how many of those do you think are really NEW
vulnerabilities? Not too many. It might be the first time a specific
software package is subject to that vulnerability, but chances are your
IDS is already looking for someone exploiting a related vuln. Therefore,
if you see a vendor releasing TONs of signatures every other week,
they're either fixing problems in their existing libraries, or trying to
get caught up to everyone else. It doesn't take THAT many people and
time to validate new vulnerabilities. I'd rather have 5 creative people
working on vuln announcement validation and researching new
threats/obfuscations, than 20 or more people who are scattered all over
the place doing other work. 


>       Performance:
>       What is the best metric to look for? I feel HTTP1.0/1.1, SMTP,
> IMAP, NNTP, TELNET, POP3 connection rate and UDP throughput for
different
> sizes is good metric. Is there anything should I look for?

I'd search the archives on this one. There has been MUCH heated debate
over the past several years on this. What it comes down to: like
everything else - it depends. Is the IDS going to sit on a web farm? If
so, then why bother with the other stuff? Etc... Anyways, there are
other people much more experienced than myself in the area of IDS
performance testing who can comment. 


>       Are there any labs, which provide testing facilities for testing
> IDS/IPS with latest vulnerabilities and with real vulnerable
applications? > I am really looking for lab which provides facilities
and allows us to
> test the IDS/IPS solution on regular basis.

http://www.neohapsis.com/ and OSEC is a good start. :)

Hope this helps!

-gary

-----
Gary Golomb
Senior Vulnerability Research Engineer
Dragon IDS Group
Enterasys Networks




-------------------------------------------------------------------------------
INTRUSION PREVENTION: READY FOR PRIME TIME?

IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities
- including intrusion identification, relevancy, direction, impact and analysis
- enabling a path to prevention.

Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at:
http://www.securityfocus.com/IntruVert-focus-ids2
-------------------------------------------------------------------------------