IDS Assessment (was: Intrusion Prevention... probably something else at one point)

["Golomb, Gary" <GGolomb () enterasys com>]

Hey Graham!

-----Original Message-----

> >  About someone saying they wouldn't look at Dragon, or ANY IDS for
that
> >  matter, because they do not participate in a specific test,
> >  is a little un-nerving.

>  but why mot?

The important aspect here is to not put ALL your eggs in one basket.
Making a decision based solely on one test (or article, marketing
glossy, or whatever) and no others could leave you with some serious
bind-spots in your perceived understanding of specific product(s) or
technologies. More below...

> > Sure, there are industry testing standards which
> > are undoubtedly the most inclusive and open to peer analysis, such
as
> > OSEC, which I would also be suspicious of anyone who refuses to
> > participate in. (We can revisit this point later.) But to take a
product
> > that continuously participates successfully in other magazine and
> > third-party tests, but does not participate in one (or another)
specific
> > test... Well, for me that would raise more red-flags about the test
than
> > the product. If Dragon (or **any** IDS) refused all the tests
publicly
> > available, then of course we could draw some conclusions like you
have
> > alluded to. Please realize that I am not the official spokesperson
for
> > these types of subjects. Just someone who's close enough to have
some
> > thoughts on it...

> so some tests are ok, but others are not ..., but which are ok and
which 
> are not..,

Good point. Same applies here too. Keep in mind, the statement I
originally replied to said something to the effect that Mr. Willams
would not consider IDS Brand X because they didn't appear in one
specific test. The point was not to say, "You picked the wrong test."
The purpose of the email was to state that there are MANY tests
performed every year. There is NO way possible for any vendor on this
planet to partake in ALL of them for reasons such as time and resources,
just to name a few. On top of that fact, no test is 100% complete in its
scrutiny of all IDS features/technologies. By stating that you would not
consider IDS X because it's not in one specific test seems almost
irresponsible.

Here's a different question, but related... IDS detection engines that
are based primarily on Pattern Matching can see things that Statistical
Anomaly based IDSes just do not have the capability or resources to
detect. The same is true in reverse. Anomaly-type detection engines can
see things that pattern matching IDS can't spare the overhead to look
for effectively. Have you ever seen a single test that treats the
testing of AND the results of two technologies differently? (A footnote
stating this doesn't count. I'm talking about underpinning of the
testing methodology here.)

(There is a potential flaw to this position, but you address it below so
I'll wait until we're there to highlight it.)

Most tests seem to compare the two different methodologies (or all the
different intrusion detection methodologies) against the same static
test data, without accounting for the individual strengths and
weaknesses of each detection methodology (which is described in much
more detail below). Since this is the very foundation of the test -
which in-turn can lead to considerable flaws in accounting for the real
strengths or weaknesses in a product; that leaves you with three
choices. 1) Examine as many test results as possible and find common
threads among them. 2) If you're only going to look at one test, try to
find the one that has the largest number of designers/implementers
behind it. Thirty-something heads from different interest groups and
backgrounds are better than five from the same. 3) Test the IDSes
yourself against the factors that are most important to you.

> > There are several issues at work when we discuss testing. First of
all -
> > and most generically - is that getting involved in a test is very
time
> > consuming. You really don't think the people who are doing the
testing
> > are actually taking a few months to learn the ins-and-outs of each
> > product before testing it, do you? Of course not!

> yes that would make sense, but the same is true for all commecrcial 
> vendors and open source products.

Absolutely! You know who I feel most sorry for when it comes to
testing??? Our competitors. When they buy Dragon (or steal, or whatever)
to do competitive testing with, they don't have the luxury of getting a
field engineer to come on-site and configure it for them! Talk about a
bum-deal, huh?! ;)

> > I've only seen it done
> > once, and it was by the same guys who wrote much of the OSEC testing
> > standard. (Incidentally, that test took almost a year to complete!)
> > Anyways, for every test we participate in, we need to send people
from
> > our Dev, QA, or R&D teams to assist on-site with the testing cycle.
> > Those are resources that are being taken away form their normal
schedule
> > to assist the testing team in question. Since there are MANY tests
> > completed every year, we have to carefully choose which to
participate
> > in, and which not to.
> >
> > Time and resources are not the only factor involved in selecting
which
> > tests to (or not to) participate in. Generally, we are given a
> > description of the testing methodology upfront. Believe it or not,
> > sometimes we're told that we cannot see the testing methodology
upfront.

> Is this not true for all vendors and open source products?, ater all
if 
> someone is going to hack you will they tell you about it in advance of
how > it will be done?

Ahhh... This is the point I referenced above as being a possible hole in
the "not all IDSes are looking for the same behavior (on purpose) so how
can you test each of them against the same data" argument. 

First, your question... You are *absolutely* correct. However, most
tests serve a slightly (and in some cases, only damn slightly) different
purpose than hacking a network. Most of the time, they are to document
someone's findings so the community may be enlightened by the results of
their tests, and so vendors can use the results to sell against each
other. Going on the understanding that most vendors are presented with
more than a few severely fubar'ed test plans every year for these
purposes (which are promptly rejected), there probably aren't too many
vendors out there who would participate in a blind test like this. Tests
like this pose the threat of potently losing a significant amount of
sales - based on a terribly flawed test in the first place. The best way
to sum it up is like this: You name an IDS product, and I will design a
test that will make it beat every other IDS on the market. (Actually,
Miercom beat me to this.) The only way to know that you (as a vendor)
have not been suckered into a test designed to make you fail is to view
the test criteria. There's also a flip-side to this. For consumer
protection. The only way for you (as a decision-making consumer) to know
if a test is worth the paper it's written on is to have the ability to
see full documentation on the testing methodology and accounting for all
equipment/products used in a test.

Now, for the point made earlier but not being addressed till now... Why
should the test accommodate the IDS methodology behind the detection
engine? Hackers don't. 

And that is exactly where most every test fails. Look at most tests, and
you'll see something like a check-list that looks something along these
lines:

- Statd overflow
- Back Orifice 2K connection
- Unicode attack
- attack d'jour with a cool sounding name and/or press
- etc...

That's all well and good, but compromises don't look like that. There
may (or may not) be a probe of some kind. The attack might be a modified
version of an existing exploit. It might be a new exploit. The attack
might require a brute-force attempt to work. Once it does work, there
will probably be a back channel of some kind which has been established.
It might be encrypted. It might not. Once the attacker has local access
they may (or may not) need to elevate their privileges. Once they have
done that, they will probably transfer over a package of some kind,
depending on their goals. Once they fully own the system, they may
repeat the cycle, or start up again with a different goal in mind.

Every IDS on the market is going to see that entire chain of events
differently. There are parts of that chain where the pattern matchers
are going to excel beyond all the rest. There are parts where the
protocol decoders are going to have deeper visibility. And there are
parts where traffic anomaly detection would trigger a completely
different type of alert. (And before we start a new thread on this
subject - YES! *ALL* IDSes implement some form or combination of all
three of those methodologies. I'm only addressing the core engine for
simplicities sake!) Every test that I have seen will pick out one part
of that chain of events and test it. It might be the scan, it might be
the attack, and it might be the backdoor. When you have umpteen
different IDSes next to each other, wouldn't you be more interested in
how IDS X responds in the event of a system on your network being
compromised as a whole? Testing for the number of individual attacks
properly detected DEFINITELY has merit and should not be discounted, but
why do you think we haven't seen tests like "generic worm propagation"
or "system compromise scenario 1," "system compromise scenario 2,"
etc... A test like that might be a little more applicable to you and
your network than the results of some other tests which are touted so
heavily. 

The problem here gets worse! Even in the tests we've talked about which
do only pick out one specific attack or probe (or many) and leave out
all the supporting actions of that specific event, they might not even
be using a real attack in the first place. They might be using a
recorded version of a benign version of a simulated attack! What does
that test? When IDS vendors develop detection algorithms, be it
signatures or modules of code, they are doing it based on real attacks
and exploits in the wild and possible variants of those. Not benign
simulations or recorded pretend versions of a hypothetical attack. Oy!
We haven't even touched the subject of background traffic for load
testing!!!


> > This dumbfounds me for all the reasons that MJR (and others) already
> > brought up. IDS testing is too easy to inadvertently (and sometimes
> > intentionally - read: Miercom's test of Intrusion.com) skew the
results
> > of any test. When I think of testing, I think of scientific process.
> > Unfortunately, many of the IDS tests you read each year do not
adhere to
> > any sort of process, much less an actual scientific methodology.

> agreed, but this is the nature of the environment.., but the key
question > is do you have a better process or testing in mind?

That exists today, in a word, OSEC. 

See the site for a full description. (http://osec.neohapsis.com/) And
what makes OSEC even better is although it's been accepted by most as an
industry standard evaluation, the OSEC folks are already working on
version 2.0. It shows the level of detail and commitment that has been
dedicated to the project.

> > methodologies. Rather than have my slanted (and VERY strong) opinion
on
> > the subject, look at the tools they use to implement their tests,
then
> > do a search on lists (like this one) to see some of the pros/cons of
> > using those tools. Put those individual discussions together, and
you'll
> > get a more clear view of the bigger picture here.

> I would like to see a more in depth discussion on this if anyone has
any 
> opions on these tests and all others? (ducking ;-) )

Also, a search of the archives on
http://marc.theaimsgroup.com/?l=focus-ids&r=1&w=2 will show years of
discussion on these topics. *Please* check there first!

Hope this rambling makes my previous rambling a little clearer, if
you're even still awake by the time you reached this point....

-gary

ps- Standard disclaimer applies. I am NOT the official spokesperson for
my employer on Dragon, testing, or anything relating (and not relating)
to those topics. With a mouth like this, I'm sure you can see why. These
are MY ***opinions,*** and not that of my employer, and in many cases
NOT of my colleagues!

----
Gary Golomb
Research Engineer
IDS Group
Enterasys Networks
410-312-3194