Re: IPS in the Enterprise UTM Firewall testing results

[Joel M Snyder <Joel.Snyder () Opus1 COM>]

Ravi Chunduru wrote:
> Am I right in saying that block rate is low, but detection rate could
> be lot higher. 

Yes.  If you were putting in a UTM firewall to be used as an IDS, then you 
certainly might say that the detection rate should be much higher.  However, I 
was testing ONLY with the devices set in "sane IPS mode."  In other words, I 
configured them as I believe a sane IPS manager would do.  I also have numbers 
for what are "insane" configurations (i.e., turn every signature on).  You'll 
see dramatic differences.  For example, SonicWALL and Juniper ISG-1000 more than 
doubled their catch rates.

Now, I am fairly opposed to putting an IDS inside your firewall---I think that 
this is asking for trouble performance-wise---but certainly there are very 
different catch rates when you configure the devices as an IDS.

BUT, there is a big issue here, which is that many of these devices were not 
engineered to be IDSes. In other words, they have a signature set configured 
specifically for IPS blocking capabilities, and the "IDS-ish" signatures and 
behaviors aren't even there.  My conclusion is that GENERALLY you will not want 
to use a UTM firewall as an IDS, because of performance and because of the 
specific design.  That being said, since Cisco & Juniper are sticking their 
off-the-shelf IDS/IPS in their boxes, of course either of those might work well 
as an IDS.  They also have coprocessing for the IDS stuff, which makes the 
performance less of a problem.

> Are there any results on vulnerability detection rate
> by these devices? I don't find this kind of result in the report. 

No, and I honestly don't think that this is the right place to put your IDS. 
Yes, you may want to tap off the lead from the firewall towards the internal 
and/or lobby/DMZ networks, but I am not seeing a major architectural or design 
benefit, at least in my mind, for most networks in sticking the IDS in the firewall.

I
> think another important factor in IPS is its false positive rate. Was
> this testing also done? I don't seem to find results on this item in
> the report.

I agree and I really wanted to put in false positive testing, but this is 
exceedingly difficult to do.  For one thing, there are no automated tools to do 
this, which means that we'd have to develop and deploy such a tool.  For 
another, it's difficult to say what is a false positive and what is not, 
especially as you get to some of the more esoteric attacks.  I am going to try 
and get a little more in-depth in IPS testing this year, so I'm going to think 
about this harder.  But if anyone out there on focus-IDS has ideas on how you 
would false positive test an IPS in a laboratory, that'd be a great thread to 
launch.

> Any comments on low performance of these devices.  It appears that
> some of these devices are suitable for Enterprise Edge and are not
> suitable for Enterprise-Core.

I think you're stating the obvious here (and I don't mean to be 
insulting---you're just restating my own conclusion), but I will point out one 
important issue: we specifically asked for 1Gbit boxes, and not faster than 
that.  In fact, we told the vendors NOT to overpower us because they would 
appear too expensive.  So it is possible that many of these boxes have a higher 
performance brother/sister (e.g., Juniper ISG-2000 to the ISG-1000 we tested; 
Fortinet 3800 to the 3600 we tested) that is more enterprise-core.

On the other hand, if you're saying "I need 1Gbps clear channel after IPS," then 
obviously there are some issues here with the boxes we tested.  I am not going 
to argue with you that any of these boxes that does 250 Mbps is good enough for 
a 1Gbps core.   Once you get to that speed, you need a faster box than we 
tested, and whether that means you need to have multiple CPUs (i.e., stacked 
boxes, one firewall and one IPS) or a big-ass box faster than any we tested is 
probably up to you as security architect.

jms



> Ravi
>
>
>
>
> On Dec 3, 2007 8:30 AM, Joel M Snyder <Joel.Snyder () opus1 com> wrote:
> > I wouldn't necessarily say that catch rates are disappointing.  With IPS, it is
> > very difficult to say what a good catch rate is.  Clearly, the ISS box "caught"
> > more things than all of the other guys, but remember that the purpose of an IPS
> > is to handle that narrow window between problem and patch--if you are relying on
> > your IPS to block SQL Slammer, you've got some major architectural conceptual
> > errors in your network that IPS won't help you with.
> >
> > I was pretty careful NOT to make any pejorative statement about the catch rate
> > (except to say that relative catch rates give you relatively 'better' IPS), and
> > I think that we ALL have to be careful in that area.
> >
> > I don't believe that anyone can credibly put a stake in the ground and say "an
> > IPS must block these specific attacks" and then defend that position.  This is
> > very different from, say, A/V or firewall, where there's a much clearer
> > black-and-white line about what you need to support.
> >
> > Clearly there are some pathological environments where an IPS somehow
> > substitutes for a firewall and where 6000 signatures is the "right number" to
> > have.  But in enterprise deployments, it's very unclear to me how to adequately
> > test an IPS for coverage.  I can do performance easily enough, but checking
> > coverage (which is what the Mu-4000 does) just seems quite dangerous.
> >
> > Anyway, I think that it is useful to see the comparative values on IPS catch
> > rate, but I would not go so far as to say that having an average catch rate in
> > the 30% to 40% range is "bad" or "good" for these products.
> >
> > I want to distance any testing we do from the bogus premise that you see in
> > tests like the ICSA certifications where they pick specific attacks and say that
> > you must block these.  To me, that's not supportable.  It may be in an IDS, but
> > IDS and IPS are entirely different beasts, and we were testing these products as
> > IPSes, not IDSes.
> >
> > jms
> >
> >
> >
> >
> > Ravi Chunduru wrote:
> > > this is really a great report and i am sure lot of effort has gone
> > > into this. catch rates and perforamance is really caught my eye.
> > >
> > > Catch rates are really disappointing across the board except for ISS.
> > > i do understand that client attack detection is new, but even the
> > > server side catch rates are awfully low. i understand that these are
> > > expensive boxes. i did not see any vendor responses  on low catch rate
> > > and performace.
> > >
> > > is this due to technology limitation or is it that devices tested are
> > > not up to mark?
> > >
> > > Ravi
> > >
> > > On 14 Nov 2007 15:28:18 -0000, jms () opus1 com <jms () opus1 com> wrote:
> > > > After months and months and months in the lab, a huge UTM test I did for Network World is now available (for free, 
> > > > folks, for free) on their web site.  I apologize in advance if you have to click 800 times to read the whole 19,000 
> > > > words, but here goes:
> > > >
> > > >
> > > > Main story starting point:
> > > >
> > > > http://www.networkworld.com/reviews/2007/111207-utm-firewall-test.html
> > > >
> > > >
> > > > Just the discussion of IPS in the UTM firewall/enterprise space:
> > > >
> > > > http://www.networkworld.com/reviews/2007/111207-utm-firewall-test-ips.html
> > > >
> > > >
> > > > Chart on catch rates based on Mu-4000 testing:
> > > >
> > > > http://www.networkworld.com/reviews/2007/111207ips.html
> > > >
> > > >
> > > > If you're not sure that enterprise should even be running IPS in their firewalls, you can click on the link below for a 
> > > > header page which has further links with some discussion on the pros and cons of that issue:
> > > >
> > > > http://www.networkworld.com/buyersguides/guide.php?cat=865480
> > > >
> > > >
> > > > Enjoy or not, as you see fit.
> > > >
> > > >
> > > > jms
> > > >
> > > >
> > > > --
> > > >
> > > > Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
> > > >
> > > > Senior Partner, Opus One       Phone: +1 520 324 0494
> > > >
> > > > jms () Opus1 COM                http://www.opus1.com/jms
> > > >
> > > >
> > > > ------------------------------------------------------------------------
> > > > Test Your IDS
> > > >
> > > > Is your IDS deployed correctly?
> > > > Find out quickly and easily by testing it
> > > > with real-world attacks from CORE IMPACT.
> > > > Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw
> > > > to learn more.
> > > > ------------------------------------------------------------------------
> > --
> >
> > Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
> > Senior Partner, Opus One       Phone: +1 520 324 0494
> > jms () Opus1 COM                http://www.opus1.com/jms

--
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One       Phone: +1 520 324 0494
jms () Opus1 COM                http://www.opus1.com/jms

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw 
to learn more.
------------------------------------------------------------------------