IPS Change management process

[mick at pauldotcom.com (Michael Douglas)]

Comments inline tagged with my initials MBD, so just grep for MBD to
see where I'm tossing my $0.02 in.  ;-)

On Fri, May 22, 2009 at 12:21 AM, Michael Dickey <lonervamp at gmail.com> wrote:
> 1. As Joel said, try to always do rules/signature and device/software
> updates as soon as possible. For appliances and the like, you should
> download daily and use whatever automated process is built in, especially
> for rules updates. For my vendor, I've only had one known issue with a new
> rule being somewhat poor and not doing what it should, but thankfully my
> device defaults to not block with new stuff.

MBD - IMO this depends on the nature of the traffic you're watching.
I don't put any rules in before running them in a non-production
environment -- ideally a lab that's air gapped so I can replay some of
the "greatest attack hits".  Chat with the management team.  We're
hear just to point out risk.  If they're OK with something that is
risky -- but not against the law -- you're probably best doing what
they ask.  Again we just educate -- at least that's my take.

> 2. Also, do what you do now and stick to IDS mode to see what traffic you
> see. You'll likely want to keep this up for quite some time. In fact, you
> can probably keep this up forever and just block what you want to block,
> especially if you're going to be good about attending to high priority
> alerts that fire.

MBD - excellent suggestion!  "Learn mode" is something every decent
solution has.  It's really good stuff.  Heck, in one instance, having
it in learn pointed out some interesting architecture issues we hadn't
noticed before.  The fix resulted in significant savings.  We
explained this to management and they were geeked out like us.  :-)
We had paid for the project and hadn't yet gone full power!

> 3. As your device learns, start to identify alerts you know are either a)
> benign, b) false positives, c) things you know are fine and don't want to
> block, or d) things that give you no value (arp and scan alerts most often).
>
> 4. For things you don't care about and can't/won't block, set up filters to
> ignore them. If your device allows, be sure to provide detaile comments on
> why you're making the change. If it doesn't, either utilize your ticket
> system to track changes or start a spreadsheet. Every change you make, mark
> it down in a new line and be as detailed as possible about where, what, and
> why on those changes.

MBD - I couldn't agree more!  The rules & filters you make will get
convoluted fast.  If you don't comment them well they turn into some
dark magic that *has* to be in the setup from now until the end of
times.  Also your auditors will want to buy you beers when you show
them a change log with this stuff.

> 5. Eventually once you get a feel for things, you may start diving into what
> alerts are possible. I'd encourage you to start blocking categories that
> likely will not have false positives very often like P2P, IM, Bot, Worm,
> etc. As before, always document your changes. And if you have a team to
> notify on your changes, feel free to do so. It sucks to block something only
> to find out a week later that it sent a team into 3 days of futile
> troubleshooting, but it didn't come up until 2 days after your change.



> You don't have to, but I think it would be useful to treat your IPS rules
> and blocks like firewall rules: document changes and definitely the why
> about them.
>
> 6. Get used to your device and make sure you can pull reports and
> information out like what alerts you all have blocked or customized. If it
> has a built-in history and audit trail and comments, that's excellent!
>
> 7. What you'll find is not only are you learning about the traffic in your
> network, but as you read up on each alert, you're going to learn a lot about
> what sorts of attacks are out there, too! Enjoy!
>
> 8. As you keep up with security in the great big world, especially Microsoft
> patches, be aware which ones are critical or wormable or already exploited
> and watch your vendor release notes for alerts based on those patches.
> Really think about immediately blocking those alerts the next chance you
> get. This can give your patch team some added breathing room.
>
> Lastly, you don't *need* heavy change management on these things if you're
> really careful, but I would wonder how valuable the IPS will be if you're
> too careful, you know?

MBD - from a technical perspective you're right-ish.  Once an IPS is
setup it can run in a fairly low needs mode... however, I'm still
saying that a strong config management for such a device is critical.
If you can't follow who approved a rule & why, Bad Things will happen
the day you get a misconfigured rule which is a DoS for some component
of your network.  Maybe I've just worked at unfortunate places, but
IPSs are remembered not for the thousands of attacks they stop, but
for how much "pain, suffering, and woe" (to quote a former coworker)
they inflict on the network.

Don't get me wrong.  I love me some IPS... it's just that so often
folks drop them on the network and don't do the proper care and
feeding of them that they quickly loose their potential.  And yes, I
do believe change management is an integral part of this!