RE: False Positives with IntruVert

[Bill Boyle <bboyle () intruvert com>]

To make it easier to follow, I have extracted your points and responded in
line.  Once again, these are my views and may/may not be those of my
company.

--bill

-----Original Message-----
From: Paul Schmehl [mailto:pauls () utdallas edu] 
Sent: Monday, March 31, 2003 5:30 PM
To: Alan Shimel
Cc: Bill Boyle; focus-ids () securityfocus com
Subject: RE: False Positives with IntruVert

On Sat, 2003-03-29 at 10:52, Alan Shimel wrote:
Since we're now into the congratulatory mode, why don't I address some
of Bill's points?


If we *knew* there would never be a false positive, then the answer
would obviously be we should.  Of course we should.  The problem is we
don't *know* that, and a vendor saying it's true isn't exactly a ringing
endorsement.  (No offense, but salesman aren't exactly a good source for
technical information like, "show me why you can say there are 0 false
positives.)

BOYLE--------------------\
You are right.  There has to be trust because the average admin will have
neither the time nor the skill to assess properly the entire signature
database.  If they did, they could write their own IDS/IPS. :)  

Have you reversed engineered your firewall code?  Have you analyzed your
antivirus signatures?  When you get a warning that an attachment is infected
with Nimda, how do you respond?  The fact that the trust has to be earned is
a different statement than IPS is not functional or not worth time or money.
I am sure there are plenty of readers who would have saved a large chunk of
change and time if they blocked code red or Nimda.
BOYLE--------------------/

No matter how important security becomes to an institution, it will
never trump availability.  After all, the purpose of a network isn't to
prevent attacks, it's to get work done.  If preventing attacks means
preventing work from being done, then we're into the risk analysis
arena, weighing the odds of attack against the need for the service.

For example, we elected four years ago to block email that had
attachments with certain "dangerous" extensions (such as .exe).  However
we did not block attachments with .doc or .xls, because even though they
can also carry viruses, the business need for availability was greater
(in our judgment) than the need to protect us at the gateway.

BOYLE----------------------\
Great statement!  The purpose of security is to ensure availability.  I
agree with the risk analysis statement--that is exactly my point.  However,
when I read your post I am confused.  You state that risk analysis is
valuable but I still see a blanket statement from you that IPS should not be
used because it might block something valid.  

Using your example applied to IPS, you are saying allow ALL attachments with
.doc and .exe because denying them MIGHT hinder productivity.  Yet, you have
done risk analysis and determined that you are better off blocking .exe.
Risk Analysis.  SELECTIVELY blocking attacks with an IPS will not only
minimize your risk of a false positive denying a service, it will IMPROVE
uptime by keeping the malicious traffic off of the wire.  In fact, the
multiple signatures of IPS permits a GREATER level of accuracy than simply
=.exe.  For example: Code Red could wreak havoc on your network.  With risk
analysis, I would feel pretty safe dropping packets with ida in the URI Path
and
\x90\x90\x68\x58\xcb\xd3\x78\x01\x90\x90\x68\x58\xcb\xd3\x78\x01\x90\x90\x68
\x58\xcb\xd3\x78\x01\x90\x90\x90\x90\x81\x90\xc3\x03\x8b\x00\x53\x1b\x53\xff
\x78\x00\x25\x75\x30\x30 in the request.
BOYLE------------------/

"More accurately" does not mean 100% however.

BOYLE-------------------\
Correct.  Nothing in totality is 100% but subsets of IPS can be, such as
individual attacks.  Reiterating my previous post--you would not set EVERY
attack to block.  You would choose attacks that are less likely to false
positive.  Regarding a good point from Brian Lang's email: How does the
average admin know which signatures will false positive and which will not?
Hopefully, the IPS vendor (who DOES know the attack inside out and has
profiled the attacks (signature or anomaly or combination of both)) has
rated the degree which the attack may be a false positive.  It helps the
admin make the decision.
BOYLE-------------------/

Why?  An IPS that does no blocking is an IDS by another name.

BOYLE---------------------\
I would agree that before you can block, you must accurately detect.  What
might clarify my statement is that waiting for a product with 0 false
positives is the wrong approach.  There are other factors that I feel are
more critical and, judging by your post, you agree with at least the
availability portion.  It is the accurate subset of attacks that makes IPS a
useful solution today.  
BOYLE---------------------/

And that is a *huge* consideration for most networks.  A single point of
failure for your *entire* Internet pipe is a *big* concern.  (Yes, you
can use load balancing and redundancy to overcome that problem, but at a
much higher cost.)

BOYLE----------------------\
Risk Analysis.  A single router or switch or firewall or... is a single
point of failure as well.  I guess what I am saying is that you are not
adding a SPOF to an environment that does not have it already.  Most
companies already have several single points of failure.  IPS can be run in
High Availability mode for those environments that necessitate that level of
redundancy.  If you are calling stability into question, there are IPS
solutions out there that are just as stable as the gear listed above.  
BOYLE-----------------------/
 
A NIDS sniffing the wire poses no threat to connectivity.

BOYLE------------------------\
Yes.  On the contrary, a NIDS cannot prevent all the other things on that
wire that do pose a threat to connectivity.  Your line of thinking would
eliminate value-adding ACLs, firewalls, antivirus, etc.
BOYLE------------------------/

For IDS, but not for blocking - unless you are willing to accept the
consequences of degraded availability.  Again, we're in to risk
analysis.

BOYLE--------------------------\
"Degraded availability"...I would make the argument that the protection an
IPS provides would IMPROVE availability.  I do agree with the statement
regarding the more generic signatures are more appropriate for IDS than IPS.
It is not all or nothing.  I would configure those generic attacks to alert
only because they will require grey-matter to be certain.
BOYLE--------------------------/

Out of curiosity, what alerts *would* you block and why?

BOYLE---------------------------\
That is a tough question to answer because one could not say "buffer
overflows" or "worms" or "backdoors".  For the same reasons, I could not
even say individual attacks like "code red" or "back orifice".  Each of
these might have multiple signatures to detect on the request, response,
probe, propagation, etc.  It might use an anomaly algorithm instead of a
signature.  It may vary by vendor.

My approach would be as follows:
Run the device in detection-only to tune the box to eliminate any glaring
false positives.  

After it has been in my network for an appropriate amount of time (mileage
may vary based upon the dynamics of traffic), I would put it in-line not
blocking anything.  

I would let it run in this fashion for a while to avoid any finger pointing
for every little thing that goes wrong on a daily basis.

Meanwhile, I would begin to choose what attacks I would begin to block.
Severity will be an important factor for the first round.  In a Microsoft
environment, I would probably start with more recent very well known attacks
such as code red, nimda, slammer, fbi top 20 attacks, etc.  I would pay
attention to vendor notes--hopefully there is a rating on the chance the
attack signature will be a false positive.  Those deemed good candidates, I
would read up on some of them using CVE references, bugtraq, Microsoft Tech
Bulletins, etc. to determine my exposure.

Once I have determined high severity attacks that are applicable to my
environment, I would assess how I would apply policy.  
        Which devices does this particular signature affect?  If only
Microsoft then only block if destined to/from a MS box.  Ignore or alert
only for Unix boxes.
        Do I care about attacks originating inside my environment?  Maybe it
is too political to block traffic outbound so I will only drop on inbound
malicious traffic.
        Do I only want to block traffic to my key web server and mail
server?  Personally protect devices I am responsible for without causing
political issues with the development and engineering departments.
        Are there key segments/IPs that I need to ignore?  Risk Analysis.
        
I would apply the blocking rules slowly to aid in troubleshooting should an
application stop working.  It may not even be because of the IPS but it is
much easier to determine that the telnet problem that arose last night is
not your issue if all you added was a mssql rule.

I would watch the console and look at the captured packet data of the
blocked traffic to help determine whether the traffic is a false positive or
maybe to prove that no action was taken by the IPS and the above telnet
issue was related to something other than IPS.
        
Some people may see value in just permitting everything until the next
worm/vulnerability and will add the signature after the attack to help
contain their exposure.

BOYLE------------------------------/    

-- 
Paul Schmehl (pauls () utdallas edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdallas.edu/~pauls/
AVIEN Founding Member

-----Original Message-----
From: Paul Schmehl [mailto:pauls () utdallas edu] 
Sent: Monday, March 31, 2003 5:30 PM
To: Alan Shimel
Cc: Bill Boyle; focus-ids () securityfocus com
Subject: RE: False Positives with IntruVert

On Sat, 2003-03-29 at 10:52, Alan Shimel wrote:
> Bill
>
> Its not often that I agree with the competition, but your email was one
> of the most constructive, right on defense for IPS that I have seen.
> Nice points!
Since we're now into the congratulatory mode, why don't I address some
of Bill's points?
> -----Original Message-----
> From: Bill Boyle [mailto:bboyle () intruvert com] 
> Sent: Friday, March 28, 2003 1:18 PM
> To: 'focus-ids () securityfocus com'
> Subject: RE: False Positives with IntruVert 
>
> There are clearly attacks that will 100% not false positive.  My
> question to
> the community is: Why would you NOT block this traffic on your network?

If we *knew* there would never be a false positive, then the answer
would obviously be we should.  Of course we should.  The problem is we
don't *know* that, and a vendor saying it's true isn't exactly a ringing
endorsement.  (No offense, but salesman aren't exactly a good source for
technical information like, "show me why you can say there are 0 false
positives.)

No matter how important security becomes to an institution, it will
never trump availability.  After all, the purpose of a network isn't to
prevent attacks, it's to get work done.  If preventing attacks means
preventing work from being done, then we're into the risk analysis
arena, weighing the odds of attack against the need for the service.

For example, we elected four years ago to block email that had
attachments with certain "dangerous" extensions (such as .exe).  However
we did not block attachments with .doc or .xls, because even though they
can also carry viruses, the business need for availability was greater
(in our judgment) than the need to protect us at the gateway.

> We
> have all been burned with false positives and I am sure that is where
> the
> mistrust originates.  But, a lot of technology has been put forth to
> more
> accurately identify attacks since IDS's inception.  
"More accurately" does not mean 100% however.

> What I see as necessary for IPS to be useful TODAY is not zero false
> positives.  It is performance, accuracy, and granularity of policy.
Why?  An IPS that does no blocking is an IDS by another name.

> PERFORMANCE:  First and foremost, the box sitting on the wire must
> process
> the packets at wire speed and not drop ANY packets. It must also have
> little
> downtime (same expectations as my switch and router or firewall
> infrastructure).  
And that is a *huge* consideration for most networks.  A single point of
failure for your *entire* Internet pipe is a *big* concern.  (Yes, you
can use load balancing and redundancy to overcome that problem, but at a
much higher cost.)

A NIDS sniffing the wire poses no threat to connectivity.

> ACCURACY:  Exemplified in Paul Schmehl's email, a signature alone is not
> always accurate enough.  Correlation must be used to enhance the
> accuracy.
> For example, Intruvert uses Boolean logic in its signature set to
> provide
> more accurate detection.  Example: sig#1 in URI request AND (sig2 OR
> sig3 in
> body).
>
> Not every signature is 100% accurate but it does not mean it is not of
> use.

For IDS, but not for blocking - unless you are willing to accept the
consequences of degraded availability.  Again, we're in to risk
analysis.

> Example: a TCP connection on port 5432.  It may be backoriface or may be
> a
> random source port that happened to be a well know backoriface port.  It
> will be a false positive but one that I would want to review.  The fact
> that
> I got this alert does not mean my IPS is worthless, it just means that I
> will not choose to automatically block this event.  Other backoriface
> attacks that have a lower false positive rate will (at MY choosing-not
> the
> vendor's).  
Out of curiosity, what alerts *would* you block and why?

-- 
Paul Schmehl (pauls () utdallas edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdallas.edu/~pauls/
AVIEN Founding Member

-----------------------------------------------------------
ALERT: Exploiting Web Applications- A Step-by-Step Attack Analysis
Learn why 70% of today's successful hacks involve Web Application
attacks such as: SQL Injection, XSS, Cookie Manipulation and Parameter 
Manipulation.
http://www.spidynamics.com/mktg/webappsecurity71