Re: XFF/ExtraData not always logged for drop rules

["Carter Waxman (cwaxman)" <cwaxman () cisco com>]

Hi Mike,

Thanks for the detailed analysis. We will be tracking this internally.

-Carter

From: Mike Cox <mike.cox52 () gmail com<mailto:mike.cox52 () gmail com>>
Date: Thursday, June 25, 2015 at 2:00 PM
To: "snort-devel () lists sourceforge net<mailto:snort-devel () lists sourceforge net>" <snort-devel () lists 
sourceforge net<mailto:snort-devel () lists sourceforge net>>
Subject: Re: [Snort-devel] XFF/ExtraData not always logged for drop rules

I did some quick tests with Snort 2.9.7.3/Stream6<http://2.9.7.3/Stream6> and saw the same behavior described above.

Any insights, agreements, or disagreements?

Testing scenario 1 is easy enough.  For scenario 2, I've attached a pcap with a simple HTTP request (with XFF header) 
and response that both require reasembly.

These rules can be used to test (only enable one at a time):

drop tcp any any -> any $HTTP_PORTS (msg:"Client request with XFF - reassembly required"; flow:established,to_server; 
content:"GET"; http_method; content:"requestmatch"; sid:36332;)

drop tcp any $HTTP_PORTS -> any any (msg:"Server response XFF test - reassembly required"; flow:established,to_client; 
content:"Server"; http_header; file_data; content:"ninja"; sid:26221;)

If 'normalize: tcp' is enabled (flush policy is *-IPS/pre-ACK), the the rules will trigger but the ExtraData with the 
XFF info is not logged.  If you change the rules to 'alert' or if you disable normalize so the sensor is in post-ACK 
mode, the ExtraData/XFF info is logged.  You'll want to run in inline mode (the dump DAQ is great for testing!) and of 
course you need to be configured to output unified2.

Thanks!

-Mike Cox

On Wed, Jun 24, 2015 at 9:49 AM, Mike Cox <mike.cox52 () gmail com<mailto:mike.cox52 () gmail com>> wrote:

VRT and Snort Devs,

The X-Forwared-For ("XFF") and/or True-Client-IP data is stored in what is called "ExtraData" and written to the 
unified2 file.  ExtraData is often not written to the unified2 file at the same time as other alert data and can depend 
on stream flushing as well as alert flushing.

Looking at Stream5, there are a number of cases where the XFF ExtraData is not logged on 'drop' rules even though it is 
available in the data stream.  From what I can tell, the ExtraData gets written by purge_alerts() which gets called by 
purge_to_seq() which gets called by purge_flushed_ackd() (sometimes these functions get called when dealing with 
TIME_WAIT timer stuff but let's ignore those cases for now).  However, purge_flushed_ackd() only purges flushed and 
acked bytes (as I understand it, bytes that are flushed and ackd).

So in these situations, when 'drop' rules trigger, the ExtraData is not written:

1) A single packet triggers the drop rule and it is inspected as a packet and not part of a (reassembled) stream.
  - The Stream5 preprocessor doesn't get involved enough to do the appropriate flushing/purging required to write the 
ExtraData.

2) A reassembled stream "packet" triggers the drop rule and the normalize_tcp preprocessor is enabled (i.e. 
'normalize_tcp: ips' which is going to be the Protocol-IPS or Footprint-IPS flush policy depending on PAF).
  - Snort is in pre-ACK mode and so it doesn't wait on the ACK to flush the data to detection.  Since the flushing 
happens before the ACK is received (and the ACK isn't processed anyway since the stream is blocked by the block rule), 
the ExtraData never gets written.

I can understand and somewhat accept why the ExtraData isn't written for scenario 1 although this happens when the HTTP 
Inspect preprocessor is already engaged so it seems feasible to log the ExtraData/XFF.  Can this be done?

For scenario 2, can I make a feature request that the ExtraData gets logged appropriately in this case?  I'm guessing 
that people who run Snort inline also have normalize and PAF enabled and it makes sense to me that 'drop' rules would 
still write ExtraData, especially since Stream5 is fully involved.

Once drop rules fire, the stream gets blocked (assuming the DAQ supports this) so it makes sense to go ahead and 
compile/write out the ExtraData since nothing else on that stream is going to get fully processed.

I haven't looked much at Stream6 although it looks like most of the code from Stream5 so I'm not sure why the version 
number change.

Thanks!

-Mike Cox

P.S. setting 'flush_on_alert' for Stream5 doesn't seem to have any affect on these scenarios.

------------------------------------------------------------------------------
Monitor 25 network devices or servers for free with OpManager!
OpManager is web-based network management software that monitors 
network devices and physical & virtual servers, alerts via email & sms 
for fault. Monitor 25 devices for free with no restriction. Download now
http://ad.doubleclick.net/ddm/clk/292181274;119417398;o

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!