Re: 2 questions about Stream5 handling of missing data

[John Eure <john.eure () gmail com>]

I've got a repro pcap; I'll send it to you with a conf file tomorrow.

My workaround for the PAF reset is, so far, to remove the code in
snort_stream5_tcp.c::UpdateFlushMgr() that disables PAF when it sees
TF_PKT_MISSED.  And then to alter stream5_paf.c::s5_paf_check(), by
removing the "we have a gap" handling, and instead wiping out my user PAF
state, and setting the PAF position to the end of the gap.  The idea is to
keep the new Stream5 behavior of flushing everything before the gap, skip
over the gap, and then to have PAF treat everything after the gap as an
entirely new TCP connection that it picked up in midstream.  I'm still
doing a lot of testing on it, but if you'd like to take a look, I can pass
it along?

I'm running snort in passive mode, and the gaps are most definitely *not*
snort's fault.  Snort is listening on an interface that's connected to a
switch on a spanning port, and the switch has been producing some
anomalies.  There are duplicate packets, which Stream5 handles just fine.
There may be a few dropped packets, which is one of the places where this
improvement would be useful.  There's some trivial packet re-ordering,
which Stream5 also appears to handle well, although I'm not 100% confident
yet.  And then there's some serious packet reordering, where the spanning
port is occasionally delivering a TCP ack packet before the data packet
that it was supposed to ack.  AFAIK, that isn't legitimate TCP, so it's not
surprising that Stream5 doesn't handle it (Wireshark certainly gets
confused).  But I've got to try to make it work anyway.  :)  Of course, the
"right" solution is to find out what's up with the switch, but that's
largely out of my hands.

(At least I was able to turn off GRO on the local interface; - thank you,
Sourcefire, for documenting in the manual that GRO could be a problem.)

I definitely understand the desire to avoid churning through midstream
data, especially when dealing with malicious traffic and baroque protocols,
but for this use case and for the protocols I'm working with, recovery is
quite easy.  There are a number of persistent TCP connections, with
requests and responses sent over it.  Some of the connections use
HTTP/SOAP, and the others use protocols where the messages all start with a
multi-byte identifier, and in every case I've seen so far, messages have
started at the beginning of a TCP packet.  And since, for this use case, I
don't have to worry about malicious traffic that's trying to deceive me, I
can just quickly skim through the packets and find the next identifier, and
then pick up parsing there.  It's kind of like UTF-8, in that respect - if
I don't mind losing the rest of the message that had the missing data, I
can easily resume parsing at the next message.  Does that make sense?

Thanks,
John


On Mon, Feb 3, 2014 at 1:22 PM, Russ Combs <rcombs () sourcefire com> wrote:

> John,
>
> Thanks for the detailed query.  Comments inline.
>
> Russ
>
>
> On Thu, Jan 30, 2014 at 10:51 PM, John Eure <john.eure () gmail com> wrote:
>
> > Hello,
> >
> > I've been using snort to do some custom traffic inspection for a client,
> > and have written a few plugins, including a preprocessor plugin that uses
> > Stream5's Protocol Aware Flushing (PAF).  I've been testing out the new
> > release (2.9.6.0), and encountered a behavior that I hadn't seen before,
> > and I'd like to find out whether it's a bug, or whether it's something I
> > should be expecting.
> >
> > Normally, every time my preprocessor plugin sees a packet, the Packet
> > struct has been zeroed out (up to the ip_options field) and then filled
> > with new data, so I get a clean struct each time.  In this release, Stream5
> > got some improvements in how it handles missing data.  (Thank you,
> > Sourcefire!)  But when that new handling is triggered, I'm seeing packets
> > that haven't been completely zeroed out.  Specifically, it's the Stream5
> > rebuilt pseudo-packet that is generated after the gap in the data, which
> > hasn't been zeroed out before the new data was added.
> >
> > I've been using a bit field (the preproc_reassembly_pkt_bits field) in
> > the Packet struct to mark packets as having been accepted or rejected by my
> > preprocessor, and so I was surprised to find that the bit field wasn't
> > reset in between packets.  Is this normal behavior that I should be
> > expecting?
>
> That looks like a bug that should get fixed.  Do you have a repro pcap and
> conf?
>
> > Also, I've got a second, more general question, for Sourcefire.  After
> > Stream5 detects missing data on a stream, PAF gets "reset", and the flush
> > policy gets set to  STREAM_FLPOLICY_FOOTPRINT, and never goes back to
> > STREAM_FLPOLICY_PROTOCOL again.  So far, I've been able to work around
> > this, but I'd much rather have a solid fix in place.  So I was wondering,
> > is this on your roadmap for future development?  At the very least, now you
> > know there's at least one person interested in that feature.  :)
>
> This will be re-examined at some point.  What is your workaround here?
>
> Also, are you running passive?  Are the gaps due to Snort drops or what?
>
> The intent was to not chew through session data looking for a match after
> initial failure for performance and to avoid the general confusion from
> midstream data.  Does your protocol have a unique way of synchronizing in
> such cases?
>
>
> > Thanks,
> > John Eure
> >
> >
> >
> > ------------------------------------------------------------------------------
> > WatchGuard Dimension instantly turns raw network data into actionable
> > security intelligence. It gives you real-time visual feedback on key
> > security issues and trends.  Skip the complicated setup - simply import
> > a virtual appliance and go from zero to informed in seconds.
> >
> > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Snort-devel mailing list
> > Snort-devel () lists sourceforge net
> > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > Archive:
> > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> >
> > Please visit http://blog.snort.org for the latest news about Snort!
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!