Re: [snort-devel] Chainning pre-processors

[Emiliano Fausto <emiliano.fausto () gmail com>]

> Hello Hui,
>
>  the "-k none" did the trick, I really really thank you for all your help.
>
>  As a summary (in case anyone else has the some doubts as me), correct me
> if I'm wrong:
>
>  1) It's possible to chain preprocessors and dynamic-preprocesors in
> SNORT. In fact, they are chained automatically by SNORT by default. The
> order will be given for the priority set when they are registered (for
> example with the function AddPreproc()).
>
>  2) The priorities are set in the file preprocids.h.
>
>  3) Using the stream5 preprocessor is as explained in the official
> Stream5.README documentation, we can add that if you are seeing too much
> discarded TCP packets in the statistics, you may want to add the "-k none"
> parameter when starting the snort process (tells snort to ignore checksum
> errors in the TCP packets).
>
>  4) Inside the preprocessor code, you may use the filter: (SFSnortPacket*)
> mypacket->flags & FLAG_REBUILT_STREAM, to be sure that this is a rebuilt
> packet.
>
>  My preprocessor and the TCP reassembling is working perfectly now with
> your help.
>
>  Again Hui, thanks a lot for everything!
>  Emiliano.
>
>
>
> 2013/12/5 Hui Cao <hcao () sourcefire com>
>
> >  Did you see many discards in snort exit. If so, you need to disable
> > checksum. Use -k none when you run the command.
> >
> > Best,
> > Hui.
> >
> > On 12/05/2013 01:40 PM, Emiliano Fausto wrote:
> >
> > Hui,
> >
> >  I was really trying to get Stream5 working and I couldn't, I'm pretty
> > sure that the problem is not related to Stream5, but I just can't get the
> > preprocessor to rebuilt TCP segments.
> >
> >  In wireshark I can see there are lot of TCP segmented packets and if I
> > select to reassemble the HTTP ones, it does it well. Backing to SRNOT, for
> > some reason the statistics that the Stream5 are showing are that there
> > where some TCP sessions, there were discarded TCP packets, etc. But in the
> > line: "TCP Rebuilt packets" shows a 0.
> >
> >  I tried with HTTP so that I can have a known protocol to start with as
> > you suggested me, and tried to download a wallpaper, which (due to its
> > size) was delivered to me in several packets.
> >
> >  I changed the snort.conf configuration file, to look like this:
> >
> >  preprocessor stream5_global: track_tcp yes, \
> >     track_udp no, \
> >     track_icmp no, \
> >      max_tcp 262144, \
> >     max_active_responses 2, \
> >      min_response_seconds 5, \
> >     show_rebuilt_packets
> > preprocessor stream5_tcp: policy linux, \
> >     overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
> >     ports both all
> >
> >  I also played a little bit with my own preprocessor to print the Flags
> > that the packet brings, and they change based on the priority I assign to
> > the AddPreproc(). But I tried different ones, from FIRST to LAST and none
> > of them matched the & FLAG_REBUILT_STREAM flag.
> >
> >  I was looking at the stream5 documentation (the README file), also
> > searched different threads in the snort users and devel community, but none
> > of them helped me.
> >
> >  Do you know where can I get more information or examples on how to use
> > properly the Stream5 preprocess, or if I'm missing something else to make
> > it works? maybe create a generic rule, or something like that?
> >
> >  I really thank you in advance for all your help on this,
> >  Emiliano.
> >
> >
> >
> > 2013/12/5 Hui Cao <hcao () sourcefire com>
> >
> > >  Make sure you have frag3 enabled. You can try the application
> > > preprocessors (such as http, smtp, ssl, sip etc) to see how it works.  If
> > > the packets are reassembled, you can see that from snort exit stats for
> > > stream etc.
> > >
> > > Stream will reassemble the payloads of TCP.
> > >
> > > Best,
> > > Hui.
> > >
> > >
> > > On 12/05/2013 08:26 AM, Emiliano Fausto wrote:
> > >
> > > Hello Hui,
> > >
> > >  you are totally right. I was confused, I didn't have to assemble
> > > fragmented IP packets.
> > >
> > >  So, I configured the Stream5 preprocessor in order it to take into
> > > account the port of my application (it's 9090).
> > >
> > >  I let my AddPreproc() with priority set to PRIORITY_APPLICATION so
> > > that it's registered after the Stream5.
> > >
> > >  Then, I tried to get any packet with (p->flags & FLAG_REBUILT_STREAM),
> > > but none of them matched, it seems that the Stream5 preprocessor isn't
> > > reassembling my TCP segments.
> > >
> > >  The SNORT configuration file is set like this:
> > >
> > >  preprocessor stream5_global: track_tcp yes, \
> > >     track_udp yes, \
> > >     track_icmp no, \
> > >     max_tcp 262144, \
> > >     max_udp 131072, \
> > >     max_active_responses 2, \
> > >     min_response_seconds 5
> > > preprocessor stream5_tcp: policy windows, detect_anomalies, require_3whs
> > > 180, \
> > >     overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
> > >     ports client 21 22, \
> > >     ports both 9090
> > > preprocessor stream5_udp: timeout 180
> > >
> > >  My app works on port 9090.
> > >
> > >  Maybe, the problem is not related to stream5, maybe the problem is
> > > originated because the one in charge of doing the segments reassembly is my
> > > preprocessor, at the application level, and not the stream5 preprocessor at
> > > the TCP level.
> > >
> > >  More on this, into wireshark I can see all the packets of the protocol
> > > TCP segmented, and if I go to preferences-->protocols-->TCP it has an
> > > option that says: "allow dissectors to reassemble TCP". It means that the
> > > reassemble isn't being done at the TCP layer, but into the proper
> > > application.
> > >
> > >  Do you know if SNORT has something of this? Or the Stream5 should
> > > reassemble the payloads of the TCP packets?
> > >
> > >  Thanks a lot in advance,
> > > Emiliano.
> > >
> > >
> > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > >
> > > >  Frag3 will deal with IP fragmentation. You might want reassembled data
> > > > from TCP. This means your preprocessor should be after stream and you
> > > > should check
> > > > (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_STREAM. Snort will only
> > > > reassemble up to the flush point defined (typically 16K). You can't expect
> > > > snort will give all the data from one reassembled packet.
> > > >
> > > > Fragmentation from wireshark might mean TCP segmentation. You might get
> > > > full IP packets.
> > > >
> > > > Best,
> > > > Hui.
> > > >  On 12/04/2013 03:17 PM, Emiliano Fausto wrote:
> > > >
> > > > Hui,
> > > >
> > > >  yes, I understood what you told me about filtering just the
> > > > de-fragmented packets, that's why I added these 3 lines:
> > > >
> > > >  SFSnortPacket *p = (SFSnortPacket *)pkt;
> > > > int fla = (p->flags & FLAG_REBUILT_FRAG);
> > > > _dpd.logMsg("Flags set? %d",fla);
> > > >
> > > >  But I generate fragmented packets (which for instance Wireshark
> > > > detects and show as fragmented and the possibility of reassembly them) and
> > > > I'm always getting with each one the output: "Flags set? 0".
> > > >
> > > >  I receive all the fragmented packets, but not the last one
> > > > reassembled.
> > > >
> > > >  Regards,
> > > > Emiliano.
> > > >
> > > >
> > > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > > >
> > > > > It will still get all packets including the raw packets. You need to
> > > > > use packet flags to filter them.
> > > > >
> > > > > Best,
> > > > > Hui.
> > > > >
> > > > > On Wed, Dec 4, 2013 at 3:00 PM, Emiliano Fausto
> > > > >  <emiliano.fausto () gmail com> wrote:
> > > > > > Yes,
> > > > > >
> > > > > > that's what I thought, but for some reason the TCP packets keep on
> > > > > coming
> > > > > > fragmented to my preprocessor.
> > > > > >
> > > > > > No I took off all the preprocessors and just let the frag3 and
> > > > > mines, I'll
> > > > > > try to figure out if they are called in the correct order, but they
> > > > > are not
> > > > > > being chained.
> > > > > >
> > > > > > Thanks,
> > > > > > Emiliano.
> > > > > >
> > > > > >
> > > > > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > > > > > > It looks good to me.
> > > > > > >
> > > > > > > Best,
> > > > > > > Hui.
> > > > > > >
> > > > > > > On Wed, Dec 4, 2013 at 2:44 PM, Emiliano Fausto
> > > > > > > <emiliano.fausto () gmail com> wrote:
> > > > > > > > Hi Hui,
> > > > > > > >
> > > > > > > > I've seen that I was using PRIORITY_TRANSPORT, which is lower
> > > > > than the
> > > > > > > > PRIORITY_NETWORK that uses frag3.
> > > > > > > >
> > > > > > > > Anyway, I put the priority: PRIORITY_LAST to my own preprocessor,
> > > > > but
> > > > > > > > when
> > > > > > > > the TCP packets keep arriving fragmented to my preprocessor.
> > > > > > > >
> > > > > > > > Is there anything else I should take into account?
> > > > > > > >
> > > > > > > > Thanks in advance,
> > > > > > > > Emiliano.
> > > > > > > >
> > > > > > > >
> > > > > > > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > > > > > > > > In sr/preprocids.h
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Hui.
> > > > > > > > >
> > > > > > > > > On 12/04/2013 02:36 PM, Emiliano Fausto wrote:
> > > > > > > > >
> > > > > > > > > Great,
> > > > > > > > >
> > > > > > > > > so, the pre-processors are "chained" by default, and the order
> > > > > that
> > > > > > > > > SNORT
> > > > > > > > > follows to call them is set by the PRIORITY variable.
> > > > > > > > >
> > > > > > > > > Do you know where's defined this PRIORITY variable? Because I
> > > > > saw that
> > > > > > > > > the
> > > > > > > > > frag3 is being registered with PRIORITY_NETWORK, so I'd like to
> > > > > set the
> > > > > > > > > priority of my own preprocessor as (PRIORITY_NETWORK -1).
> > > > > > > > >
> > > > > > > > > Thanks in advance,
> > > > > > > > > Emiliano
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > > > > > > > > > sc means snort configuration. We use PRIORITY to sort the
> > > > > processing.
> > > > > > > > > > All
> > > > > > > > > > processors enabled will be called and processed based on
> > > > > priority. You
> > > > > > > > > > have
> > > > > > > > > > to rely on the code to figure out what exactly snort does.
> > > > > > > > > >
> > > > > > > > > > The checking is correct. You will only process rebuilt packets.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Hui.
> > > > > > > > > > On 12/04/2013 02:19 PM, Emiliano Fausto wrote:
> > > > > > > > > >
> > > > > > > > > > Hello Hui,
> > > > > > > > > >
> > > > > > > > > > thanks a lot for your answer.
> > > > > > > > > >
> > > > > > > > > > Right now I have registered my preprocessor (let's call it
> > > > > > > > > > examplePreprocess as you said, because right now I'm using the
> > > > > one
> > > > > > > > > > provided
> > > > > > > > > > with the DPX) with this line:
> > > > > > > > > >
> > > > > > > > > > _dpd.addPreproc(ExampleProcess, PRIORITY_TRANSPORT, 10000,
> > > > > > > > > > PROTO_BIT__TCP);
> > > > > > > > > >
> > > > > > > > > > So, the only change is to add previous to the parameter
> > > > > > > > > > ExampleProcess,
> > > > > > > > > > the "sc". What does it mean? Do you know if there's any
> > > > > documentation
> > > > > > > > > > about
> > > > > > > > > > this chaining preprocesses?
> > > > > > > > > >
> > > > > > > > > > So, checking the flags, should be:
> > > > > > > > > >
> > > > > > > > > > (SFSnortPacket*)tcppacket)->flags & FLAG_REBUILT_FRAG
> > > > > > > > > >
> > > > > > > > > > right?
> > > > > > > > > >
> > > > > > > > > > Thanks again!
> > > > > > > > > > Emiliano.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Then, I'll have to register my own preprocessor where?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2013/12/4 Hui Cao <hcao () sourcefire com>
> > > > > > > > > > > Yes, it is possible. You can register you preprocessor like
> > > > > this:
> > > > > > > > > > > _dpd.addPreproc( sc, ExampleProcess, PRIORITY_TRANSPORT,
> > > > > You_PP_ID,
> > > > > > > > > > > PROTO_BIT__IP );
> > > > > > > > > > >
> > > > > > > > > > > Remember check the following flag in your ExampleProcess:
> > > > > > > > > > >
> > > > > > > > > > > (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_FRAG
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Hui.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 12/04/2013 12:52 PM, Emiliano Fausto wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi everybody,
> > > > > > > > > > >
> > > > > > > > > > > I'm creating a new preprocessor which needs to have the whole
> > > > > content
> > > > > > > > > > > in
> > > > > > > > > > > a packet which was fragmented.
> > > > > > > > > > >
> > > > > > > > > > > So I thought of using the frag3 preprocessor to re-assembly the
> > > > > > > > > > > packets,
> > > > > > > > > > > and then, when this reassembly is done, sent it to my own
> > > > > > > > > > > preprocessor.
> > > > > > > > > > >
> > > > > > > > > > > Do you know if this is possible? May I have the output of
> > > > > frag3 being
> > > > > > > > > > > the input of my own preprocessor?
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Emiliano.
> > > > > ------------------------------------------------------------------------------
> > > > > > > > > > > Sponsored by Intel(R) XDK
> > > > > > > > > > > Develop, test and display web and hybrid apps with a single
> > > > > code
> > > > > > > > > > > base.
> > > > > > > > > > > Download it for free now!
> > > > > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Snort-devel mailing list
> > > > > > > > > > > Snort-devel () lists sourceforge net
> > > > > > > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > > > > > > Archive:
> > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > > > > > > > > > > Please visit http://blog.snort.org for the latest news about
> > > > > Snort!
> > > > > > > > > > >
> > > > > ------------------------------------------------------------------------------
> > > > > > > > > > > Sponsored by Intel(R) XDK
> > > > > > > > > > > Develop, test and display web and hybrid apps with a single
> > > > > code
> > > > > > > > > > > base.
> > > > > > > > > > > Download it for free now!
> > > > > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Snort-devel mailing list
> > > > > > > > > > > Snort-devel () lists sourceforge net
> > > > > > > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > > > > > > Archive:
> > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > > > > > > > > > > Please visit http://blog.snort.org for the latest news about
> > > > > Snort!
> > > > > > > > > >
------------------------------------------------------------------------------
Sponsored by Intel(R) XDK 
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!