Re: [snort-devel] Chainning pre-processors

[Hui Cao <hcao () sourcefire com>]

Can you give me pcap and your configurations to take a  look?

Best,
Hui.

On 12/05/2013 01:40 PM, Emiliano Fausto wrote:
> Hui,
>
> I was really trying to get Stream5 working and I couldn't, I'm pretty 
> sure that the problem is not related to Stream5, but I just can't get 
> the preprocessor to rebuilt TCP segments.
>
> In wireshark I can see there are lot of TCP segmented packets and if I 
> select to reassemble the HTTP ones, it does it well. Backing to SRNOT, 
> for some reason the statistics that the Stream5 are showing are that 
> there where some TCP sessions, there were discarded TCP packets, etc. 
> But in the line: "TCP Rebuilt packets" shows a 0.
>
> I tried with HTTP so that I can have a known protocol to start with as 
> you suggested me, and tried to download a wallpaper, which (due to its 
> size) was delivered to me in several packets.
>
> I changed the snort.conf configuration file, to look like this:
>
> preprocessor stream5_global: track_tcp yes, \
>   track_udp no, \
>   track_icmp no, \
>     max_tcp 262144, \
>   max_active_responses 2, \
>   min_response_seconds 5, \
>   show_rebuilt_packets
> preprocessor stream5_tcp: policy linux, \
>   overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
>   ports both all
>
> I also played a little bit with my own preprocessor to print the Flags 
> that the packet brings, and they change based on the priority I assign 
> to the AddPreproc(). But I tried different ones, from FIRST to LAST 
> and none of them matched the & FLAG_REBUILT_STREAM flag.
>
> I was looking at the stream5 documentation (the README file), also 
> searched different threads in the snort users and devel community, but 
> none of them helped me.
>
> Do you know where can I get more information or examples on how to use 
> properly the Stream5 preprocess, or if I'm missing something else to 
> make it works? maybe create a generic rule, or something like that?
>
> I really thank you in advance for all your help on this,
> Emiliano.
>
>
>
> 2013/12/5 Hui Cao <hcao () sourcefire com <mailto:hcao () sourcefire com>>
>
>     Make sure you have frag3 enabled. You can try the application
>     preprocessors (such as http, smtp, ssl, sip etc) to see how it
>     works. If the packets are reassembled, you can see that from snort
>     exit stats for stream etc.
>
>     Stream will reassemble the payloads of TCP.
>
>     Best,
>     Hui.
>
>
>     On 12/05/2013 08:26 AM, Emiliano Fausto wrote:
> >     Hello Hui,
> >
> >     you are totally right. I was confused, I didn't have to assemble
> >     fragmented IP packets.
> >
> >     So, I configured the Stream5 preprocessor in order it to take
> >     into account the port of my application (it's 9090).
> >
> >     I let my AddPreproc() with priority set to PRIORITY_APPLICATION
> >     so that it's registered after the Stream5.
> >
> >     Then, I tried to get any packet with (p->flags &
> >     FLAG_REBUILT_STREAM), but none of them matched, it seems that the
> >     Stream5 preprocessor isn't reassembling my TCP segments.
> >
> >     The SNORT configuration file is set like this:
> >
> >     preprocessor stream5_global: track_tcp yes, \
> >         track_udp yes, \
> >         track_icmp no, \
> >         max_tcp 262144, \
> >         max_udp 131072, \
> >         max_active_responses 2, \
> >         min_response_seconds 5
> >     preprocessor stream5_tcp: policy windows, detect_anomalies,
> >     require_3whs 180, \
> >         overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
> >         ports client 21 22, \
> >         ports both 9090
> >     preprocessor stream5_udp: timeout 180
> >
> >     My app works on port 9090.
> >
> >     Maybe, the problem is not related to stream5, maybe the problem
> >     is originated because the one in charge of doing the segments
> >     reassembly is my preprocessor, at the application level, and not
> >     the stream5 preprocessor at the TCP level.
> >
> >     More on this, into wireshark I can see all the packets of the
> >     protocol TCP segmented, and if I go to
> >     preferences-->protocols-->TCP it has an option that says: "allow
> >     dissectors to reassemble TCP". It means that the reassemble isn't
> >     being done at the TCP layer, but into the proper application.
> >
> >     Do you know if SNORT has something of this? Or the Stream5 should
> >     reassemble the payloads of the TCP packets?
> >
> >     Thanks a lot in advance,
> >     Emiliano.
> >
> >
> >     2013/12/4 Hui Cao <hcao () sourcefire com <mailto:hcao () sourcefire com>>
> >
> >         Frag3 will deal with IP fragmentation. You might want
> >         reassembled data from TCP. This means your preprocessor
> >         should be after stream and you should check
> >         (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_STREAM. Snort
> >         will only reassemble up to the flush point defined (typically
> >         16K). You can't expect snort will give all the data from one
> >         reassembled packet.
> >
> >         Fragmentation from wireshark might mean TCP segmentation. You
> >         might get full IP packets.
> >
> >         Best,
> >         Hui.
> >         On 12/04/2013 03:17 PM, Emiliano Fausto wrote:
> > >         Hui,
> > >
> > >         yes, I understood what you told me about filtering just the
> > >         de-fragmented packets, that's why I added these 3 lines:
> > >
> > >         SFSnortPacket *p = (SFSnortPacket *)pkt;
> > >         int fla = (p->flags & FLAG_REBUILT_FRAG);
> > >         _dpd.logMsg("Flags set? %d",fla);
> > >
> > >         But I generate fragmented packets (which for instance
> > >         Wireshark detects and show as fragmented and the possibility
> > >         of reassembly them) and I'm always getting with each one the
> > >         output: "Flags set? 0".
> > >
> > >         I receive all the fragmented packets, but not the last one
> > >         reassembled.
> > >
> > >         Regards,
> > >         Emiliano.
> > >
> > >
> > >         2013/12/4 Hui Cao <hcao () sourcefire com
> > >         <mailto:hcao () sourcefire com>>
> > >
> > >             It will still get all packets including the raw packets.
> > >             You need to
> > >             use packet flags to filter them.
> > >
> > >             Best,
> > >             Hui.
> > >
> > >             On Wed, Dec 4, 2013 at 3:00 PM, Emiliano Fausto
> > >             <emiliano.fausto () gmail com
> > >             <mailto:emiliano.fausto () gmail com>> wrote:
> > >             > Yes,
> > >             >
> > >             > that's what I thought, but for some reason the TCP
> > >             packets keep on coming
> > >             > fragmented to my preprocessor.
> > >             >
> > >             > No I took off all the preprocessors and just let the
> > >             frag3 and mines, I'll
> > >             > try to figure out if they are called in the correct
> > >             order, but they are not
> > >             > being chained.
> > >             >
> > >             > Thanks,
> > >             > Emiliano.
> > >             >
> > >             >
> > >             > 2013/12/4 Hui Cao <hcao () sourcefire com
> > >             <mailto:hcao () sourcefire com>>
> > >             >>
> > >             >> It looks good to me.
> > >             >>
> > >             >> Best,
> > >             >> Hui.
> > >             >>
> > >             >> On Wed, Dec 4, 2013 at 2:44 PM, Emiliano Fausto
> > >             >> <emiliano.fausto () gmail com
> > >             <mailto:emiliano.fausto () gmail com>> wrote:
> > >             >> > Hi Hui,
> > >             >> >
> > >             >> > I've seen that I was using PRIORITY_TRANSPORT,
> > >             which is lower than the
> > >             >> > PRIORITY_NETWORK that uses frag3.
> > >             >> >
> > >             >> > Anyway, I put the priority: PRIORITY_LAST to my own
> > >             preprocessor, but
> > >             >> > when
> > >             >> > the TCP packets keep arriving fragmented to my
> > >             preprocessor.
> > >             >> >
> > >             >> > Is there anything else I should take into account?
> > >             >> >
> > >             >> > Thanks in advance,
> > >             >> > Emiliano.
> > >             >> >
> > >             >> >
> > >             >> > 2013/12/4 Hui Cao <hcao () sourcefire com
> > >             <mailto:hcao () sourcefire com>>
> > >             >> >>
> > >             >> >> In sr/preprocids.h
> > >             >> >>
> > >             >> >> Best,
> > >             >> >> Hui.
> > >             >> >>
> > >             >> >> On 12/04/2013 02:36 PM, Emiliano Fausto wrote:
> > >             >> >>
> > >             >> >> Great,
> > >             >> >>
> > >             >> >> so, the pre-processors are "chained" by default,
> > >             and the order that
> > >             >> >> SNORT
> > >             >> >> follows to call them is set by the PRIORITY variable.
> > >             >> >>
> > >             >> >> Do you know where's defined this PRIORITY
> > >             variable? Because I saw that
> > >             >> >> the
> > >             >> >> frag3 is being registered with PRIORITY_NETWORK,
> > >             so I'd like to set the
> > >             >> >> priority of my own preprocessor as
> > >             (PRIORITY_NETWORK -1).
> > >             >> >>
> > >             >> >> Thanks in advance,
> > >             >> >> Emiliano
> > >             >> >>
> > >             >> >>
> > >             >> >> 2013/12/4 Hui Cao <hcao () sourcefire com
> > >             <mailto:hcao () sourcefire com>>
> > >             >> >>>
> > >             >> >>> sc means snort configuration. We use PRIORITY to
> > >             sort the processing.
> > >             >> >>> All
> > >             >> >>> processors enabled will be called and processed
> > >             based on priority. You
> > >             >> >>> have
> > >             >> >>> to rely on the code to figure out what exactly
> > >             snort does.
> > >             >> >>>
> > >             >> >>> The checking is correct. You will only process
> > >             rebuilt packets.
> > >             >> >>>
> > >             >> >>> Best,
> > >             >> >>> Hui.
> > >             >> >>> On 12/04/2013 02:19 PM, Emiliano Fausto wrote:
> > >             >> >>>
> > >             >> >>> Hello Hui,
> > >             >> >>>
> > >             >> >>> thanks a lot for your answer.
> > >             >> >>>
> > >             >> >>> Right now I have registered my preprocessor
> > >             (let's call it
> > >             >> >>> examplePreprocess as you said, because right now
> > >             I'm using the one
> > >             >> >>> provided
> > >             >> >>> with the DPX) with this line:
> > >             >> >>>
> > >             >> >>> _dpd.addPreproc(ExampleProcess,
> > >             PRIORITY_TRANSPORT, 10000,
> > >             >> >>> PROTO_BIT__TCP);
> > >             >> >>>
> > >             >> >>> So, the only change is to add previous to the
> > >             parameter
> > >             >> >>> ExampleProcess,
> > >             >> >>> the "sc". What does it mean? Do you know if
> > >             there's any documentation
> > >             >> >>> about
> > >             >> >>> this chaining preprocesses?
> > >             >> >>>
> > >             >> >>> So, checking the flags, should be:
> > >             >> >>>
> > >             >> >>> (SFSnortPacket*)tcppacket)->flags & FLAG_REBUILT_FRAG
> > >             >> >>>
> > >             >> >>> right?
> > >             >> >>>
> > >             >> >>> Thanks again!
> > >             >> >>> Emiliano.
> > >             >> >>>
> > >             >> >>>
> > >             >> >>>
> > >             >> >>> Then, I'll have to register my own preprocessor
> > >             where?
> > >             >> >>>
> > >             >> >>>
> > >             >> >>> 2013/12/4 Hui Cao <hcao () sourcefire com
> > >             <mailto:hcao () sourcefire com>>
> > >             >> >>>>
> > >             >> >>>> Yes, it is possible. You can register you
> > >             preprocessor like this:
> > >             >> >>>>
> > >             >> >>>> _dpd.addPreproc( sc, ExampleProcess,
> > >             PRIORITY_TRANSPORT, You_PP_ID,
> > >             >> >>>> PROTO_BIT__IP );
> > >             >> >>>>
> > >             >> >>>> Remember check the following flag in your
> > >             ExampleProcess:
> > >             >> >>>>
> > >             >> >>>> (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_FRAG
> > >             >> >>>>
> > >             >> >>>> Best,
> > >             >> >>>> Hui.
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>> On 12/04/2013 12:52 PM, Emiliano Fausto wrote:
> > >             >> >>>>
> > >             >> >>>> Hi everybody,
> > >             >> >>>>
> > >             >> >>>> I'm creating a new preprocessor which needs to
> > >             have the whole content
> > >             >> >>>> in
> > >             >> >>>> a packet which was fragmented.
> > >             >> >>>>
> > >             >> >>>> So I thought of using the frag3 preprocessor to
> > >             re-assembly the
> > >             >> >>>> packets,
> > >             >> >>>> and then, when this reassembly is done, sent it
> > >             to my own
> > >             >> >>>> preprocessor.
> > >             >> >>>>
> > >             >> >>>> Do you know if this is possible? May I have the
> > >             output of frag3 being
> > >             >> >>>> the input of my own preprocessor?
> > >             >> >>>>
> > >             >> >>>> Regards,
> > >             >> >>>> Emiliano.
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             ------------------------------------------------------------------------------
> > >             >> >>>> Sponsored by Intel(R) XDK
> > >             >> >>>> Develop, test and display web and hybrid apps
> > >             with a single code
> > >             >> >>>> base.
> > >             >> >>>> Download it for free now!
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>> _______________________________________________
> > >             >> >>>> Snort-devel mailing list
> > >             >> >>>> Snort-devel () lists sourceforge net
> > >             <mailto:Snort-devel () lists sourceforge net>
> > >             >> >>>>
> > >             https://lists.sourceforge.net/lists/listinfo/snort-devel
> > >             >> >>>> Archive:
> > >             >> >>>>
> > >             http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > >             >> >>>>
> > >             >> >>>> Please visit http://blog.snort.org for the
> > >             latest news about Snort!
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             ------------------------------------------------------------------------------
> > >             >> >>>> Sponsored by Intel(R) XDK
> > >             >> >>>> Develop, test and display web and hybrid apps
> > >             with a single code
> > >             >> >>>> base.
> > >             >> >>>> Download it for free now!
> > >             >> >>>>
> > >             >> >>>>
> > >             >> >>>>
> > >             http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
> > >             >> >>>> _______________________________________________
> > >             >> >>>> Snort-devel mailing list
> > >             >> >>>> Snort-devel () lists sourceforge net
> > >             <mailto:Snort-devel () lists sourceforge net>
> > >             >> >>>>
> > >             https://lists.sourceforge.net/lists/listinfo/snort-devel
> > >             >> >>>> Archive:
> > >             >> >>>>
> > >             http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > >             >> >>>>
> > >             >> >>>> Please visit http://blog.snort.org for the
> > >             latest news about Snort!
> > >             >> >>>
> > >             >> >>>
> > >             >> >>>
> > >             >> >>
> > >             >> >>
> > >             >> >
> > >             >
> > >             >

------------------------------------------------------------------------------
Sponsored by Intel(R) XDK 
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!