Re: File preprocessor fails to capture files

[Pablo Cantos Polaino <pcantos () redborder org>]

Sure it helps, Russ. As you said, this malfunctioning is not due to
ordinary network conditions, therefore I've replayed these tests over other
interface networks with successful results (without packet reordering), and
after updating the network interface drivers over the one who reordered the
packets, this is working fine too. So now I'm going to continue my tests on
Snort and I will let you know if this problem disappears.

They're live captures that have been got from a port span (tcpdump -v -i
eth4 -s 0 -w 18kfile.pcap). The only thing I had done is filtering by the
session which the file belonged to and exporting to another PCAP file. In
any case, I've done tests with full PCAP files and its filtered ones with
same results.

I'm also attaching the original live captures you wanted to see.

Best regards and thanks for your help!


Pablo Cantos
redborder.org / pcantos () redborder org

2015-05-18 4:32 GMT+02:00 Russ <rucombs () cisco com>:

> On 5/16/15 11:04 AM, Pablo Cantos Polaino wrote:
>
> Hi Russ,
>
>  Thanks for your feedback!
>
>  I'm attaching you separately two PCAP files (but if someone else wants
> to check out these PCAP files, please let me know and I will also share
> them with them), both of them with just one TCP session inside.
>
>  - 18kfile_not_captured.pcap: it contains TCP ACKed unseen segments and
> the file is not captured by Snort.
> - 18kfile_captured.pcap: it doesn't contain TCP ACKed unseen segments and
> the file is captured by Snort.
>
>  One of the tests I've deployed to check this behavior in IPS mode is the
> next (using just one machine with Snort):
>
>  - Bridge the interface eth2 with eth4.
> - Send the 18kfile_captured.pcap through the eth2 using the program
> 'tcpreplay'.
> - Use Snort in IPS mode listening from eth4.
> - Result: There were TCP Gaps.
>
>  I replayed the test but by using tcpdump instead of Snort listening from
> eth4. Where I found is that the packets arrived disordered. The PCAP file
> has just 36 packets, and some of the first ACKs arrived before its
> corresponding Datapackets. This is what I think it was causing the TCP Gaps
> in Snort. But, when I executed tcpreplay with the option -o (which sends a
> packet every time a push the return key on my keyboard), the packets did
> not arrive disordered, and this could be this way since Datapackets can be
> processed completely before an ACK is sent and processed (supposing an ACK
> requires less time to be processed than a Datapacket).
>
>  Since this seems a common malfunctioning at a lower level, I don't know
> if maybe Snort should take this into account and try to, after an ACK
> unseen segment arrives, retain the window and check the 5 or 10 next
> packets, just in case the corresponding Datapacket is coming later, and if
> no corresponding Datapacket is comming later, move on the window as Snort
> is doing currently. I don't think this could lead to flawed reassembly and
> evasions, but please tell me if I'm wrong.
>
>   I'm not clear on the pcaps you sent; are they cooked or live capture?
> I just want to see the live capture.
>
> The malfunctioning you speak of is not due to ordinary network
> conditions.  It is due to a broken capture.  This can happen in certain
> passive scenarios on an end point and is much less likely to happen if
> captured with a network tap.  However, it simply can't happen in a real
> network with a conformant TCP implementation because in no case is data
> supposed to be ACKed before it is actually received.
>
> Accepting previously ACKed data in general definitely allows more attacks
> and performance issues arise as well.  It may be possible instead to add a
> --enable-broken-pcap mode for passive operation only that ignores an ACK of
> unseen data but there are potential issues.  To avoid stalling, a followup
> ACK of the unseen data would need to be accepted because the packet was
> probably dropped.  Also, if the ACK comes with data, should that data be
> processed?  If it is truly a bad ACK it shouldn't.  And this will mess with
> TCP timestamps too.
>
> I don't consider the current operation to be broken so won't be filing a
> bug, but we will take a look at a patch if you want to work up something.
> However, you might consider a faster machine or higher resolution capture
> like pcap-ng to avoid the issue in the first place.
>
> Hope this helps.
> Russ
>
>
>  Best Regards,
>
>    Pablo Cantos
>  redborder.org / pcantos () redborder org
>
> 2015-05-15 15:17 GMT+02:00 Russ <rucombs () cisco com>:
>
> >  Pablo,
> >
> > That's a great detailed analysis.  The reordering of data segments is
> > part of what TCP does but that applies to each side of a flow separately.
> > It is not normal that an ACK, which travels in the opposite direction of
> > the data it is acknowledging, is sent before the data is received, which is
> > what the capture implies.  Although this reordering is actually just an
> > artifact of the capture, an ACK advances the window and Snort has to simply
> > note the gap and move on.  The alternative, in almost all cases, would lead
> > to flawed reassembly and evasions.
> >
> > You haven't mentioned selective ACK which could possibly be confusing
> > Snort.  Can you send a pcap of the misordered capture?  If you capture this
> > on a different (maybe higher speed) device, is the order correct?
> >
> > Thanks
> > Russ
> >
> > On 5/15/15 5:25 AM, Pablo Cantos Polaino wrote:
> >
> > Hi Hui,
> >
> >  I'm sorry for the delay, but I wanted to test some scenarios before
> > replying you.
> >
> >  Following your advice, I've disabled some optimizations in order to not
> > to get jumbo frames, which actually were present in my PCAP file. After
> > disabling the optimization, I was able to generate other PCAP files without
> > Jumbo frames.
> >
> >  This time I got several traffic network captures by using 3 small files
> > (8.1KB, 18KB and 63KB) and 'wget' and 'tcpdump' programs. I've tried to get
> > the captures as much simple as possible in order to avoid external
> > "noises", and to do that I've got 6 PCAP files for every one of these three
> > files, which gives me 18 PCAP files. I've also "cleaned" all these PCAP
> > files by removing the rest of the sessions in which the file isn't in.
> >
> >  After that, what I found on my first tests is that in all cases every
> > file was not always captured by Snort (IDS mode). When a file was not
> > captured, there were always 1 or 2 TCP Gaps and no TCP Discards and I could
> > see with Wireshark that there were at least one packet marked as "TCP ACKed
> > unseen segment" (see filter tcp.analysis.ack_lost_segment in Wireshark),
> > which is due to one ACK that arrives before the Datapacket acked does. On
> > the other hand, when the file was captured, there were neither TCP Gaps nor
> > TCP Discards and I could check with Wireshark that there were no packet
> > marked as "TCP ACKed unseen segment". Se examples below:
> >
> >  Regular PCAP file:
> >
> >  pkt 1: server | client | seq_100 | ack_050 | len 100
> > pkt 2: server | client | seq_200 | ack_050 | len 100
> > pkt 3: server | client | seq_300 | ack_050 | len 100
> > pkt 4: server | client | seq_400 | ack_050 | len 100
> > pkt 5: client | server | seq_050 | ack_200 | len 000 <- ack for pkt 1
> > pkt 6: client | server | seq_050 | ack_300 | len 000 <- ack for pkt 2
> > pkt 7: client | server | seq_050 | ack_400 | len 000 <- ack for pkt 3
> > pkt 8: client | server | seq_050 | ack_500 | len 000 <- ack for pkt 4
> >
> >  PCAP file with one TCP ACKed unseen segment:
> >
> >  pkt 1: server | client | seq_100 | ack_050 | len 100
> > pkt 2: server | client | seq_200 | ack_050 | len 100
> > pkt 3: server | client | seq_300 | ack_050 | len 100
> > pkt 4: client | server | seq_050 | ack_200 | len 000 <- ack for pkt 1
> >  pkt 5: client | server | seq_050 | ack_300 | len 000 <- ack for pkt 2
> > pkt 6: client | server | seq_050 | ack_400 | len 000 <- ack for pkt 3
> > pkt 7: client | server | seq_050 | ack_500 | len 000 <- ack for pkt 8
> > (TCP ACKed unseen segment)
> >  pkt 8: server | client | seq_400 | ack_050 | len 100
> >
> >  I've reordered some packets In PCAP files that cause TCP Gaps in Snort
> > by putting ACKs after their corresponding Datapacket (in the example above
> > it would consist in put the pkt 8 before pkt 7). This time, Snort didn't
> > see TCP Gaps when read these reordered PCAP files and was able to capture
> > the file inside.
> >
> >  This packet disruption is causing Snort to not work properly, since
> > it's finding "false" TCP Gaps, and I say "false" because actually these
> > Datapackets exists, but their corresponding ACKs are arriving before them
> > (just one position before is enough to discard the Datapacket).
> >
> >  To understand better how far Snort is able to take into account packet
> > disruptions, I've also tried to reorder a "consistent" PCAP file, with
> > neither TCP ACKed unseen segments nor TCP Gaps or Discard in Snort and
> > whose file inside is captured by Snort. Example of "consistent" or regular
> > PCAP file:
> >
> >  pkt 1: server | client | seq_100 | ack_050 | len 100
> >  pkt 2: server | client | seq_200 | ack_050 | len 100
> >  pkt 3: server | client | seq_300 | ack_050 | len 100
> >  pkt 4: server | client | seq_400 | ack_050 | len 100
> >  pkt 5: client | server | seq_050 | ack_200 | len 000 <- ack for pkt 1
> > pkt 6: client | server | seq_050 | ack_300 | len 000 <- ack for pkt 2
> > pkt 7: client | server | seq_050 | ack_400 | len 000 <- ack for pkt 3
> > pkt 8: client | server | seq_050 | ack_500 | len 000 <- ack for pkt 4
> >
> >  What I've done with this PCAP file is to alter the position of two
> > Datapacket, but without causing TCP ACKed unseen segments. See example
> > below in which pkt 2 is moved before pkt 3:
> >
> >  pkt 1: server | client | seq_100 | ack_050 | len 100
> > pkt 2: server | client | seq_300 | ack_050 | len 100
> >  pkt 3: server | client | seq_200 | ack_050 | len 100
> > pkt 4: server | client | seq_400 | ack_050 | len 100
> >  pkt 5: client | server | seq_050 | ack_200 | len 000 <- ack for pkt 1
> > pkt 6: client | server | seq_050 | ack_300 | len 000 <- ack for pkt 2
> > pkt 7: client | server | seq_050 | ack_400 | len 000 <- ack for pkt 3
> > pkt 8: client | server | seq_050 | ack_500 | len 000 <- ack for pkt 4
> >
> >  In this case, Snort is able to detect this packet re-ordering and it
> > does not affect to the final result, since there's neither TCP Gaps nor
> > Discards and the file inside the TCP session is captured by Snort.
> >
> >  I've replayed all these tests in IPS mode with the same result, by
> > using 'tcpreplay' program and sending these PCAP files through a network
> > interface.
> >
> >  Now, my question is whether there is a way that Snort could manage the
> > "false" TCP ACKed unseen segments due to packet re-ordering, in the same
> > way it manages the Datapacket re-ordering. Since the packet re-ordering
> > seems to be a common issue in networks.
> >
> >  I've reviewed the Stream5 and Frag3 options looking for a possible
> > configuration to solve this, but I haven't found anything related to this.
> >
> >  If any Snort output, PCAP file or whatever were useful to you, please
> > don't hesitate to ask for it.
> >
> >  Best Regards and Thank you very much for your help,
> >
> >    Pablo Cantos
> >  redborder.org / pcantos () redborder org
> >
> > 2015-05-09 2:51 GMT+02:00 Hui Cao (huica) <huica () cisco com>:
> >
> > >  Hi Pablo,
> > >
> > >  I am pretty sure that the Jumbo frame is the issue for PCAP. They are
> > > even bigger than 60000, which is not normal.
> > >
> > >  For the interface one, there are lots of discards. Not all traffic are
> > > processed by snort, therefore there are lots gaps in the TCP stream. You
> > > can try inline mode, instead of passive. Either there is some configuration
> > > issue for interface, or the speed of passive interface might be too high to
> > > be processed by your CPU.
> > >
> > >  Best,
> > > Hui.
> > >
> > >   From: Pablo Cantos Polaino <pcantos () redborder org>
> > > Date: Friday, May 8, 2015 at 5:59 PM
> > >
> > > To: Hui Cao <huica () cisco com>
> > > Cc: "snort-users () lists sourceforge net" <
> > > snort-users () lists sourceforge net>
> > > Subject: Re: [Snort-users] File preprocessor fails to capture files
> > >
> > >   Hi Hui,
> > >
> > >  I've replaced config paf_max: 16000 by 60000 as you propose.
> > >
> > >  File type was not identified because I had disabled type_id option in
> > > preprocessor file_inspect. I've replayed the tests with paf_max = 60000 and
> > > both type_id enabled and disabled. In both cases the capture files are the
> > > same (number and size) when sniffering from an interface and reading from a
> > > PCAP file. So I'm pasting below the exit stats when type_id is enabled and
> > > paf_max = 60000:
> > >
> > >  Exit stats when reading the PCAP file and type_id enabled:
> > >
> > >
> > > ===============================================================================
> > >  Run time for packet processing was 3.978146 seconds
> > > Snort processed 3326 packets.
> > > Snort ran for 0 days 0 hours 0 minutes 3 seconds
> > >    Pkts/sec:         1108
> > >
> > > ===============================================================================
> > > Memory usage summary:
> > >   Total non-mmapped bytes (arena):       10190848
> > >   Bytes in mapped regions (hblkhd):      122081280
> > >   Total allocated space (uordblks):      8072896
> > >   Total free space (fordblks):           2117952
> > >   Topmost releasable block (keepcost):   133008
> > >
> > > ===============================================================================
> > > Packet I/O Totals:
> > >    Received:         3326
> > >    Analyzed:         3326 (100.000%)
> > >     Dropped:            0 (  0.000%)
> > >    Filtered:            0 (  0.000%)
> > > Outstanding:            0 (  0.000%)
> > >    Injected:            0
> > >
> > > ===============================================================================
> > > Breakdown by protocol (includes rebuilt packets):
> > >         Eth:         3333 (100.000%)
> > >        VLAN:            0 (  0.000%)
> > >         IP4:         3333 (100.000%)
> > >        Frag:            0 (  0.000%)
> > >        ICMP:            0 (  0.000%)
> > >         UDP:           40 (  1.200%)
> > >         TCP:         3293 ( 98.800%)
> > >         IP6:            0 (  0.000%)
> > >     IP6 Ext:            0 (  0.000%)
> > >    IP6 Opts:            0 (  0.000%)
> > >       Frag6:            0 (  0.000%)
> > >       ICMP6:            0 (  0.000%)
> > >        UDP6:            0 (  0.000%)
> > >        TCP6:            0 (  0.000%)
> > >      Teredo:            0 (  0.000%)
> > >     ICMP-IP:            0 (  0.000%)
> > >     IP4/IP4:            0 (  0.000%)
> > >     IP4/IP6:            0 (  0.000%)
> > >     IP6/IP4:            0 (  0.000%)
> > >     IP6/IP6:            0 (  0.000%)
> > >         GRE:            0 (  0.000%)
> > >     GRE Eth:            0 (  0.000%)
> > >    GRE VLAN:            0 (  0.000%)
> > >     GRE IP4:            0 (  0.000%)
> > >     GRE IP6:            0 (  0.000%)
> > > GRE IP6 Ext:            0 (  0.000%)
> > >    GRE PPTP:            0 (  0.000%)
> > >     GRE ARP:            0 (  0.000%)
> > >     GRE IPX:            0 (  0.000%)
> > >    GRE Loop:            0 (  0.000%)
> > >        MPLS:            0 (  0.000%)
> > >         ARP:            0 (  0.000%)
> > >         IPX:            0 (  0.000%)
> > >    Eth Loop:            0 (  0.000%)
> > >    Eth Disc:            0 (  0.000%)
> > >    IP4 Disc:            0 (  0.000%)
> > >    IP6 Disc:            0 (  0.000%)
> > >    TCP Disc:            0 (  0.000%)
> > >    UDP Disc:            0 (  0.000%)
> > >   ICMP Disc:            0 (  0.000%)
> > > All Discard:            0 (  0.000%)
> > >       Other:            0 (  0.000%)
> > > Bad Chk Sum:            0 (  0.000%)
> > >     Bad TTL:            0 (  0.000%)
> > >      S5 G 1:            3 (  0.090%)
> > >      S5 G 2:            4 (  0.120%)
> > >       Total:         3333
> > >
> > > ===============================================================================
> > > Action Stats:
> > >      Alerts:            0 (  0.000%)
> > >      Logged:            0 (  0.000%)
> > >      Passed:            0 (  0.000%)
> > > Limits:
> > >       Match:            0
> > >       Queue:            0
> > >         Log:            0
> > >       Event:            0
> > >       Alert:            0
> > > Verdicts:
> > >       Allow:         3326 (100.000%)
> > >       Block:            0 (  0.000%)
> > >     Replace:            0 (  0.000%)
> > >   Whitelist:            0 (  0.000%)
> > >   Blacklist:            0 (  0.000%)
> > >      Ignore:            0 (  0.000%)
> > >
> > > ===============================================================================
> > > Frag3 statistics:
> > >         Total Fragments: 0
> > >       Frags Reassembled: 0
> > >                Discards: 0
> > >           Memory Faults: 0
> > >                Timeouts: 0
> > >                Overlaps: 0
> > >               Anomalies: 0
> > >                  Alerts: 0
> > >                   Drops: 0
> > >      FragTrackers Added: 0
> > >     FragTrackers Dumped: 0
> > > FragTrackers Auto Freed: 0
> > >     Frag Nodes Inserted: 0
> > >      Frag Nodes Deleted: 0
> > >
> > > ===============================================================================
> > >
> > > ===============================================================================
> > > Stream statistics:
> > >             Total sessions: 24
> > >               TCP sessions: 14
> > >               UDP sessions: 10
> > >              ICMP sessions: 0
> > >                IP sessions: 0
> > >                 TCP Prunes: 0
> > >                 UDP Prunes: 0
> > >                ICMP Prunes: 0
> > >                  IP Prunes: 0
> > > TCP StreamTrackers Created: 14
> > > TCP StreamTrackers Deleted: 14
> > >               TCP Timeouts: 0
> > >               TCP Overlaps: 0
> > >        TCP Segments Queued: 2394
> > >      TCP Segments Released: 2394
> > >        TCP Rebuilt Packets: 793
> > >          TCP Segments Used: 2393
> > >               TCP Discards: 0
> > >                   TCP Gaps: 0
> > >       UDP Sessions Created: 10
> > >       UDP Sessions Deleted: 10
> > >               UDP Timeouts: 0
> > >               UDP Discards: 0
> > >                     Events: 1
> > >            Internal Events: 0
> > >            TCP Port Filter
> > >                   Filtered: 0
> > >                  Inspected: 0
> > >                    Tracked: 3286
> > >            UDP Port Filter
> > >                   Filtered: 0
> > >                  Inspected: 0
> > >                    Tracked: 10
> > >
> > > ===============================================================================
> > > HTTP Inspect - encodings (Note: stream-reassembled packets included):
> > >     POST methods:                         0
> > >     GET methods:                          10
> > >     HTTP Request Headers extracted:       10
> > >     HTTP Request Cookies extracted:       0
> > >     Post parameters extracted:            0
> > >     HTTP response Headers extracted:      10
> > >     HTTP Response Cookies extracted:      0
> > >     Unicode:                              0
> > >     Double unicode:                       0
> > >     Non-ASCII representable:              0
> > >     Directory traversals:                 0
> > >     Extra slashes ("//"):                 0
> > >     Self-referencing paths ("./"):        0
> > >     HTTP Response Gzip packets extracted: 0
> > >     Gzip Compressed Data Processed:       n/a
> > >     Gzip Decompressed Data Processed:     n/a
> > >     Total packets processed:              2433
> > >
> > > ===============================================================================
> > > SMTP Preprocessor Statistics
> > >   Total sessions                                    : 0
> > >   Max concurrent sessions                           : 0
> > >
> > > ===============================================================================
> > > dcerpc2 Preprocessor Statistics
> > >   Total sessions: 0
> > >
> > > ===============================================================================
> > >
> > > ===============================================================================
> > > SIP Preprocessor Statistics
> > >   Total sessions: 0
> > >
> > > ===============================================================================
> > > File Preprocessor Statistics
> > >   Total file type callbacks:            7
> > >   Total file signature callbacks:       7
> > >   Total files would saved to disk:      7
> > >   Total files saved to disk:            7
> > >   Total file data saved to disk:        47466737  bytes
> > >   Total files duplicated:               0
> > >   Total files reserving failed:         0
> > >   Total file capture min:               0
> > >   Total file capture max:               0
> > >   Total file capture memcap:            0
> > >   Total files reading failed:           0
> > >   Total file agent memcap failures:     0
> > >   Total files sent:                     0
> > >   Total file data sent:                 0
> > >   Total file transfer failures:         0
> > >
> > > ===============================================================================
> > > File type stats:
> > >          Type              Download   (Bytes)      Upload     (Bytes)
> > >           GZ( 33)          2          6848054      0          0
> > >          MP3( 64)          2          37257592     0          0
> > >         JPEG( 70)          2          3360645      0          0
> > >           BMP(148)          1          446          0          0
> > >
> > >             Total          7          47466737     0          0
> > >
> > >
> > >  File signature stats:
> > >          Type              Download   Upload
> > >           GZ( 33)          2          0
> > >          MP3( 64)          2          0
> > >          PNG( 69)          1          0
> > >         JPEG( 70)          2          0
> > >             Total          7          0
> > >
> > >  File type verdicts:
> > >         UNKNOWN:           7
> > >             LOG:           0
> > >            STOP:           0
> > >           BLOCK:           0
> > >          REJECT:           0
> > >         PENDING:           0
> > >    STOP CAPTURE:           0
> > >           Total:           7
> > >
> > >  File signature verdicts:
> > >         UNKNOWN:           7
> > >             LOG:           0
> > >            STOP:           0
> > >           BLOCK:           0
> > >          REJECT:           0
> > >         PENDING:           0
> > >    STOP CAPTURE:           0
> > >           Total:           7
> > >
> > >  Total files processed:             10
> > > Total files data processed:        47473897  bytes
> > > Total files buffered:              7
> > > Total files released:              7
> > > Total files freed:                 0
> > > Total files captured:              7
> > > Total files within one packet:     1
> > > Total buffers allocated:           1452
> > > Total buffers freed:               0
> > > Total buffers released:            1452
> > > Maximum file buffers used:         759
> > > Total buffers free errors:         0
> > > Total buffers release errors:      0
> > > Total memcap failures:             0
> > > Total memcap failures at reserve:  0
> > > Total reserve failures:            0
> > > Total file capture size min:       0
> > > Total file capture size max:       0
> > > Total capture max before reserve:  0
> > > Total file signature max:          0
> > > Maximum buffers can allocate:      3196
> > > Number of buffers in use:          0
> > > Number of buffers in free list:    1744
> > > Number of buffers in release list: 1452
> > >
> > > ===============================================================================
> > > Snort exiting
> > >
> > >  This time the captured files have changed:
> > >
> > >  # ls -lS
> > >  -rw------- 1 root root 24211979 May  8 21:20
> > > 8452B621DC334D1FD44470A80540CBEF2F6869AF851B9E8C684EF9402016F692
> > > -rw------- 1 root root 13045613 May  8 21:20
> > > 5CF142947C2957EE648457A91B69FB82F088F31205030F9A77B2AD827228C6E9
> > > -rw------- 1 root root  6352738 May  8 21:20
> > > DB57C532919D9ABABAC127F29DBDC05ED832394880E46CAD81A5DDE713CCB4BE
> > > -rw------- 1 root root  2936119 May  8 21:20
> > > B4127F43A3F455523B81179CC11AA4F28FC27F4C041D20E28AA08A32D85CB757
> > > -rw------- 1 root root   495316 May  8 21:20
> > > A294AA3D01CD8902BF842D320E7F2C043AF9EAD95D0E7198C3B71A0DBC9D253C
> > > -rw------- 1 root root   424526 May  8 21:20
> > > 8863DB1EC4B02D5BCC1FB4BD03D220F7458136342CDD47CE507A5B886C6BB56C
> > > -rw------- 1 root root      446 May  8 21:20
> > > 8D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1
> > >
> > >  But they are still different from the original ones:
> > >
> > >  # ls -lS
> > >  -rw-r--r-- 1 root root 1044381696 Feb 18 20:12
> > > ubuntu-14.04.2-desktop-amd64.iso
> > > -rw-r--r-- 1 root root  375187792 May  8 21:07 VMware-viclient.exe
> > > -rw-r--r-- 1 root root  101688487 Jul 10  2014 oversize_pdf_test_0.pdf
> > > -rw-r--r-- 1 root root   14955972 May  8 21:07 MakeUp.mov
> > > -rw-r--r-- 1 root root    6094376 May  8 21:07 video1.avi
> > > -rw-r--r-- 1 root root    2187725 May  8 21:07 Fighter.mpg
> > > -rw-r--r-- 1 root root        446 Mar 22  2013 tux-sw.bmp
> > >
> > >
> > > ##########################################################################
> > >
> > > ##########################################################################
> > >
> > >  Exit stats when listening from interface and type_id enabled:
> > >
> > >
> > > ===============================================================================
> > > Run time for packet processing was 108.388974 seconds
> > > Snort processed 256250 packets.
> > > Snort ran for 0 days 0 hours 1 minutes 48 seconds
> > >    Pkts/min:       256250
> > >    Pkts/sec:         2372
> > >
> > > ===============================================================================
> > > Memory usage summary:
> > >   Total non-mmapped bytes (arena):       10100736
> > >   Bytes in mapped regions (hblkhd):      122081280
> > >   Total allocated space (uordblks):      8073952
> > >   Total free space (fordblks):           2026784
> > >   Topmost releasable block (keepcost):   108544
> > >
> > > ===============================================================================
> > > Packet I/O Totals:
> > >    Received:       256250
> > >    Analyzed:       256250 (100.000%)
> > >     Dropped:            0 (  0.000%)
> > >    Filtered:            0 (  0.000%)
> > > Outstanding:            0 (  0.000%)
> > >    Injected:            0
> > >
> > > ===============================================================================
> > > Breakdown by protocol (includes rebuilt packets):
> > >         Eth:       256255 (100.000%)
> > >        VLAN:            0 (  0.000%)
> > >         IP4:       256130 ( 99.951%)
> > >        Frag:            0 (  0.000%)
> > >        ICMP:            0 (  0.000%)
> > >         UDP:           24 (  0.009%)
> > >         TCP:       132229 ( 51.601%)
> > >         IP6:            0 (  0.000%)
> > >     IP6 Ext:            0 (  0.000%)
> > >    IP6 Opts:            0 (  0.000%)
> > >       Frag6:            0 (  0.000%)
> > >       ICMP6:            0 (  0.000%)
> > >        UDP6:            0 (  0.000%)
> > >        TCP6:            0 (  0.000%)
> > >      Teredo:            0 (  0.000%)
> > >     ICMP-IP:            0 (  0.000%)
> > >     IP4/IP4:            0 (  0.000%)
> > >     IP4/IP6:            0 (  0.000%)
> > >     IP6/IP4:            0 (  0.000%)
> > >     IP6/IP6:            0 (  0.000%)
> > >         GRE:            0 (  0.000%)
> > >     GRE Eth:            0 (  0.000%)
> > >    GRE VLAN:            0 (  0.000%)
> > >     GRE IP4:            0 (  0.000%)
> > >     GRE IP6:            0 (  0.000%)
> > > GRE IP6 Ext:            0 (  0.000%)
> > >    GRE PPTP:            0 (  0.000%)
> > >     GRE ARP:            0 (  0.000%)
> > >     GRE IPX:            0 (  0.000%)
> > >    GRE Loop:            0 (  0.000%)
> > >        MPLS:            0 (  0.000%)
> > >         ARP:          125 (  0.049%)
> > >         IPX:            0 (  0.000%)
> > >    Eth Loop:            0 (  0.000%)
> > >    Eth Disc:            0 (  0.000%)
> > >    IP4 Disc:       123866 ( 48.337%)
> > >    IP6 Disc:            0 (  0.000%)
> > >    TCP Disc:            0 (  0.000%)
> > >    UDP Disc:            0 (  0.000%)
> > >   ICMP Disc:            0 (  0.000%)
> > > All Discard:       123866 ( 48.337%)
> > >       Other:           11 (  0.004%)
> > > Bad Chk Sum:          362 (  0.141%)
> > >     Bad TTL:            0 (  0.000%)
> > >      S5 G 1:            2 (  0.001%)
> > >      S5 G 2:            3 (  0.001%)
> > >       Total:       256255
> > >
> > > ===============================================================================
> > > Action Stats:
> > >      Alerts:            0 (  0.000%)
> > >      Logged:            0 (  0.000%)
> > >      Passed:            0 (  0.000%)
> > > Limits:
> > >       Match:            0
> > >       Queue:            0
> > >         Log:            0
> > >       Event:            0
> > >       Alert:            0
> > > Verdicts:
> > >       Allow:       228770 ( 89.276%)
> > >       Block:            0 (  0.000%)
> > >     Replace:            0 (  0.000%)
> > >   Whitelist:        27480 ( 10.724%)
> > >   Blacklist:            0 (  0.000%)
> > >      Ignore:            0 (  0.000%)
> > >
> > > ===============================================================================
> > > Frag3 statistics:
> > >         Total Fragments: 0
> > >       Frags Reassembled: 0
> > >                Discards: 0
> > >           Memory Faults: 0
> > >                Timeouts: 0
> > >                Overlaps: 0
> > >               Anomalies: 0
> > >                  Alerts: 0
> > >                   Drops: 0
> > >      FragTrackers Added: 0
> > >     FragTrackers Dumped: 0
> > > FragTrackers Auto Freed: 0
> > >     Frag Nodes Inserted: 0
> > >      Frag Nodes Deleted: 0
> > >
> > > ===============================================================================
> > >
> > > ===============================================================================
> > > Stream statistics:
> > >             Total sessions: 20
> > >               TCP sessions: 14
> > >               UDP sessions: 6
> > >              ICMP sessions: 0
> > >                IP sessions: 0
> > >                 TCP Prunes: 0
> > >                 UDP Prunes: 0
> > >                ICMP Prunes: 0
> > >                  IP Prunes: 0
> > > TCP StreamTrackers Created: 14
> > > TCP StreamTrackers Deleted: 14
> > >               TCP Timeouts: 0
> > >               TCP Overlaps: 0
> > >        TCP Segments Queued: 6930
> > >      TCP Segments Released: 6930
> > >        TCP Rebuilt Packets: 6331
> > >          TCP Segments Used: 6903
> > >               TCP Discards: 7
> > >                   TCP Gaps: 6570
> > >       UDP Sessions Created: 6
> > >       UDP Sessions Deleted: 6
> > >               UDP Timeouts: 0
> > >               UDP Discards: 0
> > >                     Events: 16
> > >            Internal Events: 0
> > >            TCP Port Filter
> > >                   Filtered: 0
> > >                  Inspected: 0
> > >                    Tracked: 131874
> > >            UDP Port Filter
> > >                   Filtered: 0
> > >                  Inspected: 0
> > >                    Tracked: 6
> > >
> > > ===============================================================================
> > > HTTP Inspect - encodings (Note: stream-reassembled packets included):
> > >     POST methods:                         0
> > >     GET methods:                          0
> > >     HTTP Request Headers extracted:       0
> > >     HTTP Request Cookies extracted:       0
> > >     Post parameters extracted:            0
> > >     HTTP response Headers extracted:      2
> > >     HTTP Response Cookies extracted:      0
> > >     Unicode:                              0
> > >     Double unicode:                       0
> > >     Non-ASCII representable:              0
> > >     Directory traversals:                 0
> > >     Extra slashes ("//"):                 0
> > >     Self-referencing paths ("./"):        0
> > >     HTTP Response Gzip packets extracted: 0
> > >     Gzip Compressed Data Processed:       n/a
> > >     Gzip Decompressed Data Processed:     n/a
> > >     Total packets processed:              13165
> > >
> > > ===============================================================================
> > > SMTP Preprocessor Statistics
> > >   Total sessions                                    : 0
> > >   Max concurrent sessions                           : 0
> > >
> > > ===============================================================================
> > > dcerpc2 Preprocessor Statistics
> > >   Total sessions: 0
> > >
> > > ===============================================================================
> > > SSL Preprocessor:
> > >    SSL packets decoded: 68
> > >           Client Hello: 0
> > >           Server Hello: 2
> > >            Certificate: 2
> > >            Server Done: 3
> > >    Client Key Exchange: 0
> > >    Server Key Exchange: 0
> > >          Change Cipher: 2
> > >               Finished: 0
> > >     Client Application: 0
> > >     Server Application: 1
> > >                  Alert: 0
> > >   Unrecognized records: 64
> > >   Completed handshakes: 0
> > >         Bad handshakes: 0
> > >       Sessions ignored: 1
> > >     Detection disabled: 1
> > >
> > > ===============================================================================
> > > SIP Preprocessor Statistics
> > >   Total sessions: 0
> > >
> > > ===============================================================================
> > > File Preprocessor Statistics
> > >   Total file type callbacks:            2
> > >   Total file signature callbacks:       1
> > >   Total files would saved to disk:      1
> > >   Total files saved to disk:            1
> > >   Total file data saved to disk:        446       bytes
> > >   Total files duplicated:               0
> > >   Total files reserving failed:         0
> > >   Total file capture min:               0
> > >   Total file capture max:               0
> > >   Total file capture memcap:            0
> > >   Total files reading failed:           0
> > >   Total file agent memcap failures:     0
> > >   Total files sent:                     0
> > >   Total file data sent:                 0
> > >   Total file transfer failures:         0
> > >
> > > ===============================================================================
> > > File type stats:
> > >          Type              Download   (Bytes)      Upload     (Bytes)
> > >          BMP(148)          1          446          0          0
> > >          PDF(288)          1          3057259      0          0
> > >             Total          2          3057705      0          0
> > >
> > >  File signature stats:
> > >          Type              Download   Upload
> > >          BMP(148)          1          0
> > >             Total          1          0
> > >
> > >  File type verdicts:
> > >         UNKNOWN:           2
> > >             LOG:           0
> > >            STOP:           0
> > >           BLOCK:           0
> > >          REJECT:           0
> > >         PENDING:           0
> > >    STOP CAPTURE:           0
> > >           Total:           2
> > >
> > >  File signature verdicts:
> > >         UNKNOWN:           1
> > >             LOG:           0
> > >            STOP:           0
> > >           BLOCK:           0
> > >          REJECT:           0
> > >         PENDING:           0
> > >    STOP CAPTURE:           0
> > >           Total:           1
> > >
> > >  Total files processed:             2
> > > Total files data processed:        3057705   bytes
> > > Total files buffered:              2
> > > Total files released:              1
> > > Total files freed:                 1
> > > Total files captured:              1
> > > Total files within one packet:     1
> > > Total buffers allocated:           95
> > > Total buffers freed:               94
> > > Total buffers released:            1
> > > Maximum file buffers used:         94
> > > Total buffers free errors:         0
> > > Total buffers release errors:      0
> > > Total memcap failures:             0
> > > Total memcap failures at reserve:  0
> > > Total reserve failures:            0
> > > Total file capture size min:       0
> > > Total file capture size max:       0
> > > Total capture max before reserve:  0
> > > Total file signature max:          0
> > > Maximum buffers can allocate:      3196
> > > Number of buffers in use:          0
> > > Number of buffers in free list:    3195
> > > Number of buffers in release list: 1
> > >
> > > ===============================================================================
> > > Snort exiting
> > >
> > >  This time the captured files haven't changed:
> > >
> > >  # ls -lS
> > > -rw------- 1 root root 446 May  8 21:33
> > > 8D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1
> > >
> > >  Best Regards,
> > >
> > >
> > >    Pablo Cantos
> > >  redborder.org / pcantos () redborder org
> > >
> > > 2015-05-08 21:49 GMT+02:00 Hui Cao (huica) <huica () cisco com>:
> > >
> > > >  Hi Pablo,
> > > >
> > > >  When listening from interfaces, you have lots of discards. Because
> > > > file processing relies on data that are reassembled correctly, it won’t be
> > > > called for those sessions that miss file data.
> > > >
> > > >  In the case of PCAP, no sure why file type is not identified. It is
> > > > interesting to see 47M file data for only 3326 packets. That is 24K per
> > > > packet. I guess in this case, it will always hit PAF_MAX for each packet
> > > > which might set each packet as single PDU(file). Can you try this setting?
> > > >
> > > >  config paf_max: 60000
> > > >
> > > >  Best,
> > > > Hui.
> > > >
> > > >   From: Pablo Cantos Polaino <pcantos () redborder org>
> > > > Date: Friday, May 8, 2015 at 3:29 PM
> > > > To: Hui Cao <huica () cisco com>
> > > > Cc: "snort-users () lists sourceforge net" <
> > > > snort-users () lists sourceforge net>
> > > > Subject: Re: [Snort-users] File preprocessor fails to capture files
> > > >
> > > >     IP4 Disc:       122145 ( 49.331%)
> > > >    IP6 Disc:            0 (  0.000%)
> > > >    TCP Disc:            0 (  0.000%)
> > > >    UDP Disc:            0 (  0.000%)
> > > >   ICMP Disc:            0 (  0.000%)
> > > > All Discard:       122145 ( 49.331%)
> > > >
> > > >            TCP Segments Used: 6919
> > > >               TCP Discards: 48
> > > >                   TCP Gaps: 6459
> >
> >
> > ------------------------------------------------------------------------------
> > One dashboard for servers and applications across Physical-Virtual-Cloud
> > Widest out-of-the-box monitoring support with 50+ applications
> > Performance metrics, stats and reports that give you Actionable Insights
> > Deep dive visibility with transaction tracing using APM 
> > Insight.http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> >
> >
> >
> > _______________________________________________
> > Snort-users mailing listSnort-users () lists sourceforge net
> > Go to this URL to change user options or unsubscribe:https://lists.sourceforge.net/lists/listinfo/snort-users
> > Snort-users list archive:http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
> >
> > Please visit http://blog.snort.org to stay current on all the latest Snort news!
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!