Snort mailing list archives

Re: Getting alerts for every file Snort detects and File Services preprocessor

From: Pablo Cantos Polaino <pcantos () redborder org>
Date: Tue, 31 Mar 2015 13:19:29 +0200

Hi Victor,

We already thought about using flowbits, but (and please correct me if I'm
wrong) we didn't considered it effective since once a bit is set after
detecting a file, this rule will never trigger any alert. I've launched
some tests following your proposals and the numbers I got are not much
coherent. For these tests I've used a capture with 107 GIF files inside.
The file preprocessor is able to detect these 107 files and below you could
find the number of alerts I get using the following rules:

Rules:
alert (msg: "GIF(62) file"; gid:146; sid:62;)
alert (msg: "GIF(63) file"; gid:146; sid:63;)
Number of alerts: 107

Rule:
alert tcp any any -> any any (msg: "GIF file"; file_type:GIF; sid:3000011;
rev:1;)
Number of alerts: 111 (4 alerts more than the number of files detected.
Possibly due to one or more than one file takes up one or more packet)

Rule:
alert tcp any any -> any any (msg: "GIF file"; flow:to_client,established;
file_type:GIF; sid:3000013; rev:1;)
Number of alerts: 24 (much less alerts than the number of files detected.
Possibly due to the direction flow, but as you can see below, adding 24 and
25 we don't get the total number of files)

Rule:
alert tcp any any -> any any (msg: "GIF file"; flow:to_server,established;
file_type:GIF; sid:3000014; rev:1;)
Number of alerts: 25

Rule:
alert tcp any any -> any any (msg: "GIF file flowbits";
flowbits:isnotset,gif; file_type:GIF; flowbits:set,gif; sid:4000011; rev:1;)
Number of alerts: 2

Rule:
alert tcp any any -> any any (msg: "GIF file flowbits";
flowbits:isnotset,gif; flow:to_client,established; file_type:GIF;
flowbits:set,gif; sid:4000013; rev:1;)
Number of alerts: 20

Rule:
alert tcp any any -> any any (msg: "GIF file flowbits";
flowbits:isnotset,gif; flow:to_server,established; file_type:GIF;
flowbits:set,gif; sid:4000014; rev:1;)
Number of alerts: 20

As you can see, just using the two first rules which employ directly the
File preprocessor (rules with gid:146) gives us just one alert per file,
and this is what we are interested in. This way we can be sure that,
through the alerts, we are getting information from every file detected in
File preprocessor. The only thing we have to put care into is configuring
properly the preprocessor like this below:

preprocessor file_inspect: type_id

and await the related bug could be fixed.

In a previous mail you said "The "file_identify" preprocessor is designed
to work more as a "reputation" based system. I think you will get better
millage by using the new "file_type" rule keyword in a plain-old snort
rule". Our conclusion is quite opposite to your recommendation, whose
grounds we didn't understand, I mean, why is preferable using the file_type
keyword in a plain-old snort rule instead of using a rule with gid:146?

I hope I explained my self.

Best Regards,

Pablo Cantos
redborder.org / pcantos () redborder org

2015-03-30 19:22 GMT+02:00 Victor Roemer <viroemer () cisco com>:

The solution too the "too many alerts" can be resolved with the
application of "flowbits".

For example:

file type:PDF; content:|25 50 44 46 2d|; offset:0; id:1;
alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf;
file_type:PDF; flowbits:set,pdf; sid:1000000;)

However, it is also recommended that you add a "content" option too the
rule so that you may take advantage of the fast pattern matcher.
This may seem odd, but it will be faster. In the next rule, the content
being matched "mirrors" the content in the "file" rule above exactly.

alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf;
content:"%PDF-"; file_type:PDF; flowbits:set,pdf; sid:1000001;)


~Victor


On 03/30/15 5:53, Pablo Cantos Polaino wrote:

Hi Victor,

First of all, thanks for your reply!

Regarding your proposal below:

The "file_identify" preprocessor is designed to work more as a
"reputation"

based system. I think you will get better millage by using the new
"file_type" rule keyword in a plain-old snort rule.
Something like this:
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded";
flow:to_client,established; file_type:GIF; sid:1000000;)
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded";
flow:to_client,established; file_type:PNG; sid:1000001;)


We already tried that and we got a lot of alerts. The sizer the file is
the
more alerts are fired, and this is not the behavior we expected since we
would overpopulate the disk with repeated events. For instance, after
downloading a BZ file with size of 7,5MB, we've got 855 alerts from one
just rule:

alert tcp any any -> any any (msg: "BZ file"; flow:to_client,established;
file_type:BZ; sid:3000033;)

And we get just one alert if we use the rule below and the following
configuration:

Conf (just type_id, neither signature nor capture)

preprocessor file_inspect: type_id

Rule:

alert (msg: "BZ file"; gid:146; sid:32;)

This is why we discarded some weeks ago the way of writing the rule as you
proposed now. If it were a way to get just one alert per file it would be
great. Meanwhile, we await the bug can be fixed.

Best Regards,

Pablo Cantos
redborder.org / pcantos () redborder org

2015-03-27 22:46 GMT+01:00 Jaime Nebrera <jnebrera () redborder org>:

 Hi Víctor,


I'm not that technical ;) Pablo will coment on your ideas, I just tried
to
give you a deeper context of what Pablo (as part of my team) was doing
but
for sure, they are welcomed. I let them continue
El 27/03/2015 21:25, "Victor Roemer" <viroemer () cisco com> escribió:

 Jamie,


That's neat, I'll keep an eye on it. :)

Reviewing over how the "file_identify" works currently, when doing
capture or signature
its not possible to also get the file type event via the preprocessor.
I'll open an bug for this.

In order to get the file type events with these further settings, more
sophisticated tracking
needs to be in-place in the file_identify preproc; the tricky bits being
that the file service code
is generating the events in question; however once a verdict is passed
to
file service- an
action will be taken at that time.

To clarify the verdicts -
As I understand it, passing FILE_VERDICT_UNKNOWN is how processing
continues further.
The FILE_VERDICT_PENDING when passed to file service api, should cause
the packet to be
dropped, in-order to give the interfacing code time to retrieve a true
verdict when the re-transmitted
packet is seen.

On 03/27/15 16:04, Jaime Nebrera wrote:

 Hi Víctor,


Pablo's work is geared towards a more ambitious goal:

1) Intercept the file and send it to a S3 based storage platform for
further analysis or whatever

2) Provide as much context about such interception as possible, like
from
were to whom, URL, email, etc

This we hope to open source and make it public as soon as it is usable
in
our repository www.github.com/redborder
El 27/03/2015 20:50, "Victor Roemer" <viroemer () cisco com> escribió:

    Pablo,

Sorry for the delay. The "file_identify" preprocessor is designed to
work
more as a "reputation" based system. I think you will get better
millage by
using the new "file_type" rule keyword in a plain-old snort rule.

Something like this:

alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
downloaded";
flow:to_client,established; file_type:GIF; sid:1000000;)
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
downloaded";
flow:to_client,established; file_type:PNG; sid:1000001;)


You'll still need to have "file_magic.conf" included in your Snort
configuration, but you will not need the file_identify preprocessor.

~Victor

On 03/17/15 4:57, Pablo Cantos Polaino wrote:

Hello all,

I already sent this issue to snort-devel with the same subject since I
am
not sure if either I am configuring Snort in the right way or there is
an
internal malfunction to fix.

I have been testing the new experimental preprocessor called File
Services
in order to get an event every time a file go through our network. To
carry
on these tests I have used two pcap files. The first one is a 1GB-size
pcap
with a great number of files and the second one is a short pcap
generated
on my computer when I downloaded a GIF file.

My snort.conf file is configured like this at the end:

include file_magic.conf

   preprocessor file_inspect: type_id, signature, \
               capture_queue_size 5000, \
               capture_disk /home/file_capture/tmp/



In both cases files are captured by the preprocessor, as you can see
below
(1GB pcap output):

============================================================
===================

   Action Stats:
       Alerts:            0 (  0.000%)
       Logged:            0 (  0.000%)
       Passed:            0 (  0.000%)
Limits:
        Match:            0
        Queue:            0
          Log:            0
        Event:            0
        Alert:            0
Verdicts:
        Allow:      8418451 ( 97.482%)
        Block:            0 (  0.000%)
      Replace:            0 (  0.000%)
    Whitelist:       217492 (  2.518%)
    Blacklist:            0 (  0.000%)
       Ignore:            0 (  0.000%)

============================================================
===================



============================================================
===================

   File Preprocessor Statistics
    Total file type callbacks:            576
    Total file signature callbacks:       578
    Total files would saved to disk:      574
    Total files saved to disk:            320
    Total file data saved to disk:        483039    bytes
    Total files duplicated:               254
    Total files reserving failed:         2
    Total file capture min:               0
    Total file capture max:               2
    Total file capture memcap:            0
    Total files reading failed:           0
    Total file agent memcap failures:     0
    Total files sent:                     0
    Total file data sent:                 0
    Total file transfer failures:         0

============================================================
===================
File type stats:
           Type              Download   (Bytes)      Upload
 (Bytes)
            GZ( 33)          2          5580056      0          0
           SWF( 52)          1          65991        0          0
           GIF( 62)          7          16516        0          0
           GIF( 63)          275        151718       0          0
           PNG( 69)          266        256724       0          0
          JPEG( 70)          2          35566        0          0
           BMP(148)          2          4204         0          0
           ICO(149)          21         187894       0          0
              Total          576        6298669      0          0
File signature stats:
           Type              Download   Upload
            GZ( 33)          2          0
           SWF( 52)          1          0
           GIF( 62)          7          0
           GIF( 63)          275        0
           PNG( 69)          266        0
          JPEG( 70)          2          0
           BMP(148)          2          0
           ICO(149)          21         0
              Total          576        0
File type verdicts:
          UNKNOWN:           576
              LOG:           0
             STOP:           0
            BLOCK:           0
           REJECT:           0
          PENDING:           0
     STOP CAPTURE:           0
            Total:           576
File signature verdicts:
          UNKNOWN:           578
              LOG:           0
             STOP:           0
            BLOCK:           0
           REJECT:           0
          PENDING:           0
     STOP CAPTURE:           0
            Total:           578
Total files processed:             68985
Total files data processed:        97156439  bytes
Total files buffered:              576
Total files released:              574
Total files freed:                 2
Total files captured:              574
Total files within one packet:     561
Total buffers allocated:           641
Total buffers freed:               64
Total buffers released:            577
Maximum file buffers used:         64
Total buffers free errors:         0
Total buffers release errors:      0
Total memcap failures:             0
Total memcap failures at reserve:  0
Total reserve failures:            0
Total file capture size min:       0
Total file capture size max:       0
Total capture max before reserve:  2
Total file signature max:          0
Maximum buffers can allocate:      3196
Number of buffers in use:          0
Number of buffers in free list:    2619
Number of buffers in release list: 577

============================================================
===================



Following the instructions given as examples inside the file
README.file, I
have included the following rules to get an alert every time Snort
detects
a file:

alert (msg: "GIF file"; gid:146; sid:63; rev:1; metadata: rule-type

   preproc;)



After that, no alert showed up.

I went deep inside the code to find out what the reason is and found
the
following piece of code that confused me:

snort/src/dynamic-preprocessors/file/file_agent.c:601-614

/*

    * File type callback when file type is identified
   *
   * For file capture or file signature, FILE_VERDICT_PENDING must be
returned
   */
static File_Verdict file_agent_type_callback(void* p, void* ssnptr,
          uint32_t file_type_id, bool upload, uint32_t file_id)
{
      file_inspect_stats.file_types_total++;
      if (file_signature_enabled || file_capture_enabled)
          return FILE_VERDICT_UNKNOWN;
      else
          return FILE_VERDICT_LOG;
}



You can read on the description that FILE_VERDICT_PENDING must be
returned
when file capture OR file signature is enabled, but what really the
code
does is to return FILE_VERDICT_UNKNOWN when capture or signature are
enabled.

After see that, I have modified the snort.conf by carrying on the
following
changes:

Replace this:

preprocessor file_inspect: type_id, signature, \

                capture_queue_size 5000, \
               capture_disk /home/file_capture/tmp/



By:

preprocessor file_inspect: type_id


This way I forced to go through the ELSE and return a
FILE_VERDICT_LOG.
After this change, and using the same two alert rules, we run snort,
getting alerts like these below:

03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}

   192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/

-> 192.168.203.61:38976
03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}
192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/>
->
192.168.203.61:38976
03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}
192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/>
->
192.168.203.61:38977


...

   and getting the following output at the end:

============================================================
===================

   Action Stats:
       Alerts:          275 (  0.003%)
       Logged:          275 (  0.003%)
       Passed:            0 (  0.000%)
Limits:
        Match:            0
        Queue:            0
          Log:            0
        Event:            0
        Alert:            0
Verdicts:
        Allow:      8418514 ( 97.482%)
        Block:            0 (  0.000%)
      Replace:            0 (  0.000%)
    Whitelist:       217429 (  2.518%)
    Blacklist:            0 (  0.000%)
       Ignore:            0 (  0.000%)

============================================================
===================




============================================================
===================
File Preprocessor Statistics
    Total file type callbacks:            576
    Total file signature callbacks:       0
    Total files would saved to disk:      0
    Total files saved to disk:            0
    Total file data saved to disk:        0         bytes
    Total files duplicated:               0
    Total files reserving failed:         0
    Total file capture min:               0
    Total file capture max:               0
    Total file capture memcap:            0
    Total files reading failed:           0
    Total file agent memcap failures:     0
    Total files sent:                     0
    Total file data sent:                 0
    Total file transfer failures:         0

============================================================
===================
File type stats:
           Type              Download   (Bytes)      Upload
 (Bytes)
            GZ( 33)          2          0            0          0
           SWF( 52)          1          0            0          0
           GIF( 62)          7          0            0          0
           GIF( 63)          275        0            0          0
           PNG( 69)          266        0            0          0
          JPEG( 70)          2          0            0          0
           BMP(148)          2          0            0          0
           ICO(149)          21         0            0          0
              Total          576        0            0          0
File signature stats:
           Type              Download   Upload
              Total          0          0
File type verdicts:
          UNKNOWN:           0
              LOG:           576
             STOP:           0
            BLOCK:           0
           REJECT:           0
          PENDING:           0
     STOP CAPTURE:           0
            Total:           576
File signature verdicts:
          UNKNOWN:           0
              LOG:           0
             STOP:           0
            BLOCK:           0
           REJECT:           0
          PENDING:           0
     STOP CAPTURE:           0
            Total:           0
Total files processed:             68987
Total files data processed:        42751396  bytes
Total files buffered:              0
Total files released:              0
Total files freed:                 0
Total files captured:              0
Total files within one packet:     0
Total buffers allocated:           0
Total buffers freed:               0
Total buffers released:            0
Maximum file buffers used:         0
Total buffers free errors:         0
Total buffers release errors:      0
Total memcap failures:             0
Total memcap failures at reserve:  0
Total reserve failures:            0
Total file capture size min:       0
Total file capture size max:       0
Total capture max before reserve:  0
Total file signature max:          0

============================================================
===================



As you can see, in the "File type verdicts" section I got all the
files
with verdict LOG. Also, I got 275 alerts that match the 275 GIF files
detected by Snort.

I am not sure if this is the expected behavior of this feature or
maybe
I
am not configuring Snort properly.

Am I doing something wrong or configuring the preprocessor in a wrong
way?

Thanks for your help and best Regards,

Pablo Cantosredborder.org / pcantos () redborder org



This body part will be downloaded on demand.



This body part will be downloaded on demand.



------------------------------------------------------------
------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub
for
all
things parallel software development, from weekly thought leadership
blogs
to
news, videos, case studies, tutorials and more. Take a look and join
the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest
Snort news!

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!

Current thread:

Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Mar 12)
- <Possible follow-ups>
- Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Mar 17)
  - Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Mar 27)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Jaime Nebrera (Mar 27)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Mar 27)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Jaime Nebrera (Mar 27)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Mar 30)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Mar 30)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Mar 31)
    - Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Mar 31)