Snort mailing list archives
Re: Getting alerts for every file Snort detects and File Services preprocessor
From: Victor Roemer <viroemer () cisco com>
Date: Wed, 01 Apr 2015 10:56:17 -0400
Ok, I'm guessing that this is due to internal default's for event_queue and the RNG in rule evaluation ordering. Try adding the following to your snort.conf config event_queue: max_queue 100 log 100 and adding the flag "-H" when running Snort. The -H removes the RNG in Snort runtime, so using it in production would not be a good idea; we only use it for testing on our side. On 04/01/15 9:18, Pablo Cantos Polaino wrote:
Hi Victor, Since there is no PDF files in my PCAP file, I've tried what you propose for the following file types: MSEXE, ZIP, GZ, SWF, GIF, PNG, JPEG, BMP and ICO. Using the file types below (getting from file_magic.conf) and the rules below I've got something quite coherent, but every time I run snort I get a different number of alerts, lower or equal to the right number or files. For instance, there are 566 files inside the PCAP file (constant number got from the File preprocessor) and I've got volatile number of alerts like 558, 556, 557, ... File types: file type:MSEXE; id:21; category:Executables,Dynamic Analysis Capable; msg:"Windows/DOS executable file "; rev:1; content:| 4D 5A|; offset:0; file type:ZIP; id:29; category:Archive; msg:"PKZIP archive file"; rev:1; content:| 50 4B 03 04 |; offset:0; file type:GZ; id:33; category:Archive; msg:"GZ"; rev:1; content:| 1F 8B 08 |; offset:0; file type:SWF; id:52; category:Multimedia; msg:"Flash file "; rev:1; content:| 43 57 53 |; offset:0; file type:GIF; id:62; category:Graphics; msg:"GIF"; rev:1; content:| 47 49 46 38 37 61 |; offset:0; group:multimedia; file type:GIF; id:63; category:Graphics; msg:"GIF"; rev:1; content:| 47 49 46 38 39 61 |; offset:0; group:multimedia; file type:PNG; id:69; category:Graphics; msg:"Portable Network Graphics file"; rev:1; content:| 89 50 4E 47 0D 0A 1A 0A |; offset:0; group:multimedia; file type:JPEG; id:70; category:Graphics; msg:"JPEG/JFIF graphics file"; rev:1; content:| FF D8 FF E0 |; offset:0; group:multimedia; file type:BMP; id:148; category:Graphics; msg:"Bitmap image file"; rev:1; content:|42 4D |; offset:0; group:multimedia; file type:ICO; id:149; category:Graphics; msg:"Windows icon file"; rev:1; content:| 00 00 01 00 |; offset:0; Rules: alert tcp any any -> any any (msg:"MSEXE: MZ"; content:"|4D 5A|"; offset:0; file_type:MSEXE; sid:5000001;) alert tcp any any -> any any (msg:"ZIP: PK"; content:"|50 4B 03 04|"; offset:0; file_type:ZIP; sid:5000002;) alert tcp any any -> any any (msg:"GZ: "; content:"|1F 8B 08|"; offset:0; file_type:GZ; sid:5000003;) alert tcp any any -> any any (msg:"SWF: "; content:"|43 57 53|"; offset:0; file_type:SWF; sid:5000004;) alert tcp any any -> any any (msg:"GIF: GIF87a"; content:"|47 49 46 38 37 61|"; offset:0; file_type:GIF; sid:5000005;) alert tcp any any -> any any (msg:"GIF: GIF89a"; content:"|47 49 46 38 39 61|"; offset:0; file_type:GIF; sid:5000006;) alert tcp any any -> any any (msg:"PNG: "; content:"|89 50 4E 47 0D 0A 1A 0A|"; offset:0; file_type:PNG; sid:5000007;) alert tcp any any -> any any (msg:"JPEG: "; content:"|FF D8 FF E0|"; offset:0; file_type:JPEG; sid:5000008;) alert tcp any any -> any any (msg:"BMP: "; content:"|42 4D|"; offset:0; file_type:BMP; sid:5000009;) alert tcp any any -> any any (msg:"ICO: "; content:"|00 00 01 00|"; offset:0; file_type:ICO; sid:5000010;) If I use the rules below (going through the File preprocessor) I always get the same number of alerts that's equal to the number of files (566). alert (msg: "MSEXE file"; gid:146; sid:21;) alert (msg: "ZIP file"; gid:146; sid:29;) alert (msg: "GZ file"; gid:146; sid:33;) alert (msg: "SWF file"; gid:146; sid:52;) alert (msg: "GIF(62) file"; gid:146; sid:62;) alert (msg: "GIF(63) file"; gid:146; sid:63;) alert (msg: "PNG file"; gid:146; sid:69;) alert (msg: "JPEG file"; gid:146; sid:70;) alert (msg: "BMP file"; gid:146; sid:148;) alert (msg: "ICO file"; gid:146; sid:149;) I've also tried to detect just one type of files and the results are similar, when I use the way I get a volatile number of alerts, but I manage to get the same number of alerts if I follow the second way. To us the second way is more reliable since we don't want to skip any file detected by File preprocessor. On the other hand, we have also managed to log these alerts to unified2, so this would not be a disadvantage.
Ok, for some reason I believed that u2 logging would be possible. If you would like to use the preprocessor events that is fine; but as you've already identified, the file type events do not work when signature or capture is enabled. Until we fix that, you'd have to resolve that in your own version of the file_inspect preprocessor or wait for us to address it. (The bug was written, but is currently unknown when we will work on it)
Best Regards, Pablo Cantos redborder.org / pcantos () redborder org 2015-03-31 17:54 GMT+02:00 Victor Roemer <viroemer () cisco com>:Pablo, Yes, your right. Flowbits apply per-session so the rule would only be capable of alerting 1x (per-session). In that case, you could also just do a rule like so: file type:PDF; content:|25 50 44 46 2d|; offset:0; id:1; alert tcp any any -> any any (msg:"PDF"; content:"%PDF-"; offset:0; file_type:PDF; sid:1000002;) What I've done is removed the flowbits and added offset:0; from the previous rule. By matching the content that triggers the file rule to identify PDF, this rule should alert on the start of every PDF file seen. On 03/31/15 7:19, Pablo Cantos Polaino wrote:Hi Victor, We already thought about using flowbits, but (and please correct me if I'm wrong) we didn't considered it effective since once a bit is set after detecting a file, this rule will never trigger any alert. I've launched some tests following your proposals and the numbers I got are not much coherent. For these tests I've used a capture with 107 GIF files inside. The file preprocessor is able to detect these 107 files and below you could find the number of alerts I get using the following rules: Rules: alert (msg: "GIF(62) file"; gid:146; sid:62;) alert (msg: "GIF(63) file"; gid:146; sid:63;) Number of alerts: 107 Rule: alert tcp any any -> any any (msg: "GIF file"; file_type:GIF; sid:3000011; rev:1;) Number of alerts: 111 (4 alerts more than the number of files detected. Possibly due to one or more than one file takes up one or more packet) Rule: alert tcp any any -> any any (msg: "GIF file"; flow:to_client,established; file_type:GIF; sid:3000013; rev:1;) Number of alerts: 24 (much less alerts than the number of files detected. Possibly due to the direction flow, but as you can see below, adding 24 and 25 we don't get the total number of files) Rule: alert tcp any any -> any any (msg: "GIF file"; flow:to_server,established; file_type:GIF; sid:3000014; rev:1;) Number of alerts: 25 Rule: alert tcp any any -> any any (msg: "GIF file flowbits"; flowbits:isnotset,gif; file_type:GIF; flowbits:set,gif; sid:4000011; rev:1;) Number of alerts: 2 Rule: alert tcp any any -> any any (msg: "GIF file flowbits"; flowbits:isnotset,gif; flow:to_client,established; file_type:GIF; flowbits:set,gif; sid:4000013; rev:1;) Number of alerts: 20 Rule: alert tcp any any -> any any (msg: "GIF file flowbits"; flowbits:isnotset,gif; flow:to_server,established; file_type:GIF; flowbits:set,gif; sid:4000014; rev:1;) Number of alerts: 20 As you can see, just using the two first rules which employ directly the File preprocessor (rules with gid:146) gives us just one alert per file, and this is what we are interested in. This way we can be sure that, through the alerts, we are getting information from every file detected in File preprocessor. The only thing we have to put care into is configuring properly the preprocessor like this below: preprocessor file_inspect: type_id and await the related bug could be fixed. In a previous mail you said "The "file_identify" preprocessor is designed to work more as a "reputation" based system. I think you will get better millage by using the new "file_type" rule keyword in a plain-old snort rule". Our conclusion is quite opposite to your recommendation, whose grounds we didn't understand, I mean, why is preferable using the file_type keyword in a plain-old snort rule instead of using a rule with gid:146? I hope I explained my self.* Because it doesn't work the way you want it to is 'one' reason. * Text rules are more managable and less error prone than custom preprocessors. * You can take advantage of logging to unified2 vs the one-the-side mechanism currently employed by the file preprocessor. But specifically, your intentions were not known to me when I sent that. :/Best Regards, Pablo Cantos redborder.org / pcantos () redborder org 2015-03-30 19:22 GMT+02:00 Victor Roemer <viroemer () cisco com>: The solution too the "too many alerts" can be resolved with theapplication of "flowbits". For example: file type:PDF; content:|25 50 44 46 2d|; offset:0; id:1; alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf; file_type:PDF; flowbits:set,pdf; sid:1000000;) However, it is also recommended that you add a "content" option too the rule so that you may take advantage of the fast pattern matcher. This may seem odd, but it will be faster. In the next rule, the content being matched "mirrors" the content in the "file" rule above exactly. alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf; content:"%PDF-"; file_type:PDF; flowbits:set,pdf; sid:1000001;) ~Victor On 03/30/15 5:53, Pablo Cantos Polaino wrote: Hi Victor,First of all, thanks for your reply! Regarding your proposal below: The "file_identify" preprocessor is designed to work more as a "reputation" based system. I think you will get better millage by using the new"file_type" rule keyword in a plain-old snort rule. Something like this: alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded"; flow:to_client,established; file_type:GIF; sid:1000000;) alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded"; flow:to_client,established; file_type:PNG; sid:1000001;) We already tried that and we got a lot of alerts. The sizer the fileis the more alerts are fired, and this is not the behavior we expected since we would overpopulate the disk with repeated events. For instance, after downloading a BZ file with size of 7,5MB, we've got 855 alerts from one just rule: alert tcp any any -> any any (msg: "BZ file"; flow:to_client,established; file_type:BZ; sid:3000033;) And we get just one alert if we use the rule below and the following configuration: Conf (just type_id, neither signature nor capture) preprocessor file_inspect: type_id Rule: alert (msg: "BZ file"; gid:146; sid:32;) This is why we discarded some weeks ago the way of writing the rule as you proposed now. If it were a way to get just one alert per file it would be great. Meanwhile, we await the bug can be fixed. Best Regards, Pablo Cantos redborder.org / pcantos () redborder org 2015-03-27 22:46 GMT+01:00 Jaime Nebrera <jnebrera () redborder org>: Hi Víctor,I'm not that technical ;) Pablo will coment on your ideas, I just tried to give you a deeper context of what Pablo (as part of my team) was doing but for sure, they are welcomed. I let them continue El 27/03/2015 21:25, "Victor Roemer" <viroemer () cisco com> escribió: Jamie,That's neat, I'll keep an eye on it. :) Reviewing over how the "file_identify" works currently, when doing capture or signature its not possible to also get the file type event via the preprocessor. I'll open an bug for this. In order to get the file type events with these further settings, more sophisticated tracking needs to be in-place in the file_identify preproc; the tricky bits being that the file service code is generating the events in question; however once a verdict is passed to file service- an action will be taken at that time. To clarify the verdicts - As I understand it, passing FILE_VERDICT_UNKNOWN is how processing continues further. The FILE_VERDICT_PENDING when passed to file service api, should cause the packet to be dropped, in-order to give the interfacing code time to retrieve a true verdict when the re-transmitted packet is seen. On 03/27/15 16:04, Jaime Nebrera wrote: Hi Víctor,Pablo's work is geared towards a more ambitious goal: 1) Intercept the file and send it to a S3 based storage platform for further analysis or whatever 2) Provide as much context about such interception as possible, like from were to whom, URL, email, etc This we hope to open source and make it public as soon as it is usable in our repository www.github.com/redborder El 27/03/2015 20:50, "Victor Roemer" <viroemer () cisco com> escribió: Pablo, Sorry for the delay. The "file_identify" preprocessor is designed towork more as a "reputation" based system. I think you will get better millage by using the new "file_type" rule keyword in a plain-old snort rule. Something like this: alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded"; flow:to_client,established; file_type:GIF; sid:1000000;) alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded"; flow:to_client,established; file_type:PNG; sid:1000001;) You'll still need to have "file_magic.conf" included in your Snort configuration, but you will not need the file_identify preprocessor. ~Victor On 03/17/15 4:57, Pablo Cantos Polaino wrote: Hello all, I already sent this issue to snort-devel with the same subject since I am not sure if either I am configuring Snort in the right way or there is an internal malfunction to fix. I have been testing the new experimental preprocessor called File Services in order to get an event every time a file go through our network. To carry on these tests I have used two pcap files. The first one is a 1GB-size pcap with a great number of files and the second one is a short pcap generated on my computer when I downloaded a GIF file. My snort.conf file is configured like this at the end: include file_magic.conf preprocessor file_inspect: type_id, signature, \ capture_queue_size 5000, \ capture_disk /home/file_capture/tmp/ In both cases files are captured by the preprocessor, as you can see below (1GB pcap output): ============================================================ =================== Action Stats: Alerts: 0 ( 0.000%) Logged: 0 ( 0.000%) Passed: 0 ( 0.000%) Limits: Match: 0 Queue: 0 Log: 0 Event: 0 Alert: 0 Verdicts: Allow: 8418451 ( 97.482%) Block: 0 ( 0.000%) Replace: 0 ( 0.000%) Whitelist: 217492 ( 2.518%) Blacklist: 0 ( 0.000%) Ignore: 0 ( 0.000%) ============================================================ =================== ============================================================ =================== File Preprocessor Statistics Total file type callbacks: 576 Total file signature callbacks: 578 Total files would saved to disk: 574 Total files saved to disk: 320 Total file data saved to disk: 483039 bytes Total files duplicated: 254 Total files reserving failed: 2 Total file capture min: 0 Total file capture max: 2 Total file capture memcap: 0 Total files reading failed: 0 Total file agent memcap failures: 0 Total files sent: 0 Total file data sent: 0 Total file transfer failures: 0 ============================================================ =================== File type stats: Type Download (Bytes) Upload (Bytes) GZ( 33) 2 5580056 0 0 SWF( 52) 1 65991 0 0 GIF( 62) 7 16516 0 0 GIF( 63) 275 151718 0 0 PNG( 69) 266 256724 0 0 JPEG( 70) 2 35566 0 0 BMP(148) 2 4204 0 0 ICO(149) 21 187894 0 0 Total 576 6298669 0 0 File signature stats: Type Download Upload GZ( 33) 2 0 SWF( 52) 1 0 GIF( 62) 7 0 GIF( 63) 275 0 PNG( 69) 266 0 JPEG( 70) 2 0 BMP(148) 2 0 ICO(149) 21 0 Total 576 0 File type verdicts: UNKNOWN: 576 LOG: 0 STOP: 0 BLOCK: 0 REJECT: 0 PENDING: 0 STOP CAPTURE: 0 Total: 576 File signature verdicts: UNKNOWN: 578 LOG: 0 STOP: 0 BLOCK: 0 REJECT: 0 PENDING: 0 STOP CAPTURE: 0 Total: 578 Total files processed: 68985 Total files data processed: 97156439 bytes Total files buffered: 576 Total files released: 574 Total files freed: 2 Total files captured: 574 Total files within one packet: 561 Total buffers allocated: 641 Total buffers freed: 64 Total buffers released: 577 Maximum file buffers used: 64 Total buffers free errors: 0 Total buffers release errors: 0 Total memcap failures: 0 Total memcap failures at reserve: 0 Total reserve failures: 0 Total file capture size min: 0 Total file capture size max: 0 Total capture max before reserve: 2 Total file signature max: 0 Maximum buffers can allocate: 3196 Number of buffers in use: 0 Number of buffers in free list: 2619 Number of buffers in release list: 577 ============================================================ =================== Following the instructions given as examples inside the file README.file, I have included the following rules to get an alert every time Snort detects a file: alert (msg: "GIF file"; gid:146; sid:63; rev:1; metadata: rule-type preproc;) After that, no alert showed up. I went deep inside the code to find out what the reason is and found the following piece of code that confused me: snort/src/dynamic-preprocessors/file/file_agent.c:601-614 /* * File type callback when file type is identified * * For file capture or file signature, FILE_VERDICT_PENDING must be returned */ static File_Verdict file_agent_type_callback(void* p, void* ssnptr, uint32_t file_type_id, bool upload, uint32_t file_id) { file_inspect_stats.file_types_total++; if (file_signature_enabled || file_capture_enabled) return FILE_VERDICT_UNKNOWN; else return FILE_VERDICT_LOG; } You can read on the description that FILE_VERDICT_PENDING must be returned when file capture OR file signature is enabled, but what really the code does is to return FILE_VERDICT_UNKNOWN when capture or signature are enabled. After see that, I have modified the snort.conf by carrying on the following changes: Replace this: preprocessor file_inspect: type_id, signature, \ capture_queue_size 5000, \ capture_disk /home/file_capture/tmp/ By: preprocessor file_inspect: type_id This way I forced to go through the ELSE and return a FILE_VERDICT_LOG. After this change, and using the same two alert rules, we run snort, getting alerts like these below: 03/16-12:50:22.350000 [**] [146:63:1] GIF [**] [Priority: 0] {TCP} 192.168.202.78:80 <http://192.168.202.78/> < http://192.168.202.78/ -> 192.168.203.61:38976 03/16-12:50:22.350000 [**] [146:63:1] GIF [**] [Priority: 0] {TCP} 192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/> -> 192.168.203.61:38976 03/16-12:50:22.350000 [**] [146:63:1] GIF [**] [Priority: 0] {TCP} 192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/> -> 192.168.203.61:38977 ... and getting the following output at the end: ============================================================ =================== Action Stats: Alerts: 275 ( 0.003%) Logged: 275 ( 0.003%) Passed: 0 ( 0.000%) Limits: Match: 0 Queue: 0 Log: 0 Event: 0 Alert: 0 Verdicts: Allow: 8418514 ( 97.482%) Block: 0 ( 0.000%) Replace: 0 ( 0.000%) Whitelist: 217429 ( 2.518%) Blacklist: 0 ( 0.000%) Ignore: 0 ( 0.000%) ============================================================ =================== ============================================================ =================== File Preprocessor Statistics Total file type callbacks: 576 Total file signature callbacks: 0 Total files would saved to disk: 0 Total files saved to disk: 0 Total file data saved to disk: 0 bytes Total files duplicated: 0 Total files reserving failed: 0 Total file capture min: 0 Total file capture max: 0 Total file capture memcap: 0 Total files reading failed: 0 Total file agent memcap failures: 0 Total files sent: 0 Total file data sent: 0 Total file transfer failures: 0 ============================================================ =================== File type stats: Type Download (Bytes) Upload (Bytes) GZ( 33) 2 0 0 0 SWF( 52) 1 0 0 0 GIF( 62) 7 0 0 0 GIF( 63) 275 0 0 0 PNG( 69) 266 0 0 0 JPEG( 70) 2 0 0 0 BMP(148) 2 0 0 0 ICO(149) 21 0 0 0 Total 576 0 0 0 File signature stats: Type Download Upload Total 0 0 File type verdicts: UNKNOWN: 0 LOG: 576 STOP: 0 BLOCK: 0 REJECT: 0 PENDING: 0 STOP CAPTURE: 0 Total: 576 File signature verdicts: UNKNOWN: 0 LOG: 0 STOP: 0 BLOCK: 0 REJECT: 0 PENDING: 0 STOP CAPTURE: 0 Total: 0 Total files processed: 68987 Total files data processed: 42751396 bytes Total files buffered: 0 Total files released: 0 Total files freed: 0 Total files captured: 0 Total files within one packet: 0 Total buffers allocated: 0 Total buffers freed: 0 Total buffers released: 0 Maximum file buffers used: 0 Total buffers free errors: 0 Total buffers release errors: 0 Total memcap failures: 0 Total memcap failures at reserve: 0 Total reserve failures: 0 Total file capture size min: 0 Total file capture size max: 0 Total capture max before reserve: 0 Total file signature max: 0 ============================================================ =================== As you can see, in the "File type verdicts" section I got all the files with verdict LOG. Also, I got 275 alerts that match the 275 GIF files detected by Snort. I am not sure if this is the expected behavior of this feature or maybe I am not configuring Snort properly. Am I doing something wrong or configuring the preprocessor in a wrong way? Thanks for your help and best Regards, Pablo Cantosredborder.org / pcantos () redborder org This body part will be downloaded on demand. This body part will be downloaded on demand. ------------------------------------------------------------ ------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
Current thread:
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 01)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Apr 01)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 02)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 07)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Apr 08)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 09)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 16)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Pablo Cantos Polaino (Apr 02)
- Re: Getting alerts for every file Snort detects and File Services preprocessor Victor Roemer (Apr 01)