Re: Snort.org Blog: Snort 2.9.1 HTTP and SMTP logging features

[Jason Haar <Jason.Haar () trimble co nz>]

Hi Joel

Great feature - are you going to generalize this to other protocols too?
I'd love to see filenames and authentication details for CIFS for starters?

Keep up the good work

Jason

On 17/09/11 10:01, Joel Esler wrote:
> I thought this one was of special interest to our community, so forgive the cross post.
>
> This post is by: Bhagyashree Bantwal, one oelof our Snort Developers!
>
> ------
>
> http://blog.snort.org/2011/09/snort-291-http-and-smtp-logging.html
>
> Snort 2.9.1 HTTP and SMTP logging features
>
> To provide better context to the alerts generated by Snort, Snort version 2.9.1 introduced some new logging features 
> for HTTP and SMTP which will help the user to better analyze the alerts. This logging of extra data by HTTP Inspect 
> and SMTP preprocessors is similar to the X-Forwarded-For/True-Client-IP logging introduced prior to Snort 2.9.1.
>
> HTTP Logging:
>
> Let's talk about HTTP URI and Hostname logging.
>
> How to enable logging of HTTP URI and Hostname?
>
> To turn on the logging of the HTTP Request URI, the config option to use is "log_uri" and to enable the logging of 
> hostnames, use the config option "log_hostname" as shown in the example below:
>
> preprocessor http_inspect: global memcap 
> preprocessor http_inspect_server: log_uri \
> log_hostname
>
> The memcap here will determine the maximum amount of memory the HTTP Inspect preprocessor will use for logging the 
> URI and Hostname data. You can refer to the Snort Manual for further details.
>
> When these features are turned on in HTTP Inspect, the HTTP Request URI and HTTP Request hostname headers are 
> extracted and logged to unified2 as an "extra data event" with each alert generated for that particular session. It 
> is recommended to turn on stream5 reassembly with PAF on HTTP ports to ensure correctness and accuracy.
>
> When a HTTP Request URI is greater than 2048 or when a HTTP hostname (specified in the "Host" Request header) is 
> greater than 256, Snort will log the truncated the URI and/or hostname. A preprocessor alert with GID:119 and SID:25 
> is generated when hostname exceeds 256 bytes.
>
> There is also a preprocessor alert with GID:119 and SID:24 for multiple "Host" headers in one HTTP request.
>
> Please note that the URI and hostname are only logged in Unified2 mode and not logged with -A cmg.
>
> So, how do you read the output?
>
> Unified2 can be read using the Snort tool u2spewfoo. You can find it under the tools/u2spewfoo directory in the Snort 
> tarball.
>
> Example of the Output:
>
> (Event)
> sensor id: 0 event id: 2 event second: 1299776137 event microsecond: 355217
> sig id: 1 gen id: 1 revision: 0 classification: 0
> priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
> src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0
>
> Packet
> sensor id: 0 event id: 2 event second: 1299776137
> packet second: 1299776137 packet microsecond: 355217
> linktype: 1 packet_length: 214
> [ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
> [ 16] 00 C8 00 04 00 00 40 06 5C 19 0A 01 02 03 0A 09 ......@.\.......
> [ 32] 08 07 ED 26 00 50 00 00 00 02 00 00 00 02 50 10 ...&.P........P.
> [ 48] 01 00 0C 19 00 00 47 45 54 20 2F 20 48 54 54 50 ......GET / HTTP
> [ 64] 2F 31 2E 30 0D 0A 48 6F 73 74 3A 20 77 77 77 2E /1.0..Host: www.
> [ 80] 77 33 2E 0D 0A 20 6F 72 67 20 0D 0A 55 73 65 72 w3... org ..User
> [ 96] 2D 41 67 65 6E 74 3A 20 4D 6F 7A 69 6C 6C 61 2F -Agent: Mozilla/
> [ 112] 35 2E 30 20 28 58 31 31 3B 20 55 3B 20 4C 69 6E 5.0 (X11; U; Lin
> [ 128] 75 78 20 78 38 36 5F 36 34 3B 20 65 6E 2D 55 53 ux x86_64; en-US
> [ 144] 3B 20 72 76 3A 31 2E 39 2E 32 2E 31 35 29 20 47 ; rv:1.9.2.15) G
> [ 160] 65 63 6B 6F 2F 32 30 31 31 30 33 30 33 20 55 62 ecko/20110303 Ub
> [ 176] 75 6E 74 75 2F 31 30 2E 31 30 20 28 6D 61 76 65 untu/10.10 (mave
> [ 192] 72 69 63 6B 29 20 46 69 72 65 66 6F 78 2F 33 2E rick) Firefox/3.
> [ 208] 36 2E 31 35 0D 0A 6.15..
>
>
>
> (Event)
> sensor id: 0 event id: 3 event second: 1299776137 event microsecond: 355254
> sig id: 1 gen id: 1 revision: 0 classification: 0
> priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
> src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0
>
> Packet
> sensor id: 0 event id: 3 event second: 1299776137
> packet second: 1299776137 packet microsecond: 355254
> linktype: 1 packet_length: 176
> [ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
> [ 16] 00 A2 00 05 00 00 40 06 5C 3E 0A 01 02 03 0A 09 ......@.\>......
> [ 32] 08 07 ED 26 00 50 00 00 00 A2 00 00 00 02 50 10 ...&.P........P.
> [ 48] 01 00 8F E5 00 00 41 63 63 65 70 74 2D 45 6E 63 ......Accept-Enc
> [ 64] 6F 64 69 6E 67 3A 20 67 7A 69 70 2C 64 65 66 6C oding: gzip,defl
> [ 80] 61 74 65 0D 0A 41 63 63 65 70 74 2D 43 68 61 72 ate..Accept-Char
> [ 96] 73 65 74 3A 20 49 53 4F 2D 38 38 35 39 2D 31 2C set: ISO-8859-1,
> [ 112] 75 74 66 2D 38 3B 71 3D 30 2E 37 2C 2A 3B 71 3D utf-8;q=0.7,*;q=
> [ 128] 30 2E 37 0D 0A 4B 65 65 70 2D 41 6C 69 76 65 3A 0.7..Keep-Alive:
> [ 144] 20 31 31 35 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 115..Connection
> [ 160] 3A 20 6B 65 65 70 2D 61 6C 69 76 65 0D 0A 0D 0A : keep-alive....
>
>
> (ExtraDataHdr)
> event type: 4 event length: 33
>
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1299776137
> type: 9 datatype: 1 bloblength: 9 HTTP URI: /
>
>
> (ExtraDataHdr)
> event type: 4 event length: 43
>
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1299776137
> type: 10 datatype: 1 bloblength: 19 HTTP Hostname: www.w3.org
>
>
> In the output above, two extra data events are generated for the event with GID:1 and SID: 1. The first extra data 
> event displays the URI and the second displays the hostname associated with that session.
>
> It is important to note that the unique "event id" and "event second" are used to correlate the alerts with their 
> extra data. The "event type" indicates the presence of an "extra data" record and the unique "type" will determine 
> the type of the extra data event logged. Here is a list of extra data types used in Snort.
>
> Type 1: True-Client-IP/XFF IPv4 address
> Type 2: True-Client-IP/XFF IPv6 address
> Type 4: HTTP Gzip decompressed data
> Type 5: SMTP filename
> Type 6: SMTP MAIL FROM addresses
> Type 7: SMTP RCPT TO addresses
> Type 8: SMTP Email headers
> Type 9: HTTP Request URI
> Type 10: HTTP Request Hostname
> Type 11: Packet's IPv6 Source IP Address
> Type 12: Packet's IPv6 Destination IP Address
>
> SMTP Logging:
>
> The SMTP preprocessor, similar to HTTP Inspect preprocessor, can log extra data associated with each alert. The data 
> that SMTP logs are as follows:
>
> 1. The email addresses in the MAIL FROM command.
> 2. The email addresses in the RCPT TO command. When there are multiple RCPT TO headers, the email addresses are 
> concatenated using commas.
> 3. The filename of the MIME attachment extracted from the "Content-Disposition" header. Multiple filenames are 
> appended with commas.
> 4. SMTP email headers.
>
> How to enable logging of SMTP "extra data"?
>
> SMTP preprocessor can log the extra data mentioned above using the following config options.
>
> preprocessor smtp: memcap 838860 \
> log_mailfrom \
> log_rcptto \
> log_filename \
> log_email_hdrs \
> email_hdrs_log_depth 56
>
> The config options "log_mailfrom", "log_rcptto", "log_filename" and "log_email_hdrs" do not take any arguments. 
> However, email_hdrs_log_depth requires the user to pass a number between 0-20480, which will determine in bytes the 
> amount of email headers logged. The config option "memcap" is used to determine the maximum memory used for logging 
> the above mentioned data.
>
> Again, stream5 reassembly needs to be turned on for SMTP ports for this feature to work correctly. Without stream5 
> reassembly, the SMTP extra data won't be logged.
>
> How to read the Output?
>
> Like HTTP extra data, SMTP extra data can be read using the u2spewfoo and "event id" and "event second" are used to 
> correlate the alerts with their extra data events.
>
> Example of the Output:
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1273194706
> type: 6 datatype: 1 bloblength: 33 SMTP MAIL FROM Addresses: 
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1273194706
> type: 7 datatype: 1 bloblength: 57 SMTP RCPT TO Addresses: ,
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1273194706
> type: 5 datatype: 1 bloblength: 26 SMTP Attachment Filename: sample.txt,foo.txt
>
> (ExtraData)
> sensor id: 0 event id: 2 event second: 1273194706
> type: 8 datatype: 1 bloblength: 323 SMTP EMAIL HEADERS:
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Type: multipart/mixed; boundary="_----------=_12731947069000"
> Date: Thu, 6 May 2010 21:11:46 -0400
> From: bbantwal () sourcefire com
> To: bbantwal () sourcefire com
> Subject: Email Sent via Perl
> X-Mailer: MIME::Lite 3.027 (F2.74; T1.29; A2.06; B3.07; Q3.07
> <bbantwal () sourcefire com><bbantwal () sourcefire com><foo-barfoo () gmail com>
>
> We will not be updating the spo_database output method to insert this information into the database.  As a reminder, 
> this output method will be removed in a future Snort release in 2012.  Sourcefire has turned over the maintenance of 
> the "ACID" database schema to the Barnyard2 group, and will also, no longer be maintaing it either.
>
> --
> Joel Esler
> Senior Research Engineer, VRT
> OpenSource Community Manager
> Sourcefire
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> http://p.sf.net/sfu/rim-devcon-copy2
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users
>
> Please visit http://blog.snort.org to stay current on all the latest Snort news!

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

------------------------------------------------------------------------------
BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
http://p.sf.net/sfu/rim-devcon-copy2

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!