Re: snort3: problem with metadata: service http in sample.rules

[Russ <rucombs () cisco com>]

On 3/5/17 1:21 PM, Marcin Dulak wrote:
> On Sun, Mar 5, 2017 at 3:05 PM, Russ <rucombs () cisco com 
> <mailto:rucombs () cisco com>> wrote:
>
>
>
>     On 3/4/17 5:09 PM, Marcin Dulak wrote:
>     > Hi,
>     >
>     > this is a follow up to http://seclists.org/snort/2017/q1/593
>     <http://seclists.org/snort/2017/q1/593>
>     To make it clear for others, that problem is due to mixing the old
>     HTTP
>     inspector and friends (rule options) with the new one.  We are phasing
>     out the old one before the beta release (it was retained for certain
>     testing scenarios).
>
>     To avoid that problem delete the old library
>     (install/lib/snort_extra/inspectors//http_server.so) or use
>     --plugin-path install/lib/snort_extra/codecs/ so you just pick up the
>     external codec(s) you need.
>     > Using  --plugin-path /usr/lib64/snort_extra/codecs alone is not enough to
>     > get http traffic detected,
>     > if snort3 sample.rules are present.
>     This is a different issue; see below.
>     >
>     > The service option present in metadata in
>     >
>     https://github.com/snortadmin/snort3/blob/89bae69d5cd980ae56ef0322b6ef7cca87a75cf2/lua/sample.rules
>     <https://github.com/snortadmin/snort3/blob/89bae69d5cd980ae56ef0322b6ef7cca87a75cf2/lua/sample.rules>
>     > seems to cause
>     > http to be undetected. To reproduce the problem:
>     >
>     > # cat /etc/yum.repos.d/copr-marcindulak-snort.repo
>     > [copr-marcindulak-snort]
>     > name=copr-marcindulak-snort
>     > baseurl=
>     >
>     https://copr-be.cloud.fedoraproject.org/results/marcindulak/snort/epel-$releasever-$basearch
>     <https://copr-be.cloud.fedoraproject.org/results/marcindulak/snort/epel-$releasever-$basearch>
>     > enabled=1
>     > gpgcheck=1
>     > gpgkey=
>     >https://copr-be.cloud.fedoraproject.org/results/marcindulak/snort/pubkey.gpg
>     <https://copr-be.cloud.fedoraproject.org/results/marcindulak/snort/pubkey.gpg>
>     >
>     > # yum -y install snort snort-extra
>     >
>     > # cp -f /etc/snort/sample.rules /etc/snort/rules/snort.rules
>     > # echo 'alert tcp any any -> any 80 (msg:"test";
>     > flow:to_server,established; http_uri; content:"/test";
>     sid:3000001;)' >>
>     > /etc/snort/rules/snort.rules
>     This rule does not have service.  Instead of deleting service from all
>     the sample rules, add service to this rule in one of the following
>     ways:
>
>     alert tcp any any -> any 80 (msg:"test";
>     flow:to_server,established; http_uri; content:"/test";
>     metadata:service http; sid:3000001;)
>
>     alert http any any -> any 80 (msg:"test";
>     flow:to_server,established; http_uri; content:"/test"; sid:3000001;)
>
>     With that I get (using -A csv):
>
>     02/26-08:19:45.017007, 5, TCP, stream_tcp, 57, C2S,
>     192.168.17.20:34616 <http://192.168.17.20:34616>,
>     192.168.17.30:80 <http://192.168.17.30:80>, 1:3000001:0, allow
>
>     stream_tcp indicates that this alert is firing on a PDU or PDU buffer,
>     not on a raw packet.  If no service rules exist, then the rules
>     will be
>     evaluated against PDUs.  With service and non-service rules,
>     non-service
>     rules only work on raw packets.  This distinction is important to
>     improve performance and reduce false positives.
>
>
> together with the default sample.rules, this rule
>
> alert tcp any any -> any 80 (msg:"test"; flow:to_server,established; 
> http_uri; content:"/test"; metadata:service http; sid:3000001;)
>
> results in buffer (type 3) instead of packet (type 2) being logged in 
> unified2.log as u2spewfoo shows below. Is this expected?
Yes
> The buffer record type is not described at 
> https://www.snort.org/faq/readme-unified2 and u2boat does not seem to 
> handle it.
You should refer to the Snort++ manual ... but that doesn't explain it 
either.  We will update it.

u2boat currently outputs packets and skips over events and other records 
like extra data.  Buffer is not a packet.  We are planning to log the 
triggering packet so that you get something, but this will not be a 
reassembled packet.
> https://github.com/jasonish/py-idstools does not understand unified2 
> buffer record type either.
>
> SNORT_LUA_PATH=/etc/snort LUA_PATH=/usr/include/snort/lua/?.lua snort 
> --daq-dir /usr/lib64/daq -c /etc/snort/snort.lua --plugin-path 
> /usr/lib64/snort_extra/codecs -R /etc/snort/rules/snort.rules -r 
> test.txt -A csv -q
> 03/05-16:30:01.076418, 5, TCP, stream_tcp, 50, C2S, 
> 192.168.17.20:44018 <http://192.168.17.20:44018>, 192.168.17.30:80 
> <http://192.168.17.30:80>, 1:3000001:1, allow
>
> # u2spewfoo /var/log/snort/unified2.log
>
> (Event)
>     sensor id: 0    event id: 36    event second: 1488729771    event 
> microsecond: 741612
>     sig id: 3000001    gen id: 1    revision: 1  classification: 1
>     priority: 3    ip source: 192.168.17.20    ip destination: 
> 192.168.17.30
>     src port: 59580    dest port: 80    ip_proto: 255 impact_flag: 
> 0    blocked: 0
>     mpls label: 0    vland id: 0    policy id: 0
>
> Buffer
>     sensor_id: 0    event_id: 36    event_second: 1488729771
>     packet_second: 1488729771    packet_microsecond: 741612
>     packet_length: 50
> [    0] 55 73 65 72 2D 41 67 65 6E 74 3A 20 63 75 72 6C User-Agent: curl
> [   16] 2F 37 2E 32 39 2E 30 0D 0A 48 6F 73 74 3A 20 73  /7.29.0. 
> <http://7.29.0.>.Host: s
> [   32] 6E 6F 72 74 30 0D 0A 41 63 63 65 70 74 3A 20 2A nort0..Accept: *
> [   48] 2F 2A /*
>
>
> However, removing "flow:to_server,established; http_uri;" we are back 
> at the raw packet, and packet logged in unified2.
Yes, but matching on raw TCP packets is easily evaded by segmentation 
and overlaps.
> alert tcp any any -> any 80 (msg:"test"; content:"/test"; metadata: 
> service http; sid:3000001;)
>
> SNORT_LUA_PATH=/etc/snort LUA_PATH=/usr/include/snort/lua/?.lua snort 
> --daq-dir /usr/lib64/daq -c /etc/snort/snort.lua --plugin-path 
> /usr/lib64/snort_extra/codecs -R /etc/snort/rules/snort.rules -r 
> test.txt -A csv -q
> 03/05-16:30:01.076418, 4, TCP, raw, 138, C2S, 192.168.17.20:44018 
> <http://192.168.17.20:44018>, 192.168.17.30:80 
> <http://192.168.17.30:80>, 1:3000001:1, allow
>
> What is the relation between http_uri and metadata: service http?
http_uri selects the URI buffer populated by http_inspect whereas 
service metadata puts this rule in the http group (same as alert http).  
The URI buffer is searched for fast patterns in that group. The packet 
buffer is not searched.  That is different from Snort 2.X.  Also, in 
Snort++ it is not possible to populate a service buffer w/o knowing the 
service, another difference from Snort 2.X.
> Marcin
>
>
>     Key takeaway:  always indicate service in rules that have service
>     buffers like http_uri.
>     >
>     > # SNORT_LUA_PATH=/etc/snort
>     LUA_PATH=/usr/include/snort/lua/?.lua snort
>     > --daq-dir /usr/lib64/daq -c /etc/snort/snort.lua --plugin-path
>     > /usr/lib64/snort_extra/codecs -R /etc/snort/rules/snort.rules -r
>     test.txt
>     > -A alert_fast -q
>     >
>     > # sed -i 's/service http//' /etc/snort/rules/snort.rules
>     > # sed -i 's/,,/,/' /etc/snort/rules/snort.rules
>     > # sed -i 's/:,/:/' /etc/snort/rules/snort.rules
>     >
>     > # SNORT_LUA_PATH=/etc/snort
>     LUA_PATH=/usr/include/snort/lua/?.lua snort
>     > --daq-dir /usr/lib64/daq -c /etc/snort/snort.lua --plugin-path
>     > /usr/lib64/snort_extra/codecs -R /etc/snort/rules/snort.rules -r
>     test.txt
>     > -A alert_fast -q
>     > 02/26-13:19:45.017007 [**] [1:3000001:0] "test" [**] [Priority:
>     0] {TCP}
>     > 192.168.17.20:34616 <http://192.168.17.20:34616> ->
>     192.168.17.30:80 <http://192.168.17.30:80>
>     >
>     > By the way most snort3 rules are incompatible with snort2 (
>     >
>     https://github.com/snortadmin/snort3/blob/master/doc/differences.txt
>     <https://github.com/snortadmin/snort3/blob/master/doc/differences.txt>).
>     > I tried to use pulledpork's modifysig to convert
>     community-rules.tar.gz
>     > into a snort3 format, but that's not a reliable way.
>     You should use snort2lua to convert rules files from 2.X to 3.0
>     format.
>     Don't let the name fool you, it converts confs and/or rules.
>     > How are you planning to transition into snort3 rules? By implementing
>     > snort3 rules support in snort2?
>     At first 2.X rules will be translated with snort2lua.  I'll defer any
>     further input to Talos.
>     >
>     > I noticed also that some type of attachments are stripped when
>     posting on
>     > snort-users.
>     > I'm attaching test.txt (pcap), but no guarantee it will be
>     available on the
>     > list.
>     >
>     > Cheers,
>     >
>     > Marcin
>     >
>     >
>     >
>     ------------------------------------------------------------------------------
>     > Check out the vibrant tech community on one of the world's most
>     > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>     >
>     >
>     > _______________________________________________
>     > Snort-users mailing list
>     > Snort-users () lists sourceforge net
>     <mailto:Snort-users () lists sourceforge net>
>     > Go to this URL to change user options or unsubscribe:
>     > https://lists.sourceforge.net/lists/listinfo/snort-users
>     <https://lists.sourceforge.net/lists/listinfo/snort-users>
>     > Snort-users list archive:
>     >
>     http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>     <http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users>
>     >
>     > Please visit http://blog.snort.org to stay current on all the
>     latest Snort news!
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>     _______________________________________________
>     Snort-users mailing list
>     Snort-users () lists sourceforge net
>     <mailto:Snort-users () lists sourceforge net>
>     Go to this URL to change user options or unsubscribe:
>     https://lists.sourceforge.net/lists/listinfo/snort-users
>     <https://lists.sourceforge.net/lists/listinfo/snort-users>
>     Snort-users list archive:
>     http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>     <http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users>
>
>     Please visit http://blog.snort.org to stay current on all the
>     latest Snort news!

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!