Stream4 and other stuff

[Martin Roesch <roesch () sourcefire com>]

Hi everyone,
     Sorry I've been quiet lately, I'm busier than a one legged man at
an ass-kicking contest lately and I haven't been very good about
answering my email, but I have a good reason.  
     Those of you who have been tracking CVS lately may have noticed a
new module go into the mix within the last week: spp_stream4.  I just
checked what I believe to be stable version of the new plugin into CVS,
so it's available and turned on by default in version 1.8-beta8 (build
29).  Stream4 is an entirely new preprocessor that preforms two
functions:

1) Stateful inspection of TCP sessions
2) TCP stream reassembly

I implemented stream4 out of the desire to have more robust stream
reassembly capabilities and the desire to defeat the latest "stateless
attacks" that have been coming out against Snort (c.f. stick and snot). 
Stream4 is written with the intent to let Snort be able to handle
performing stream reassembly for "enterprise class" users, people who
need to track and reassemble more than 256 streams simultaneously.  I've
optimized the code fairly extensively to be robust, stable, and fast. 
The testing and calculations I've performed lead me to be fairly
confident that stream4 can provide full stream reassembly for several
thousand simultaneous connections and stateful inspection for upwards of
64,000 simultaneous sessions.

Stream4 is a large and complex piece of code (almost 2000 lines) and
there are a lot of options associated with its runtime configuration, so
I'll go over them here.

preprocessor stream4: [noinspect], [keepstats], [timeout <seconds>],
[memcap <bytes>], [noalerts]

noinspect - disable stateful inspection

keepstats - record session summary information in <logdir>/session.log

timeout <seconds> - amount of time to keep an inactive stream in the
state table, sessions that are flushed will automatically be picked up
again if more activity is seen, default is 30 seconds

memcap <bytes> - number of bytes to set the memory cap at, if this limit
is exceeded stream4 will aggressively prune inactive sessions, default
is 8MB

noalerts - turns off alerts for stream events of note, such as evasive
RST packets, data on the SYN packet, and out of window sequence numbers

preprocessor stream4_reassemble: [clientonly], [serveronly], [noalerts],
[ports <portlist>]

clientonly - provide reassembly for the client side of a connection only

serveronly - provide reassembly for the server side of a connection only

noalerts - don't alert on events that may be insertion or evasion
attacks

ports <portlist> - a whitespace separated lit of ports to perform
reassembly for, "all" provides reassembly for all ports, "default"
provides reassembly for ports 21 23 25 53 80 110 111 143 and 513

Just setting the stream4 and stream4_reassemble directives without
arguments in the snort.conf file will set them up in their default
configuration 

[*] stream4 defaults:
Session timeout: 30 seconds
Session memory cap: 8388608 bytes
Stateful Inspection: ACTIVE
Stream Reassembly: INACTIVE
Stream Stats: INACTIVE
State Alerts: ACTIVE

[*] stream4_reassemble defaults: 
Reassemble client: ACTIVE
Reassemble server: INACTIVE
Reassemble ports: 21 23 25 53 80 143 110 111 513
Reassembly alerts: ACTIVE

There is a new command line switch that is used in concert with the
stream4 code, "-z".  The -z switch can take one of two arguments: "est"
and "all".  The "all" argument is the default if you don't specify
anything and tells Snort to alert normally.  If the -z switch is
specified with the "est" argument, Snort will only alert (for TCP
traffic) on streams that have been established via a three way handshake
or streams where cooperative bidirectional activity has been observed
(i.e. where some traffic went one way and something other than a RST or
FIN was seen going back to the originator).  With "-z est" turned on,
Snort completely ignores TCP-based stick/snot "attacks".

Besides that, there is another new command line switch, "-k".  The -k
switch allows you to modify the checksum tests that Snort performs in
the decoder stage.  It can take the following arguments:

"noip" - turn off IP checksum verification

"notcp" - turn off TCP checksum verification

"noudp" - turn off UDP checksums

"noicmp" - turn off ICMP checksums

"none" - turn off all checksums

Ths is done as a mechanism to let people have control over a subsystem
in Snort that can eat a lot of CPU cycles if you're not careful.  For
example, in many networks today, packets with bad IP checksums never
make it past the router or switch, and so IP checksum verification is a
waste of time for Snort on these networks.  Allowing people to turn that
subsystem off (or tune it) lets them get better performance without
having to make code tweaks.

I've also added some new "stuff" to the output stage of Snort recently. 
If you've noticed lately, there's some new data in the alerts that looks
something like "[1:523:1]" just before the alert message.  This is the
new Snort unique rule ID information that we've recently integrated into
the rule set that will allow much better correlation of Snort rules to
other vulnerability databases (CVE, Bugtraq, etc) and provide much
easier machine processing of Snort output as well.  

The fields in the block are [generator:sid:revision].  Generator is the
piece of Snort that generated the alert.  The generators.h file contains
the list of generators and the SIDs that they can generate, except for
the primary detection engine (generator 1).  The SID is the "Snort ID"
of the rule, a unique rule ID assigned to the rule by myself.  The
partitioning of the SID space is as follows:

0-100:       Reserved for Marty
101-999,999: Reserved for "official" Snort rules
1,000,000+:  User land SID space

Each there is a full set of SIDs available per generator, and the SID
space is a 32-bit unsigned integer inside Snort, so the values can go up
to 4 billion or so.

The last number in the block is the revision number of the rule.  As
rules change and their messages change (today's "SYN FIN scan" can
become tomorrow's "SCAN - SYN FIN") we want to be able to track the
revision history of the rules so that people who are doing machine
processing of Snort rules (like ARIS, incidents.org, etc) can
effectively perform their own back-end tasks.  The revision number
updating depends on the ruleset maintainer, but in general anytime you
tweak a rule, you should update the revision number.  If you modify a
rule that ships with Snort, please set the revision number to "0" so
that any machine processors will know you're running modified standard
rules.

Snort will also print out the transport protocol designation in
fast/syslog alerts now as well.

In output land, we have a new output plugin called spo_unified.  This
plugin is used for both alerting and logging, and writes alerts and logs
to two separate binary formatted files.  The alert file contains the
following information of interest:

Event data (generator, sid, rev, classification, priority,
event_reference)
timestamp
source IP
dest IP
source port
dest port
protocol
tcp flags (if any)

The log file contains the event data, flags that indicate the nature of
the stored packet (reassembled frag, etc), and the raw binary packet
(like in the log_tcpdump plugin).

Using this system we will be able to write Snort output to high speed
binary spool files and then use a second process to reformat the data
for consumption by the user.  I'm currently writing a program to perform
just that task called 'barnyard'.  Barnyard can read the binary file
formats of the unified alert and log spool files and present that data
to output plugins, which can then format the data for output to whatever
reporting and storage system is desired (database, XML, pcap, syslog,
etc).  Once this program is ready I will go into more detail on using
the integrated system to enable higer performance implementations of
Snort.

That's it for right now.  If you're feeling sufficiently motivated,
please check out Snort version 1.8-beta8 (build 29) from CVS and test it
out for performance, memory leaks, and stability.  If all goes well,
this is the codebase that 1.8 will be based on.


     -Marty


--
Martin Roesch
roesch () sourcefire com
http://www.sourcefire.com - http://www.snort.org

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
http://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users