Re: SIM and stuff

[Anton Chuvakin <anton () chuvakin org>]

All,

Dave - sorry for cluttering your list, but this is fun :-)

> For most correlation tasks, you can only use the pieces of one log format
> that are present in other log formats.  Take the example of a firewall log
> entry and an IDS log entry.  You extract 6 pieces of information from the
> IDS event (timestamp, source addr, source port, dest addr, dest port,
> protocol) and 7 pieces of data from the firewall log (the same 6 as the IDS
> event plus whether the firewall logged a permit or a deny for that event)
> and perform a 1-to-1 match.  The 6 values are numeric, easy to search for,
> sort, and match.  Even time offset sloughing is easy to do at this level
> (hint to  SIM vendors: pay attention, most of you don't do this but should).
> That's one bucketing expression for the IDS logs, one bucketing expression
> for the firewall logs, and one more expression to parse the firewall log for
> allow/deny.  This should be a very small amount of code.

"one bucketing expression for the firewall logs"
OMG, this is so naive :-) Admittedly, a connection denied/allowed
might for some firewalls, given some luck and planet alignment, be
covered by one regex. But how about all hundreds of other messages,
such as 'failover failed', 'memory overflowed', 'VPN connection
established', etc. They do not even have the above "6 magic fields."
Some firewalls count up to 1100 distinct messages...

> > OSSIM seems stagnant; I haven't seem any new features for quite some time.
> And, just as
> > mentioned by Dave Aitel, device support is a big issue for adoption. If
> you read the PIX 6.1
> > logs just fine, there is nothing that tell you that you will deal with PIX
> 6.2 logs just as
> > fine... Thus, there is a good reason that many SIM softwares cost a bit
> more than the above
> > number :-)
>
> I agree that there's a lot of work to be done in terms of device support
> before a SIM can pick up market share, but this is one specific thing that
> an open source project should be able to make huge advances at.  What's
> required is modular parsers, perhaps based on Perl regex or awk (or
> something standard and easy to learn).  Then people can write their own and
> contribute them back to the project.  Some commercial products offer this
> already.

Well, supposedly a majority of Snort sigs and Nessus checks (given two
major open source [well, not quite, in case of Nessus] security
projects) are supposeduly written by a relatively small group of
people. It is actually kinda fun to do it! But, writing regex parsers
for an ever increasing number of log messages is not nearly as much
fun - if it were fun, it would be done by now :-) Thus, I doubt that
community would do a lot here...

> Where the large SIM vendors will continue to succeed in justifying big price
> tags, and where open source SIM projects are likely to fail, are in the
> large lexicons of expressions necessary to perform event categorization and
> threat analysis.  It's easy to extract the information necessary to match
> logs from different sources.  To take that information and determine its
> relevance is a very different, highly subjective undertaking.

Agreed on the categorization side. What the world really needs :-) is
a standard event classification that commercial and free products can
use... Yeah, I know, wishful thinking :-)

And, no, its not always easy to even extract the info; see this
discussion, for example:
http://lists.shmoo.com/pipermail/loganalysis/2005-December/002906.html

BTW, this discussion actually belongs on the loganalysis list
(http://lists.shmoo.com/mailman/listinfo/loganalysis)

Best,
--
Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
     http://www.chuvakin.org
 http://www.securitywarrior.com