Re: Views and Correlation in Intrusion Detection

["SecurIT Informatique Inc." <securit () iquebec com>]

At 01:32 PM 17/06/2003, Blake Matheny wrote:

> Two areas that I have recently been doing research in, are views and their
> connection to correlation techniques.

>  There are really several issues here. First of all, a tremendous amount of
> time is being spent, trying to correlate all the relevant information. This is
> something that _can_ be automated. Second, the applications logs may not be
> trustworthy. Third, and to me, most importantly, is the fact that this is such
> a 'basic' thing that people using ID systems have to do, and there is no piece
> of software yet that does this.

Totally agree with you. I've been working on the same problems myself 
during the last 1-2 years.

>  So something we have been working on, is a system to deal with this basic
> type of scenario. This will entail data transformations into an intermediary
> language, an event description language, offline state analysis and several
> other components (there is more information at http://www.nongnu.org/babe/).
> If you spend some time thinking about everything involved to do this in a
> scalable fashion, it's an enormous task (I said basic, not simple). What I am
> finding frustrating, is that much of the base research has not yet even been
> done. Much of the research that has been done, is either too primitive or too
> impractical to be implemented. Is this due to the infancy and immaturity of
> the field, do people not see this as being feasible and therefor aren't
> spending the research time, or is this simply too far down the line? In any
> case, feedback welcome. Thanks.

Believe me, I do see this as being feasible, but I think that this topic is 
probably still in its infancy stages, which is why there is not much work 
published around this.  As I said, I worked on these problems myself, and 
have released my tools (with documentation, which also covers the theory 
nehind it) approx 3 weeks ago, so maybe you have missed it.  My 
implementation works on Win NT/2K/XP platforms, but could very well work 
with logs coming from *NIX systems if you can get them to a UNC share on a 
Windows box.  You can download my tools at http://securit.iquebec.com/.

I looked at your doc and flow chart, and we have roughly the same approach 
to the problem, although we have some fondamental development differences 
(for example, my tools don't have DB support for log storage).  To gather 
application and Event Viewer log files to a central location, I have made 
LogAgent (now version 4.0, available in Open Source and Pro versions).  So 
with it, you can monitor-and-centralize on the fly log files for antivirus, 
personnal firewall, main firewall if applicable, NIDS like Snort, 
etc...  LogAgent 4.0 also comes with a HIDS program that checks for file 
system integrity, and an Alternate Data Stream scanner (ADS are a way to 
hide files on a Windows system).  It also generates forensics-related data 
like running services, startup conf, open shares, that is then matched 
against a list of allowed ressources.  I have also developped a command 
prompt (cmd.exe) logger, ComLog (now version 1.05, OS and Pro), so it is 
now possible to keep a log of hacking incidents where commands were passed 
this way (with a cryptcat tunnel, for example).

Now that all these logs from various applications are gathered from all 
over the network to a central place, the challenge is to analyse them in an 
efficient way.  You say you want to work on data output also, but you have 
no screenshot or description of what you have in mind.  Any interface 
ideas?  I tried to make something new.  To monitor and analyse these logs, 
I made the console program LogIDS 1.0 (av. in OS and Pro), which will 
monitor these log files for you and apply rules in order to sift through 
them and select what is worthy of attention.  The interface is a 
representation of your network map, where each node have its own monitoring 
window, and icons can be specified to illustrate the event reported in the 
log.  LogIDS is very flexible, you get to define the fields of every log 
file you include, and apply rules using these fileds definition.  The flow 
of data is somewhat similar to what you describe on your flow chart 
http://www.nongnu.org/babe/papers/data_flow.png.

I have to admit that it is not perfect tough, as you mentionned this is 
still a very new topic, and far from being as mainstream as, say, firewalls 
technology.  Feel free to look at it to get ideas/make suggestions, it's 
nice to see that something similar is growing in the *NIX world.  I already 
have some plans for future releases, like distributed analysis, allow for 
new modules to be added easily to LogAgent, some performance tweaks, an 
enhanced ruleset, and a couple other nice features.  Not to be expected 
before at least a few months.

Hope this helps!

Adam Richard, aka Floydman
SécurIT Informatique Inc.

> Cheers,
>
> -Blake
>
> --
> Blake Matheny           "... one of the main causes of the fall of the
> bmatheny () mkfifo net      Roman Empire was that, lacking zero, they had
> http://www.mkfifo.net    no way to indicate successful termination of
> http://ovmj.org/GNUnet/  their C programs." --Robert Firth

-------------------------------------------------------------------------------
Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the 
world's premier technical IT security event! 10 tracks, 15 training sessions, 
1,800 delegates from 30 nations including all of the top experts, from CSO's to 
"underground" security specialists.  See for yourself what the buzz is about!  
Early-bird registration ends July 3.  This event will sell out. www.blackhat.com
-------------------------------------------------------------------------------