No Budget Static Log Analysis

[Stephen Mullins <steve.mullins.work () gmail com>]

List,

I am involved in a project that performs analysis of a static set of
logs provided in ASCII/plain text format looking for signs of
malicious activity using lists of known "indicators" (IP addresses,
domain names, user agent strings, etc.).  The logs can be from any
number of devices (firewalls, web proxies, DNS servers, etc.) and can
be formatted/delimited in whatever format is native to the device that
generated the logs.  The smallest set of data received thus far was
200 gigabytes and the largest 2.3 terabytes.

So far I've be slogging through the logs with *nix tools, shell
scripts, perl, etc.  But this takes a lot of time, doesn't scale, and
requires custom work for each new set of logs.

I have no budget to procure a commercial solution like Splunk,
ArcSight, Saw Mill, etc.  I do have a couple of relatively decent
servers and some Network Area Storage devices to work with.  I've done
some research into "free" solutions and come up with the following
short list:

LogStash
GrayLog2
log.io
Log Parser (Windows)

Has anyone tackled a similar problem in the past?  If so, do you have
any advice or recommendations?  Of the "free" solutions researched the
first one I am inclined to try is LogStash.

Thanks,

Steve Mullins

------------------------------------------------------------------------
Securing Apache Web Server with thawte Digital Certificate
In this guide we examine the importance of Apache-SSL and who needs an SSL certificate.  We look at how SSL works, how 
it benefits your company and how your customers can tell if a site is secure. You will find out how to test, purchase, 
install and use a thawte Digital Certificate on your Apache web server. Throughout, best practices for set-up are 
highlighted to help you ensure efficient ongoing management of your encryption keys and digital certificates.

http://www.dinclinx.com/Redirect.aspx?36;4175;25;1371;0;5;946;e13b6be442f727d1
------------------------------------------------------------------------