RE: No Budget Static Log Analysis

[Dave Kleiman <dave () davekleiman com>]

Hello Steve,

There are a wide variety of excellent log  tools, Logstash, Splunk. Sawmill, etc. as previously mentioned.  If you 
decide to use Log Parser you can create a toolbox of scripts and have them ready:

Some like:

SELECT time,
       date,
       cs-uri-stem,
       c-ip 
INTO ISAlertByIP.csv
FROM ex*.log
WHERE
       (cs-uri-stem LIKE '%ALERT%')
       AND
       (IPV4_TO_INT(c-ip) BETWEEN IPV4_TO_INT('192.168.0.0') AND IPV4_TO_INT('192.168.255.255'))
GROUP BY c-ip,time,date,cs-uri-stem

Save as IISAlertByIP.sql  and call it from the logparser dir.  Logparser file: IISAlertByIP.sql  


I have a bunch of scripts you might find handy on my site:  
http://www.computerforensicsllc.com/computer-forensics-expert-florida-miami-palm-beach-lauderdale-dave-kleiman-forensic-training-files/

Feel free to ping me with any questions,

Respectfully,

Dave Kleiman - http://www.ComputerForensicsLLC.com - http://www.computerforensicsexpertwitnesses.com/

4371 Northlake Blvd #314
Palm Beach Gardens, FL 33410
561.310.8801 






-----Original Message-----
From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Stephen Mullins
Sent: Wednesday, July 25, 2012 15:34
To: security-basics () securityfocus com
Subject: No Budget Static Log Analysis

List,

I am involved in a project that performs analysis of a static set of logs provided in ASCII/plain text format looking 
for signs of malicious activity using lists of known "indicators" (IP addresses, domain names, user agent strings, 
etc.).  The logs can be from any number of devices (firewalls, web proxies, DNS servers, etc.) and can be 
formatted/delimited in whatever format is native to the device that generated the logs.  The smallest set of data 
received thus far was
200 gigabytes and the largest 2.3 terabytes.

So far I've be slogging through the logs with *nix tools, shell scripts, perl, etc.  But this takes a lot of time, 
doesn't scale, and requires custom work for each new set of logs.

I have no budget to procure a commercial solution like Splunk, ArcSight, Saw Mill, etc.  I do have a couple of 
relatively decent servers and some Network Area Storage devices to work with.  I've done some research into "free" 
solutions and come up with the following short list:

LogStash
GrayLog2
log.io
Log Parser (Windows)

Has anyone tackled a similar problem in the past?  If so, do you have any advice or recommendations?  Of the "free" 
solutions researched the first one I am inclined to try is LogStash.

Thanks,

Steve Mullins

------------------------------------------------------------------------
Securing Apache Web Server with thawte Digital Certificate In this guide we examine the importance of Apache-SSL and 
who needs an SSL certificate.  We look at how SSL works, how it benefits your company and how your customers can tell 
if a site is secure. You will find out how to test, purchase, install and use a thawte Digital Certificate on your 
Apache web server. Throughout, best practices for set-up are highlighted to help you ensure efficient ongoing 
management of your encryption keys and digital certificates.

http://www.dinclinx.com/Redirect.aspx?36;4175;25;1371;0;5;946;e13b6be442f727d1
------------------------------------------------------------------------