Re: parsing logs ultra-fast inline

["Marcus J. Ranum" <mjr () ranum com>]

Brian Loe wrote:
> Picking on me again already! Sheesh...

Nope, actually I'm picking on a superclass of companies and individuals
among whom you are an individual member. It's nothing personal! :)

> Still have no idea, really, how to
> configure syslog-ng and write a perl script as described - but I'll
> fumble through it.

Googling for "parse pix log script" returns me 380,000
possible references and the first 3 look immediately promising.
Googling for "parse AIX log script" returns me 314,000
possible references and the first page has about 4 items that
look promising.
etc.

> Question: Better to do it inline or off-line (for starters anyway)?

For testing and getting things working, I'd say to collect
the data to a hard disk then use a secondary process that
runs against the data on the disk. Once you have all that
working then you can put things in place to rotate the data
out when you're done with it.

A typical approach to doing this would be to use syslog-ng
to separate the log messages into the different apps that
you want to deal with and then deal with them each in
separate scripts that assess that app's logfiles. Note that
syslog-ng is not exactly "lightweight" but as long as you
can resist the urge to try to stick this stuff into a database
you will probably be fine.

> I figure it would require less
> overhead to analyze individual files by type (and therefore similar
> messages) 

Yup! Basically, you're talking about using syslog-ng as that
first-level of your parse tree that breaks things into sub-branches
by application. Of course syslog-ng is a gigantic sledgehammer
of a chunk of software to do something that simple, but it's
easy and flexible, etc.

> Second question: Hasn't anyone else ever written these scripts? You
> would think they'd be pretty widely available

There's this awesome website called www.google.com you
really ought to check out!!! It's for finding things on the internet!
And it's free and it's really fast!

mjr. 

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards