Re: parsing logs ultra-fast inline

[John Adams <jna+dated+1139879645.1e269a () retina net>]

I wrote a PIX log analysis tool awhile back which isn't extremely good, 
but it might give you a good start. It sucks the log file into a mysql 
databse, and then allows you to use a PHP based interface to browse it and 
get statistics.

http://www.retina.net/~jna/pixie/

It's old, though. I haven't worked on it in quite some time.
-j

On Wed, 8 Feb 2006, Marcus J. Ranum wrote:

> Brian Loe wrote:
> > Picking on me again already! Sheesh...
>
> Nope, actually I'm picking on a superclass of companies and individuals
> among whom you are an individual member. It's nothing personal! :)
>
> > Still have no idea, really, how to
> > configure syslog-ng and write a perl script as described - but I'll
> > fumble through it.
>
> Googling for "parse pix log script" returns me 380,000
> possible references and the first 3 look immediately promising.
> Googling for "parse AIX log script" returns me 314,000
> possible references and the first page has about 4 items that
> look promising.
> etc.
>
> > Question: Better to do it inline or off-line (for starters anyway)?
>
> For testing and getting things working, I'd say to collect
> the data to a hard disk then use a secondary process that
> runs against the data on the disk. Once you have all that
> working then you can put things in place to rotate the data
> out when you're done with it.
>
> A typical approach to doing this would be to use syslog-ng
> to separate the log messages into the different apps that
> you want to deal with and then deal with them each in
> separate scripts that assess that app's logfiles. Note that
> syslog-ng is not exactly "lightweight" but as long as you
> can resist the urge to try to stick this stuff into a database
> you will probably be fine.
>
> > I figure it would require less
> > overhead to analyze individual files by type (and therefore similar
> > messages)
>
> Yup! Basically, you're talking about using syslog-ng as that
> first-level of your parse tree that breaks things into sub-branches
> by application. Of course syslog-ng is a gigantic sledgehammer
> of a chunk of software to do something that simple, but it's
> easy and flexible, etc.
>
> > Second question: Hasn't anyone else ever written these scripts? You
> > would think they'd be pretty widely available
>
> There's this awesome website called www.google.com you
> really ought to check out!!! It's for finding things on the internet!
> And it's free and it's really fast!
>
> mjr.
>
> _______________________________________________
> firewall-wizards mailing list
> firewall-wizards () honor icsalabs com
> http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

--
J. Adams                                        http://www.retina.net/~jna

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards