Re: parsing logs ultra-fast inline

[Brian Loe <knobdy () gmail com>]

On 2/7/06, Marcus J. Ranum <mjr () ranum com> wrote:

> I think it's because a lot of webserver analysis tools are designed to
> rip through the data and provide statistical summaries and sorted
> hit-lists, whereas the security-oriented log processing tools are
> aimed at audit functions. Since the security problem is less well-bounded
> than "show me the top 50 pages on my site!" the designers of those
> systems often reach for the biggest hammer in their toolbox and
> stuff everything into a SQL database, which promptly falls over,
> leading them to conclude "it can't be done."

Picking on me again already! Sheesh...

Okay, so I've gotten them to order some more ram and drive space for
my linux box. Going to start very small with one or two of our
internal PIXen...see how it goes. Still have no idea, really, how to
configure syslog-ng and write a perl script as described - but I'll
fumble through it.

Question: Better to do it inline or off-line (for starters anyway)? I
will turn it on for a day or so just to collect the first set of data
to begin writing the scripts with. Pretty sure syslog-ng will allow me
to create logs based on sources, so I figure it would require less
overhead to analyze individual files by type (and therefore similar
messages) like all of the PIXes, all of the ??Routers, AIX boxes,
etc.. I hate thinking about writing scripts for a month per device
type, but...

Second question: Hasn't anyone else ever written these scripts? You
would think they'd be pretty widely available - especially for things
like a PIX or 2600 or AIX. I mean, yes they're site specific but if
you know all of the errors/messages a PIX can provide (someone said
26k or so?) then the "meat" of a script could be generic enough...the
most common messages aren't likely to differ by much from site to
site...place your IPs/whatever in and run... or start to run...??
_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards