Re: Handling large log files

[Gyöngyösi Péter <gyp () balabit hu>]

(Disclaimer: I work for BalaBit, the company behind syslog-ng.)

Nate Hausrath wrote:
> Hello everyone,
>
> I have a central log server set up in our environment that would
> receive around 200-300 MB of messages per day from various devices
> (switches, routers, firewalls, etc).  With this volume, logcheck was
> able to effectively parse the files and send out a nice email.  Now,
> however, the volume has increased to around 3-5 GB per day and will
> continue growing as we add more systems.  Unfortunately, the old
> logcheck solution now spends hours trying to parse the logs, and even
> if it finishes, it will generate an email that is too big to send.
>   
The others have given lots of useful tips about log handling, but if
you're just having perfomance issues with logcheck, you should have a
look at the db-parser feature in the new syslog-ng 3.0.

The best places to find out more about it are these blog posts:

http://marci.blogs.balabit.com/2009/04/db-parser-high-speed-log-message-parser.html
http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog-ng-db.html
http://bazsi.blogs.balabit.com/2008/10/syslog-ng-message-parsing.html

It's able to handle (that means, classify based on log message contents,
filter based on this classification and store or forward) this kind of
traffic on commodity hardware. A ready-to-use pattern database converted
from logcheck's regexp list and for Cisco PIX messages can be downloaded
from the website and it's quite easy to write your own rules (the blog
posts mentioned above contain good examples).


Peter

_______________________________________________
firewall-wizards mailing list
firewall-wizards () listserv icsalabs com
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards