Firewall Wizards mailing list archives
Re: Handling large log files
From: Paul Melson <pmelson () gmail com>
Date: Tue, 5 May 2009 23:38:49 -0400
On Tue, May 5, 2009 at 6:41 PM, Nate Hausrath <hausrath () gmail com> wrote:
Hello everyone, I have a central log server set up in our environment that would receive around 200-300 MB of messages per day from various devices (switches, routers, firewalls, etc). With this volume, logcheck was able to effectively parse the files and send out a nice email. Now, however, the volume has increased to around 3-5 GB per day and will continue growing as we add more systems. Unfortunately, the old logcheck solution now spends hours trying to parse the logs, and even if it finishes, it will generate an email that is too big to send.
[...][
Are there other solutions that would be better suited to log volumes like this? Should I look at commercial products? Any comments/criticisms/suggestions would be greatly appreciated! Please let me know if I need to provide more information. Again, my lack of experience in this area causes me hesitant to make a solid decision without asking for some guidance first. I don't want to spend a lot of time going in one direction, only to find that I was completely wrong.
What are you trying to achieve with your log analysis, as in, what sort of actions would the review of this daily log report trigger? Would you want to or should you move to a model where search/analysis is happening in near-real time instead of once daily? That's going to be helpful in knowing what kind of solution you should be looking at. Also, while it's overpowering your logcheck scripts, 5GB/day of log data is nothing when you're talking about firewall logs. PaulM _______________________________________________ firewall-wizards mailing list firewall-wizards () listserv icsalabs com https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards
Current thread:
- Handling large log files Nate Hausrath (May 05)
- Re: Handling large log files Marcin Antkiewicz (May 05)
- Re: Handling large log files Nate Hausrath (May 06)
- Re: Handling large log files david (May 06)
- Re: Handling large log files Marcus J. Ranum (May 06)
- Re: Handling large log files Nate Hausrath (May 06)
- Re: Handling large log files Paul Melson (May 05)
- Re: Handling large log files david (May 06)
- Re: Handling large log files Swaminathan, Gayathri (May 06)
- Re: Handling large log files hugh.fraser (May 07)
- Re: Handling large log files sai (May 08)
- Re: Handling large log files Nate Hausrath (May 08)
- Re: Handling large log files Gyöngyösi Péter (May 11)
- Re: Handling large log files Marcin Antkiewicz (May 05)