Firewall Wizards mailing list archives
Re: Handling large log files
From: Nate Hausrath <hausrath () gmail com>
Date: Wed, 6 May 2009 09:54:26 -0400
First, thanks for the great responses! Aside from the fact that we need a beefier system (2x P3 1.4 GHz, 3 GB RAM, RAID-5... ouch), it looks like I have a lot of work to do. Also, thanks for providing some idea of the specs I will need to use for a central log server. I believe our goal is to have around 300 servers sending logs (most of them should be less chatty than the current ones). If you don't mind me asking, roughly how many servers should I expect to have generate 1 GB of logs? I realize there really isn't an accurate answer here, but I'm trying to get a rough ballpark figure.
Marcin wrote: - see if the architecture can be improved. Can you use multiple log servers? Is there a logical way of segmenting the log traffic - OS to box 1, db transactions to box 2, etc.? Post to the project's mailing list, there should be people who use it for larger installations, and willing/able to provide specific suggestions.
I'll see if this is an option. Along these lines, I'd eventually like to be able to turn log messages into events and be able to correlate them with other messages, IDS alerts, etc. I think that once I compress the duplicates, and get rid of a lot of noise, I could forward the results to an OSSIM box and use it for correlation, alerts, etc.
Paul wrote: What are you trying to achieve with your log analysis, as in, what sort of actions would the review of this daily log report trigger? Would you want to or should you move to a model where search/analysis is happening in near-real time instead of once daily? That's going to be helpful in knowing what kind of solution you should be looking at. Also, while it's overpowering your logcheck scripts, 5GB/day of log data is nothing when you're talking about firewall logs. PaulM
We are primarily looking for security related events. Real time analysis/reporting of events is an eventual goal, but that seems a lot more difficult to do in some regards. Initially, I'd like to at least have a summary I can look at daily (probably along the lines of what David posted below) and then I could transition to more real-time analysis. Does that sound reasonable?
David wrote: I don't like the idea of filtering out messages completely, the number of times that an otherwise 'unintersting' message shows up can be significant (if the number of requests for a web image per day suddenly jumps to 100 times what it was before, that's a significant thing to know)
Duly noted. Thanks!
the key is to categorize and summarize the data. I have not found a good commercial tool to do this job (there are good tools for drilling down and querying the logs), the task of summarizing the data is just too site specific. I currently get 40-80G of logs per day and have a nightly process that summarizes them.
This is good to know as well. I'd like to avoid commercial tools if possible to save money (although Splunk seems pretty darn useful).
*Solid plan of attack from David*
Thanks for all the great information from everyone. I'll be jumping into this today! -Nate _______________________________________________ firewall-wizards mailing list firewall-wizards () listserv icsalabs com https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards
Current thread:
- Handling large log files Nate Hausrath (May 05)
- Re: Handling large log files Marcin Antkiewicz (May 05)
- Re: Handling large log files Nate Hausrath (May 06)
- Re: Handling large log files david (May 06)
- Re: Handling large log files Marcus J. Ranum (May 06)
- Re: Handling large log files Nate Hausrath (May 06)
- Re: Handling large log files Paul Melson (May 05)
- Re: Handling large log files david (May 06)
- Re: Handling large log files Swaminathan, Gayathri (May 06)
- Re: Handling large log files hugh.fraser (May 07)
- Re: Handling large log files sai (May 08)
- Re: Handling large log files Nate Hausrath (May 08)
- Re: Handling large log files Gyöngyösi Péter (May 11)
- Re: Handling large log files Marcin Antkiewicz (May 05)