Firewall Wizards mailing list archives
Re: parsing logs ultra-fast inline
From: "Marcus J. Ranum" <mjr () ranum com>
Date: Tue, 07 Feb 2006 14:52:38 -0500
Chuck Swiger wrote:
Without fighting too hard, many log analysis tools for things like webserver or squid or firewall rules seem to process ~10K lines or events per second, which works out to a gigabyte every ten minutes or so, whereas other tools seem hopelessly incapable of handling large data sets.
I think it's because a lot of webserver analysis tools are designed to rip through the data and provide statistical summaries and sorted hit-lists, whereas the security-oriented log processing tools are aimed at audit functions. Since the security problem is less well-bounded than "show me the top 50 pages on my site!" the designers of those systems often reach for the biggest hammer in their toolbox and stuff everything into a SQL database, which promptly falls over, leading them to conclude "it can't be done." As we discussed last week; if you put some thought into figuring out what you want to get from your log analysis, you can do it at extremely high speeds, pre-compute all the running totals you need, cache views into the data-sets as necessary, etc. But I used that evil phrase "put some thought into..." and we know that most IT managers would rather buy a $20,000 thingamajig than "put some thought into..." anything. mjr. _______________________________________________ firewall-wizards mailing list firewall-wizards () honor icsalabs com http://honor.icsalabs.com/mailman/listinfo/firewall-wizards
Current thread:
- parsing logs ultra-fast inline Marcus J. Ranum (Feb 02)
- Re: parsing logs ultra-fast inline Chuck Swiger (Feb 02)
- RE: parsing logs ultra-fast inline Tina Bird (Feb 02)
- Re: parsing logs ultra-fast inline Adrian Grigorof (Feb 03)
- Re: parsing logs ultra-fast inline Chuck Swiger (Feb 07)
- Re: parsing logs ultra-fast inline Marcus J. Ranum (Feb 07)
- Re: parsing logs ultra-fast inline Brian Loe (Feb 08)
- Message not available
- Re: parsing logs ultra-fast inline Marcus J. Ranum (Feb 08)
- Re: parsing logs ultra-fast inline John Adams (Feb 09)
- Re: parsing logs ultra-fast inline Adrian Grigorof (Feb 03)
- RE: parsing logs ultra-fast inline Paul Melson (Feb 15)
- Re: parsing logs ultra-fast inline Anton Chuvakin (Feb 07)
- Re: parsing logs ultra-fast inline Adrian Grigorof (Feb 07)
- Re: parsing logs ultra-fast inline Patrick M. Hausen (Feb 07)
- RE: parsing logs ultra-fast inline Tina Bird (Feb 07)
- <Possible follow-ups>
- RE: parsing logs ultra-fast inline Behm, Jeffrey L. (Feb 08)