Firewall Wizards mailing list archives
RE: Tool for large log file reporting? (was: RE: WebSens e)
From: "Desai, Ashish" <Ashish.Desai () fmr com>
Date: Fri, 22 Sep 2000 15:47:13 -0400
you could try renaming the tables periodically(once a week). That way you don't hamper the proxy that's doing inserts. BTW, we ended up writing perl/shell script code as we get approx 8Gb of proxy logs a day! (This is not a joke) We gzip them up. However its hurts us, as uncompressing them in memory takes a while. Ashish
-----Original Message----- From: Behm, Jeffrey L. [SMTP:BehmJL () bvsg com] Sent: Wednesday, September 20, 2000 9:55 AM To: 'Joe Ippolito '; 'firewall-wizards () nfr net ' Subject: RE: Tool for large log file reporting? (was: RE: [fw-wiz] WebSens e) Thanks for the response! We tried Sql server, agreed, it is easy to set up, slows down the proxy a bit, etc., but the show stopper for us was when we tried to remove the older entries from the db. At that point, the Proxy server hung until the (pseudo-sql command follows) "delete from x where date < [today - 7 days (or whatever)]" would finish. At least our prognosis was that it would continue once the query finished, but we couldn't let the query run longer than 10 minutes, as all http traffic stopped through the proxy server. We tried creating indexes on the logdate/time field, no difference. Perhaps some setting in Sql Server would do the trick, but of course, this is the FW group, not Sql-server, so no flames please. In short, The sql server functionality seemed to be the holy grail, as the queries we ran came back in seconds instead of minutes. All seemed great until we needed to truncate the older, no-longer-needed data. The amount of data this client generates _just in MSProxy log files_ is incredible (400MB/day and growing) and we haven't found anything that will resonably manage (i.e. inserts, queries, _and_ deletes) the larger amount of data. Thoughts? Jeff _______________________________________________ Firewall-wizards mailing list Firewall-wizards () nfr net http://www.nfr.net/mailman/listinfo/firewall-wizards
_______________________________________________ Firewall-wizards mailing list Firewall-wizards () nfr net http://www.nfr.net/mailman/listinfo/firewall-wizards
Current thread:
- RE: Tool for large log file reporting? (was: RE: WebSens e) Desai, Ashish (Sep 22)