RE: Tool for large log file reporting? (was: RE: WebSens e)

["Desai, Ashish" <Ashish.Desai () fmr com>]

you could try renaming the tables periodically(once a week). 
That way you don't hamper the proxy that's doing inserts. 

BTW, we ended up writing perl/shell script code as we get approx 8Gb
of proxy logs a day! (This is not a joke) 
We gzip them up. However its hurts us, as  uncompressing them in memory
takes a while. 


Ashish

> -----Original Message-----
> From: Behm, Jeffrey L. [SMTP:BehmJL () bvsg com]
> Sent: Wednesday, September 20, 2000 9:55 AM
> To:   'Joe Ippolito '; 'firewall-wizards () nfr net '
> Subject:      RE: Tool for large log file reporting? (was: RE: [fw-wiz]
> WebSens e)
> Thanks for the response!
>
> We tried Sql server, agreed, it is easy to set up, slows down the proxy a
> bit, etc., but the show stopper for us was when we tried to remove the
> older
> entries from the db. At that point, the Proxy server hung until the
> (pseudo-sql command follows) "delete from x where date <  [today - 7 days
> (or whatever)]" would finish. At least our prognosis was that it would
> continue once the query finished, but we couldn't let the query run longer
> than 10 minutes, as all http traffic stopped through the proxy server. We
> tried creating indexes on the logdate/time field, no difference. Perhaps
> some setting in Sql Server would do the trick, but of course, this is the
> FW
> group, not Sql-server, so no flames please.
>
> In short, The sql server functionality seemed to be the holy grail, as the
> queries we ran came back in seconds instead of minutes. All seemed great
> until we needed to truncate the older, no-longer-needed data.
>
> The amount of data this client generates _just in MSProxy log files_ is
> incredible (400MB/day and growing) and we haven't found anything that will
> resonably manage (i.e. inserts, queries, _and_ deletes) the larger amount
> of
> data.
>
> Thoughts?
>
> Jeff
>
> _______________________________________________
> Firewall-wizards mailing list
> Firewall-wizards () nfr net
> http://www.nfr.net/mailman/listinfo/firewall-wizards

_______________________________________________
Firewall-wizards mailing list
Firewall-wizards () nfr net
http://www.nfr.net/mailman/listinfo/firewall-wizards