Re: DOS From "inktomi.com"?

[Nexus <nexus () PATROL I-WAY CO UK>]

Hi folks,

----- Original Message -----
From: PARKIN, MICHAEL M (PBI) <mparkin () PBI NET>
To: <INCIDENTS () SECURITYFOCUS COM>
Sent: Wednesday, August 23, 2000 12:55 AM
Subject: Re: DOS From "inktomi.com"?


> I've seen repeated connections from inktomi's web crawler bots, but never
at
> a level that affected performance.  As I remember, the inktomi bots do
> respect the robots.txt protocol and will go away if your system tells them
> to.

That has not been my experience, my test lab web server came to their
crawler's attention once, almost continuously hammering away with requests
for robots.txt.
I had removed that file (as well as a shedload of dll's from that IIS box)
when I built it but despite the continual 404's, the crawler kept hammering
away, doing a nice job of filling the logs.   They earned the first 'reject'
rule I had implemented on the firewall ;-)   And it still didn't go away...
persistant little bugger, I'll say that !
I seem to recall having to go upstream for a solution as well.

Regards,
        JJ