nanog mailing list archives
Re: yahoo crawlers hammering us
From: Ken Chase <ken () sizone org>
Date: Wed, 8 Sep 2010 12:20:07 -0400
On Wed, Sep 08, 2010 at 12:04:07AM -0700, Matthew Petach said: >I *am* curious--what makes it any worse for a search engine like Google >to fetch the file than any other random user on the Internet? In either case, >the machine doing the fetch isn't going to rate-limit the fetch, so >you're likely >to see the same impact on the machine, and on the bandwidth. I think that the difference is that there's a way to get to Yahoo and ask them WTF. Whereas the guy who mass downloads your site with a script in 2 hrs you have no recourse to (modulo well funded banks dispatching squads with baseball bats to resolve hacking incidents). I also expect that Yahoo's behaviour is driven by policy, not random assholishness (I hope :), and therefore I should expect such incidents often. I also expect whinging on nanog might get me some visiblity into said policy and leverage to change it! </dream> /kc -- Ken Chase - ken () heavycomputing ca - +1 416 897 6284 - Toronto CANADA Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.
Current thread:
- yahoo crawlers hammering us Ken Chase (Sep 07)
- Re: yahoo crawlers hammering us Leslie (Sep 07)
- Re: yahoo crawlers hammering us Harry Strongburg (Sep 07)
- Re: yahoo crawlers hammering us Matthew Petach (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- RE: yahoo crawlers hammering us Nathan Eisenberg (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- Re: yahoo crawlers hammering us Valdis . Kletnieks (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- Re: yahoo crawlers hammering us Ken Chase (Sep 08)
- Re: yahoo crawlers hammering us Matthew Petach (Sep 08)