nanog mailing list archives
RE: yahoo crawlers hammering us
From: Nathan Eisenberg <nathan () atlasnetworks us>
Date: Wed, 8 Sep 2010 10:24:38 +0000
Possibly because that other user is who the customer pays have their content delivered to?
Customers don't want to deliver their content to search engines? That seems silly. http://www.last.fm/robots.txt (Note the final 3 disallow lines...)
Current thread:
- yahoo crawlers hammering us Ken Chase (Sep 07)
- Re: yahoo crawlers hammering us Leslie (Sep 07)
- Re: yahoo crawlers hammering us Harry Strongburg (Sep 07)
- Re: yahoo crawlers hammering us Matthew Petach (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- RE: yahoo crawlers hammering us Nathan Eisenberg (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- Re: yahoo crawlers hammering us Valdis . Kletnieks (Sep 08)
- Re: yahoo crawlers hammering us Bruce Williams (Sep 08)
- Re: yahoo crawlers hammering us Ken Chase (Sep 08)
- Re: yahoo crawlers hammering us Matthew Petach (Sep 08)