nanog mailing list archives

RE: yahoo crawlers hammering us


From: Nathan Eisenberg <nathan () atlasnetworks us>
Date: Wed, 8 Sep 2010 10:24:38 +0000

Possibly because that other user is who the customer pays have their content
delivered to?

Customers don't want to deliver their content to search engines?  That seems silly.
 
http://www.last.fm/robots.txt (Note the final 3 disallow lines...)





Current thread: