Interesting People mailing list archives
IP: a bit more Every 30 minute spiders
From: Dave Farber <farber () cis upenn edu>
Date: Sun, 03 Nov 1996 09:53:19 -0500
Date: Sat, 02 Nov 1996 17:31:27 -0500 To: farber () central cis upenn edu (David Farber) From: Jock Gill <jgill () penfield-gill com> Subject: Every 30 minute spiders Dave, Brad Cox wrote:
Apart from the "few weeks old" bit, this sounds routine and innocuous. Search engines (Yahoo, AltaVista) routinely rebuild their indices with spiders just as you describe.
These new spiders are visiting some locations twice per hour and snarfing every page each time they visit. This is not required for index re-building. Perhaps it is time to think about micro-payments for transactions between machines, cyber-serfs, and cyber-serfs and machines. Perhaps to only think of micro-payments when humans are involved is too human-centric. Perhaps also a system such as Agoric Inc.'s Digital Silk Road would be an appropriate response - or part of one? See www.agorics.com/%7Eagorics/ Send your spiders as often as you see fit, but pay as you go. Turn spider visits into profitable exercises. Financial incentives and penalties have worked well in the history of the West. They form a very interesting constraint system. Regards, Jock ____________________________________________________________________________ Jock Gill Penfield Gill, Inc. Boston, MA jgill () penfield-gill com <http://www.penfield-gill.com/gill> ____________________________________________________________________________
Current thread:
- IP: a bit more Every 30 minute spiders Dave Farber (Nov 03)