funsec mailing list archives

Why is Whitehouse.gov telling search engines to go away?


From: "Richard M. Smith" <rms () computerbytesman com>
Date: Tue, 27 Dec 2005 12:08:20 -0500

What an odd robots.txt file at Whitehouse.gov.  Why is the Whitehouse
Webmaster telling searching engines like Google to not search and cache the
Whitehouse Web site?  The robots.txt files is 92K bytes of disallows.

Richard M. Smith
http://www.ComputerBytesMan.com

============================================================== 
 
# robots.txt for http://www.whitehouse.gov/

User-agent:     *
Disallow:       /cgi-bin
Disallow:       /search
Disallow:       /query.html
Disallow:       /help
Disallow:       /360pics/iraq
Disallow:       /360pics/text
Disallow:       /911/911day/iraq
Disallow:       /911/911day/text
Disallow:       /911/heroes/iraq
Disallow:       /911/heroes/text
Disallow:       /911/iraq
Disallow:       /911/messages/text
Disallow:       /911/patriotism/iraq
Disallow:       /911/patriotism/text
Disallow:       /911/patriotism2/iraq
Disallow:       /911/patriotism2/text
Disallow:       /911/progress/iraq
Disallow:       /911/progress/text
Disallow:       /911/remembrance/iraq
Disallow:       /911/remembrance/text
Disallow:       /911/response/iraq
Disallow:       /911/response/text
Disallow:       /911/sept112002/iraq
Disallow:       /911/sept112002/text
Disallow:       /911/text
Disallow:       /QA-test/text
Disallow:       /afac/index.htm/text

...
_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.


Current thread: