Security Incidents mailing list archives

Re: ports ports and more ports

From: getch () RM-R NET (David Getchell)
Date: Tue, 15 Feb 2000 08:55:47 -0600


Hello Tyler,

Friday, February 11, 2000, 10:37:56 AM, you wrote:

T> On a side note, anyone know why /robots.txt is being requested maybe 3-4
T> times daily on my webserver? It happens quite often, and with other
T> webservers too. I'm not too concerned about it just a little curious.

Well behaved robots request that file first before requesting anything
else from your site.  This allows you to keep search engine spiders from
rummaging about in your cgi scripts, etc.  See
http://info.webcrawler.com/mak/projects/robots/norobots.html.

Most of this is probably search engine spiders.  If you are curious, see
http://www.searchenginewatch.com/webmasters/spiders.html for more advice
on figuring out what is visiting you.  For a nice reference chart, see
http://www.searchenginewatch.com/webmasters/spiderchart.html.

--
Best regards,
 David                            mailto:getch () rm-r net

Current thread:

ports ports and more ports Tyler (Feb 11)
- Re: ports ports and more ports David Getchell (Feb 15)
- Dispostion of UPD/137 packets? Bill Pennington (Feb 15)
- Re: ports ports and more ports Robert Lau (Feb 15)
- succesful crack Bob Lockie (Feb 15)
  - Re: succesful crack Gene Harris (Feb 16)
    - Re: succesful crack **read nine (Feb 17)
    - Re: succesful crack R. Gupta (Feb 17)
- Port Scanning (perhaps related to "A very strange port scan") Warren Belfer (Feb 15)
- MASSIVE ssh attack attempt Mark Shirley (Feb 15)
  - Re: MASSIVE ssh attack attempt Omachonu Ogali (Feb 16)
    - Re: MASSIVE ssh attack attempt Jose Nazario (Feb 17)

(Thread continues...)