Impolite searching of web trees for non-existent pages

[aland () FREERADIUS ORG (Alan DeKok)]

  Related to Cedric Amand's comment on BugTraq a few days ago, I'd
like to mention a vaguely similar issue.

  A few weeks ago, I noticed a particular IP was scanning my web
sites.  The unusual part was that it wasn't a standard search engine
following links, or someone following old or expired links.  Instead,
it was methodically requesting pages which had *never* existed.

  Not only that, the names of the pages that it was looking for made
me suspicious as to their intent.  When I looked at the web page of
the machine scanning me, I was redirected to:

   http://www.cyveillance.com/response1.html

  I sent them the following message, and after two weeks, have heard
no response, so I'm making this post to the Incidents list.

  Their web page says (in part):

  (quote)
     Please be assured that Cyveillance is a responsible corporate
     citizen.

     ...

     If we have not addressed all of your concerns about our visit to
     your site, please send e-mail to issues () cyveillance com, and we
     will respond in a timely manner.

  (end quote)

  My impression after nearly three weeks without a response is that
their web page isn't entirely correct.

  My comments here aren't meant to reflect a security bug or explicit
attack, but are made to make administrators aware of additional
unfriendly systematic scans of web sites for trivially "hidden" or
"private" material.

  Alan DeKok.

         ------ original message ----------------

Date: Wed, 12 Jan 2000 16:35:05 -0500
Message-Id: <200001122135.QAA32356 () freeradius org>
From: aland () freeradius org
To: issues () cyveillance com
Subject: You're "following" links which don't exist
cc: aland () freeradius org

  Your response page at http://www.cyveillance.com/response1.html says:

> Our technology is designed to find only publicly available materials; ...

  Well, here's a sample of my log, in which you search for pages which
NEVER HAVE EXISTED.

  In addition, the search for 'private/' and 'forms/webfeedback/' make
me wonder what's going on.

  Can you provide me with the links which caused you look for these
URLs on http://www.freeradius.org/ ?  I would be very interested in
knowing what was going on.

  Alan DeKok.

[Wed Jan 12 15:15:17 2000] access to SERVER_ROOT/5.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:17 2000] access to SERVER_ROOT/167.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:17 2000] access to SERVER_ROOT/3163.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/171.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/169.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/1649.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/private/1951.shtml failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/1079.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:18 2000] access to SERVER_ROOT/173.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:19 2000] access to SERVER_ROOT/forms/webfeedback/ failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:19 2000] access to SERVER_ROOT/869.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:19 2000] access to SERVER_ROOT/3316.html failed for 216.32.64.10, reason: File does not exist
[Wed Jan 12 15:15:19 2000] access to SERVER_ROOT/3315.html failed for 216.32.64.10, reason: File does not exist