Re: Problem with Icons of the Web

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 25 Aug 2010 10:33:47 -0500
Dan Davis <danieldavis () me com> wrote:

> Hi, a buddy of mine runs a site called vNES (virtualnes.com).
> According to alexa.com, it should be on your favicon image. The
> search  on your site however shows it is not available. 
>
> According to alexa: Ranked 109,666  in the world, 76,567 in the U.S.
>
> Full text from http://www.alexa.com/siteinfo/virtualnes.com#
>
> "Virtualnes.com is ranked #109,666 in the world according to the
> three-month Alexa traffic rankings. The site's visitors view an
> average of 5.1 unique pages per day. While approximately 32% of
> visitors to the site come from the US, where it is ranked #76,567, it
> is also popular in Venezuela, where it is ranked #5,732. The time
> spent in a typical visit to Virtualnes.com is approximately five
> minutes, with 39 seconds spent on each pageview. Relative to the
> overall population of internet users, the site's users are
> disproportionately male, and they are disproportionately low-income,
> childless people under the age of 35 who browse from home."
>
> I checked the possible problems and solutions list at the bottom of
> the image. vNES checks through all of them. 
>
> Given this information, I question the accuracy of your favicon data
> map.
>
> Thanks,
> Dan


Hi Dan,

I just checked the Alexa scan and the reason virtualnes.com is not in
the data is that the host timed out of the scan:

Nmap scan report for www.virtualnes.com (96.9.166.154)
Host is up, received user-set (0.15s latency).
Skipping host www.virtualnes.com (96.9.166.154) due to host timeout
Final times for host: srtt: 148380 rttvar: 148380  to: 741900

The scan was started at 2010-01-24 23:13 UTC 

The scan command was:

/home/bmenrigh/projects/security/nmap/nmap/nmap --datadir /home/bmenrigh/projects/security/nmap/nmap -iL fifo_alexa_4 
-v -d2 --script=favicon-progs/favicon-survey.nse -n -PN -p80 --min-parallelism 768 --max-parallelism 768 -T4 
--min-hostgroup 8192 --host-timeout=30m --script-args 
hashdir=/home/bmenrigh/favicon_survey/hashdirs/alexa_4/,icondir=/home/bmenrigh/favicon_survey/icondirs/alexa_4/,http.useragent='Mozilla/5.0
 (compatible; favicon.ico Internet survey; research only)' -oA scan_alexa_4

The scan finished at:

# Nmap done at Mon Jan 25 18:57:23 2010 -- 82408 IP addresses (82408 hosts up) scanned in 71048.79 seconds

(Note this was a small fraction of the total Alexa scan)

As you can see the host timeout was 30 minutes.  We did a lot of
testing to determine a reasonable host timeout that would let majority
of even slow sites finish without letting a very small minority of
sites hold up the whole scan.

The scan came from 132.239.181.229 and was somewhere in the 71ksec scan
window.  The User-Agent string was "Mozilla/5.0 (compatible;
favicon.ico Internet survey; research only)".  Perhaps your buddy can
check his webserver logs to get a better idea of what requests it
received and how it responded?

There are a number of constrained resources for a scan of this scale.
We had socket pressure, disk space pressure, memory pressure, and strict
time constraints.

I'm sorry www.virtualnes.com did not end up in the final data.  I feel
we made a reasonable effort to include it though.

Regards,

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)

iEYEARECAAYFAkx1eXYACgkQqaGPzAsl94K+TgCaAv1k8ypF8xbYhD4cQ691ypLV
hSUAnAn43xoseDm62T/fW5kg7kdBsGkS
=Hbmc
-----END PGP SIGNATURE-----
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/