Re: favicon survey script

[David Fifield <david () bamsoftware com>]

On Thu, Aug 06, 2009 at 08:26:12PM +0000, Brandon Enright wrote:
> On Thu, 6 Aug 2009 11:49:03 -0600 David Fifield <david () bamsoftware com> wrote:
> > On Thu, Aug 06, 2009 at 08:27:24AM +0200, Vlatko Kosturjak wrote:
> > > David Fifield wrote:
> > > > Vlatko, did you ever finish mapping the hashes back to favicons
> > > > in your research?
> > >
> > > Yes, I did. But extracted only top 10 from each survey done
> > > (dmoz,80,443) and have summarized that into favicon-db (just updated
> > > favicon-db in attachment to reflect survey done).
> ...snip...
> > Awesome. I would prefer to keep only the hashes that we have measured
> > to be common. João Correa is going to do some scanning and Brandon
> > Enright has been scanning as well.
> >
> > The hash A8FE5B8AE2C445A33AC41B33CCC9A120 is by far the most common
> > one I found in my scanning, and I think in Brandon's too. Just like
> > you noted, it is really HTML text:
>
> Indeed, I have been scanning ;-)
>
> Here is what I scanned:
>
> * 100M random IPs (small percentage actually listening on 80)
> * 450k IPs resolved from links in Wikipedia (>99% listening on 80)
> * 3M names (not IPs) from open directory/dmoz, (>99% listening on 80)
>
> I'm making a compressed (7Zip) tarball of the entire favicon directory
> available at:
>
> http://noh.ucsd.edu/~bmenrigh/favicon.tar.7z

I'm downloading it now. João, are you getting a copy too? Brandon did a
huge part of the work by scanning all these hosts. Now we have to find
out the server software for each of the hashes, as Vlatko did in his
scans. It should be pretty easy by visiting the sites in the hash/
directory; you can also look at the icon in the icon/ directory with an
image viewer.

50 initially strikes me as a pretty good number for the size of the
database. When you're looking up software be sure to refer to the list
Vlatko already made at http://seclists.org/nmap-dev/2009/q3/0475.html.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org