Scanning DNS names fast (was Re: favicon survey script)

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 06 Aug 2009 22:51:23 +0200
Vlatko Kosturjak <kost () linux hr> wrote:

> Brandon Enright wrote:
> > Indeed, I have been scanning ;-)
> > Here is what I scanned:
> > * 100M random IPs (small percentage actually listening on 80)
> > * 450k IPs resolved from links in Wikipedia (>99% listening on 80)
> > * 3M names (not IPs) from open directory/dmoz, (>99% listening on
> > 80)
>
> Wow! You really scanned a lot! :)

Yeah, actually the only aspect of this scanning that was non-trivial
was the 3M open directory names.

In the past I asked the list for ideas on how to get Nmap to scan
millions of names quickly and I got a few decent ideas but nothing
distilled down to something anybody could just apply.

The issue is that Nmap uses gethostbyname() serially for each name so
resolving even a few thousand names takes forever.  Until we build some
solution into Nmap for this, here is how to get around the problem.

First, install a caching nameserver locally.  I used djbdns and
increased the cache size to 1GB but I bet even something like dnsmasq
could be used if you can control its cache size.  Point
your /etc/resolve.conf to your local DNS resolver.

Second, make a file with a list of all of the names you want to scan.

Third, download the attached perl script for fast name resolving
(pre-caching).

Fourth, setup your scan like so:

$ cat names.txt | ./fastprelookup.pl | nmap -iL - -n <other options>

The reason this works is that fastprelookup.pl acts as a simple input
buffer that resolves DNS names very quickly and gets them into your
cache.  Then when Nmap tries to resolve the names it goes very quickly.

Instead of taking years to resolve/scan these names, it took a night.

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)

iEYEARECAAYFAkp7RX0ACgkQqaGPzAsl94J5TgCfWpCVg5cFTkCGwiNWUV8k6beR
QDIAniHzu6QZrtgWI5tLbmGdykt7brSd
=Syal
-----END PGP SIGNATURE-----

Attachment: fastprelookup.pl
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org