Re: Slow name-resolution of very large target list

[doug () hcsw org]

Hi Brandon,

On Thu, May 22, 2008 at 02:13:49PM -0700 or thereabouts, Fyodor wrote:
> On Thu, May 22, 2008 at 07:16:31AM +0000, Brandon Enright wrote:
> > I've tried the scan from another network that has access to many very
> > fast local DNS servers and have specified them with --dns-servers but
> > that didn't seem to make any noticeable difference.

Fyodor beat me to it--I was about to say almost the exact same thing.
The nmap DNS system is for reverse DNS only and other types of DNS
were never in the specs when nmap_dns was created. Like Fyodor says,
when you do need to do this, it is usually easiest to use a separate
program. I think the adns library has an example that does exactly
this. (?)

SHAMELESS PLUG: Nuff has a program that will do this for you, but
how well it can scale to millions of resolution, I don't know:

http://www.hcsw.org/nuff/code.html#section.auto.4.2.3

$ printf 'a.com\nb.com\n' | nuff resolve -stdin


Alternatively, there is a pattern I frequently use for doing things
in parallel. I call it fork/wait:

#!/usr/bin/perl
# fork/wait skeleton by frax (patent pending)

$parallelism = 10;

while(<>) {
  chomp;
  if (fork() == 0) {
    system("host -t A $_");
    exit;
  }
  $children++;
  if ($children >= $parallelism) {
    wait();
    $children--;
  }
}

while ($children > 0) {
  wait();
  $children--;
}


The above will do up to 10 resolutions in parallel. Use it like this:

cat domains.txt | perl pwnyresolv0r.pl > output.txt

The output should be like the following. It will require a little
post-processing and the results WILL NOT be in the same order as your
input file.

hcsw.org has address 65.98.116.106
slashdot.org has address 66.35.250.150
google.com has address 64.233.187.99
google.com has address 72.14.207.99
google.com has address 64.233.167.99

Note for doing millions of resolutions this may not be efficient enough
because every resolution spawns 3 (!) processes: Another perl instance,
the shell created by system(3), and an instance of host(1). For short
lived tasks like DNS resolution the process overhead might be too much.
However, fork/wait is a great way to parallelise many types of IO
bound programs.

Hope this helps,

Doug

PS. Be careful about locking your output stream. I think the above
code is OK because they will all be atomic writes, but if you're not
careful you can get things like:

hcsw.org hslashdot.org has address 66.35.250.150
as address 65.98.116.106

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org