Re: Simultaneous invocations of nmap

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rick,

We run 64 instances of Nmap from the same box.  We've resolved
most of the issues that make this a problem.  I'll discuss our
infrastructure and highlight a few of the problems you may run into.

We have a modern 10 Gb core with most edge infrastructure connected at
1 Gb.  Most end hosts are still connected at 100Mb.  Except for a few
places on campus, all switching gear is very modern and can handle more
flows than a single host can generate.  Since we have about 40k
machines online at a time spread across 3 /16 netblocks we need to be
able to scan very quickly.  Nmap currently does congestion control per
host-group rather than per host.  We assume that all detected
congestion is the fault of the target host, not the network.  To work
around this, the full network scanning we do is on a per-host basis.

The scanning box we have is a 8 core, 8 GB Ram, e1000 NIC.  We run
64 Nmaps in parallel with these flags:

nmap -sV –allports -p- --open -ttl 10 -O2 -vv -d -P0 -T5
- --initial-rtt-timeout 80 --max-rtt-timeout 200 --min-rtt-timeout=1
- --min-parallelism 128 --max-parallelism 256 --max-retries=1
- --host-timeout 600000 –oA log --excludefile excludelist <target host>

This has not been without problems.  The first thing we run out of is
file handles which we increased like so (sysctl.conf):

fs.file-max = 16384

The next problem we had was getting packets into the driver's
ring-buffer fast enough so we increased the kernel backlogs like so:

net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 16384

This helped but didn't fix the problem.  Turns out that the default
txqueuelen of 1000 isn't enough so we increased it like so:

# ifconfig eth0 txqueuelen 16384

This solved all of Nmap's problems with raw sockets.  Unfortunately we
were still having trouble with UDP and TCP sockets used in other
software like traceroute and some custom code of ours.  To fix these we
had to give network a lot more memory:

net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 65536 262144 33554432
net.ipv4.tcp_wmem = 65536 262144 33554432
net.ipv4.tcp_mem = 33554432 67108864 134217728

And we needed to recycle sockets faster with:

net.ipv4.tcp_tw_recycle = 1

This has taken care of nearly all of our networking problems on the box
and as best we can tell, each Nmap is returning the expected results
with no noticeable negative effect on other scans.

Unfortunately we still have one networking issue left to fix.  We can
not SCP to or from the machine.  SCP connects and asks for a password
but hangs forever.  We've troubleshot this a little bit since SSH still
works to and from the machine.  One interesting thing to note is that
we can't do the cat file | ssh ... 'cat > file' trick unless we disable
TTY allocation with -T.

This setup is working very well for us.  We scan all hosts on campus
about once a week.  The box averages 100,000 packets per second which
translates to about 10 TB of sent data each month.

We're really looking forward to a version of Nmap with per-host
congestion control so that we don't need to do all of this.  I suppose
that means I should start on a patch...

Brandon


On Tue, 19 Feb 2008 17:54:01 -0800 or thereabouts doug () hcsw org wrote:

> Hi Rick,
>
> Good question. Yes, people have thought about doing this before. In
> Fyodor's soon-to-be-released book on Nmap there is a short section
> about executing multiple instances in parallel. Fyodor warns that
> performance can actually suffer due to context switching and memory
> overhead.
>
> Usually one Nmap instance will be able to fully saturate your network
> connection to the target networks, but this isn't to say that you
> shouldn't give it a shot. If you do, please let us know how it turns
> out. In particular, I'm curious as to whether this tactic could
> improve performance by taking advantage of multiple processors/cores
> in modern SMP machines.
>
> Good luck,
>
> Doug
>
>
> On Tue, Feb 19, 2008 at 08:29:30PM -0500 or thereabouts, Rick
> Rineholt wrote:
> > Hi
> > Has anyone done any performance studies on invoking simultaneous 
> > instances of nmap on the same machine each covering separate
> > subnets versus executing a single instance specifying all the
> > subnets?  One reason I ask is I thought I read some info that nmap
> > has some built in mechanism to throttle back sending out
> > explanatory packets if it starts losing too many and surely having
> > more than one instance at a time running I would think would
> > interfere with that mechanism. Are there any guide lines available
> > to help performance of discoveries on very large networks?
> > Thanks
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHu+mVqaGPzAsl94IRArG8AKCFHWm+SSQ6pWeN/VdqdzzTkYlF1ACgo8qK
radwefEWgzxZm2TPuMJ6Lj0=
=H8oo
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org