Re: massping-migration and other dev testing results

[David Fifield <david () bamsoftware com>]

On Tue, Sep 11, 2007 at 07:55:28AM +0000, Brandon Enright wrote:
> Okay, I have testing results finished.  To recap from the last few weeks,
> David wanted to compare the speed of host discovery with the current release
> (4.20) with the new migrated code in the svn nmap branch and some
> experimental congestion control changes in the massping-migration branch.
> I triple checked that I was using the right options with the right version
> of Nmap.
>
> I'm presenting the scan in the order that they were done.  Since these
> results represent a significant amount of time scanning, a decrease of
> about 2000 hosts discovered is to be expected expected from start to finish.
>
> For each group, I did two scans.  A scan 'a' and a scan 'b'.  Scan 'a' had
> a min/max parallelism of 1024.  Scan 'b' just had -T5 specified.
>
> massping-migration branch results first:
>
> r5815 has all the tweaks.
> massping/r5815a:
> # Nmap done at Tue Sep 11 00:13:17 2007 -- 186368 IP addresses (10582 hosts
> up) scanned in 995.604 seconds
>
> massping/r5815b:
> # Nmap done at Tue Sep 11 00:52:54 2007 -- 186368 IP addresses (12586 hosts
> up) scanned in 3371.708 seconds
>
>
> r5786 does not have the cap of 50 on the cwnd scaling factor
> massping/r5786a:
> # Nmap done at Tue Sep 11 01:11:46 2007 -- 186368 IP addresses (10214 hosts
> up) scanned in 1000.378 seconds
>
> massping/r5786b:
> # Nmap done at Tue Sep 11 01:48:37 2007 -- 186368 IP addresses (12164 hosts
> up) scanned in 3210.024 seconds

It makes sense that these should be about the same. The version without
the cap is a little faster, but it doesn't make much difference.

> r5780 does not scale with the inverse of the packet receipt ratio
> massping/r5780a:
> # Nmap done at Tue Sep 11 02:20:55 2007 -- 186368 IP addresses (9884 hosts
> up) scanned in 957.285 seconds
>
> massping/r5780b:
> # Nmap done at Tue Sep 11 04:27:53 2007 -- 186368 IP addresses (13667 hosts
> up) scanned in 8573.451 seconds
>
>
> Now for the latest nmap SVN branch:
>
> nmap/r5817a:
> # Nmap done at Tue Sep 11 05:25:43 2007 -- 186368 IP addresses (11211 hosts
> up) scanned in 274.352 seconds
>
> nmap/r5817b:
> # Nmap done at Tue Sep 11 07:16:00 2007 -- 186368 IP addresses (14122 hosts
> up) scanned in 5556.831 seconds

This is unexpected. The only difference between current /nmap and r5780
from massping-migration with regard to scanning is that r5780 doesn't
adjust congestion parameters for ICMP destination unreachables. You can
verify this yourself with

        svn diff svn://svn.insecure.org/nmap@5817 svn://svn.insecure.org/nmap-exp/david/nmap-massping-migration@5780

That could account for the difference between 8573 and 5557 seconds, but
not for the difference between 957 and 274 seconds. The effect of
ignoring or not ignoring destination unreachables is completely negated
when you set the min and max parallelism to be equal, a fact which your
david_mpm_r5780a.svg supports.

Actually another thought just occurred to me. The
nmap-massping-migration branch also has debugging and profiling flags
turned on. I wonder if that could have such a large effect? I've noticed
that these flags make Nmap use a lot more CPU on my system, but I
haven't measured them to actually slow down any scans.

I'll remove those flags in preparation for the next few tests I'd like
you to do (see below).

> Now for the latest release of nmap (4.20) results.  The a and b scans are
> the same as the previous a/b.  The c replaced --min-parallelism 1024 with
> - --min-hostgroup 1024 and the d uses a --min-hostgroup of 2048:
>
> 4.20a:
> # Nmap run completed at Tue Sep 11 05:40:03 2007 -- 186368 IP addresses
> (13257 hosts up) scanned in 2561.333 seconds
>
> 4.20b:
> # Nmap run completed at Tue Sep 11 05:52:08 2007 -- 186368 IP addresses
> (13235 hosts up) scanned in 3285.450 seconds
>
> 4.20c:
> # Nmap run completed at Tue Sep 11 06:40:57 2007 -- 186368 IP addresses
> (13213 hosts up) scanned in 3436.940 seconds
>
> 4.20d:
> # Nmap run completed at Tue Sep 11 07:44:46 2007 -- 186368 IP addresses
> (13168 hosts up) scanned in 3268.226 seconds
>
>
> This is a lot of data recorded over almost 12 hours of time.  I'm more than
> willing to run any two of these scans side-by-side to get a better idea for
> their speed and accuracy when the network is the same for both scans.  Or
> if there is another SVN revision or scan args I should try, let me know.

I have two requests: First, please generate graphs from your nmap/r5817
tests. The /nmap branch prints enough information to make the graphs.
Second, please run side-by-side the latest /nmap (the same as your test
above) and the latest nmap-massping-migration (r5824, the same as above
the exception of the removal of debugging flags). Run your 'a' tests
concurrently, then your 'b' tests concurrently.

> - From these results, I'd say that the SVN nmap branch is both the fastest,
> more accurate, and most adjustable.  Two things strike me as odd in these
> tests.  The first is that the congestion control ideas implemented in
> massping-migration sound excellent but perform poorly.  The second is that
> identical scans with nmap 4.20 took twice as long in this test.  I'm going
> to experiment more to try to reproduce the release version scanning our
> network in about 23 minutes on average.

I wouldn't say it performs poorly when it beats nmap trunk with
unspecified parallelism (3372 vs. 5557 seconds), assuming the difference
of 1536 hosts is due to the difference of six hours between the scans. I
hope to isolate whether the debugging flags were slowing down the tests
with a high parallelism, although I think that use case is secondary to
having fast scans without specifying the parallelism.

It would be pretty sweet to have the best of both worlds: 180,000 hosts
scanned in an hour with high accuracy, or 180,000 hosts scanned in under
five minutes (!) with a 20% loss in accuracy. That's a lot of
flexibility compared with massping, and I think it's attainable.

It is curious that the 4.20 scans have such a large difference from
before.

> To aid in figuring out what is going on, I've generated graphs for each of
> the massping-migration tests.  They are at
> http://noh.ucsd.edu/~bmenrigh/nmap/

It looks like you do have a bunch of blocks with no hosts up in them.
The congestion window is stuck at 10 for the length of those blocks.
--randomize-hosts might be good option for you, as it both increases the
group size and might mix some live hosts into those long empty
stretches. But please leave things as they are for your current tests.

> I have a few more tests I want to run but in the mean time, if anyone has
> any questions or requests, let me know.

Thanks so much for your help so far. Has anyone else given this a try?

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org