Re: massping-migration and other dev testing results

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 11 Sep 2007 12:36:05 -0600
David Fifield <david () bamsoftware com> wrote:

> This is unexpected. The only difference between current /nmap and r5780
> from massping-migration with regard to scanning is that r5780 doesn't
> adjust congestion parameters for ICMP destination unreachables. You can
> verify this yourself with
>
>       svn diff svn://svn.insecure.org/nmap@5817
> svn://svn.insecure.org/nmap-exp/david/nmap-massping-migration@5780
>
> That could account for the difference between 8573 and 5557 seconds, but
> not for the difference between 957 and 274 seconds. The effect of
> ignoring or not ignoring destination unreachables is completely negated
> when you set the min and max parallelism to be equal, a fact which your
> david_mpm_r5780a.svg supports.
>
> Actually another thought just occurred to me. The
> nmap-massping-migration branch also has debugging and profiling flags
> turned on. I wonder if that could have such a large effect? I've noticed
> that these flags make Nmap use a lot more CPU on my system, but I
> haven't measured them to actually slow down any scans.

I had noticed the additional CPU usage but I didn't think much of it.  I've
compiled debug and profiling code before and not seen this kind of
performance hit.  Perhaps profiling has a lot more work to do with C++ code?

> I'll remove those flags in preparation for the next few tests I'd like
> you to do (see below).
>
>
> I have two requests: First, please generate graphs from your nmap/r5817
> tests. The /nmap branch prints enough information to make the graphs.

Done.  Sorry I didn't provide them last night.  Locate in the same place.

> Second, please run side-by-side the latest /nmap (the same as your test
> above) and the latest nmap-massping-migration (r5824, the same as above
> the exception of the removal of debugging flags). Run your 'a' tests
> concurrently, then your 'b' tests concurrently.

This made the difference we were looking for.

> > - From these results, I'd say that the SVN nmap branch is both the
> > fastest, more accurate, and most adjustable.  Two things strike me as
> > odd in these tests.  The first is that the congestion control ideas
> > implemented in massping-migration sound excellent but perform poorly.
> > The second is that identical scans with nmap 4.20 took twice as long in
> > this test.  I'm going to experiment more to try to reproduce the
> > release version scanning our network in about 23 minutes on average.
>
> I wouldn't say it performs poorly when it beats nmap trunk with
> unspecified parallelism (3372 vs. 5557 seconds), assuming the difference
> of 1536 hosts is due to the difference of six hours between the scans.

Poor choice of wording on my part, I should have been more explicit about
what I meant.  The trunk scans were done much later than the MPM branch and
even though they took longer with just -T5, found many more hosts.  I felt
like the speed/accuracy ratio was pretty poor with MPM versus trunk.  This
appears to have been fixed by turning off the debugging and profiling
options.

> I hope to isolate whether the debugging flags were slowing down the
> tests with a high parallelism, although I think that use case is
> secondary to having fast scans without specifying the parallelism.
>
> It would be pretty sweet to have the best of both worlds: 180,000 hosts
> scanned in an hour with high accuracy, or 180,000 hosts scanned in under
> five minutes (!) with a 20% loss in accuracy. That's a lot of
> flexibility compared with massping, and I think it's attainable.

Agreed.  Please don't think for a minute that I'm not happy with your
migration work.  There are so many factors involved in these tests I'm
doing that it is very hard to filter out sources of jitter to get down to
the real speed/accuracy ratio of different versions and the effect that
various tweaks have.

> It is curious that the 4.20 scans have such a large difference from
> before.

I'm going to play with this some more.

> > To aid in figuring out what is going on, I've generated graphs for each
> > of the massping-migration tests.  They are at
> > http://noh.ucsd.edu/~bmenrigh/nmap/
>
> It looks like you do have a bunch of blocks with no hosts up in them.
> The congestion window is stuck at 10 for the length of those blocks.

Students aren't back.  We're missing about 10k hosts.  Most of these blocks
will go away in 2 weeks.

> --randomize-hosts might be good option for you, as it both increases the
> group size and might mix some live hosts into those long empty
> stretches.

I was under the impression that --randomize-hosts only randomizes host
within a ping group.  The fact that it also increases the size of the
ping group is *huge*.  The man doesn't make this all that clear.  I went
ahead and looked at the code for this and I have a couple of thoughts.

First, PING_GROUP_SZ is set to 4096.  When you use --randomize-hosts
'o.ping_group_sz = PING_GROUP_SZ * 4;' is run.  The man says the group can
grow to up to 8096.  There doesn't appear to be any special cap so the
group size would actually be 16384.

Second, it really surprises me that an important value like this isn't
adjustable.  I thought --min-hostgroup set the ping group size but after
looking at the code, this doesn't appear to be the case.  I suppose most
people aren't scanning 10k+ hosts so it doesn't matter much.  For those
that do though, it really matters.

Since this value is already so large, using the value from --min-hostgroup
is probably not a good idea.  Perhaps another option like --min-ping-group.

I have preliminary results that show larger ping_groups (using either
- --randomize-hosts, or recompiling) to really help.  Some of the scans are
still going though so I'll have to send a follow up email illustrating this
when I get home.

> But please leave things as they are for your current tests.
>
> > I have a few more tests I want to run but in the mean time, if anyone
> > has any questions or requests, let me know.
>
> Thanks so much for your help so far. Has anyone else given this a try?
>
> David Fifield

Okay, here are scan results.  To recap, my 'a' scan uses -T5 and has a
min/max parallelism of 1024 against ports 135,139,445,3389 on 180k IPs.  My
'b' scan just uses -T5.

The 'a' scans were started simultaneously for the Nmap SVN trunk as well
as the massping-migration branch (MPM).  The same goes for the 'b' scans.


david_mpm_r5824a.nmap:
# Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11790 hosts
up) scanned in 288.591 seconds

david_nmap_r5824a.nmap:
# Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11936 hosts
up) scanned in 287.305 seconds

All I can say here is whoa, that's close. And fast.


david_mpm_r5824b.nmap:
# Nmap done at Wed Sep 12 00:17:31 2007 -- 186368 IP addresses (15628 hosts
up) scanned in 2640.259 seconds

david_nmap_r5824b.nmap:
# Nmap done at Wed Sep 12 00:49:08 2007 -- 186368 IP addresses (15901 hosts
up) scanned in 4536.876 seconds

Okay, now your new code is starting to shine.  Much faster, almost as
accurate.


I'm going to follow up with tweaked PING_GROUP_SZ results but here is a
preview. I ran david_mpm_r5824b.nmap (took 2640 seconds) with
- --randomize-hosts and shaved off 600 seconds:

david_mpm_r5824c.nmap:
# Nmap done at Wed Sep 12 01:27:32 2007 -- 186368 IP addresses (13327 hosts
up) scanned in 2019.837 seconds

This scan did find 2k fewer hosts, but since they were done around 5pm
local time some of this drop-off is hosts being turned off.

Hopefully all that helps!  Let me know where to go from here.

Brandon


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFG50nXqaGPzAsl94IRAhwUAJ9A69fRbaip/NZHaeGHA/W4APIyNQCffl8Y
AUDYWsTKEskzeftysQv9v5I=
=Jczo
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org