Re: New development in host discovery: response rate scaled congestion control

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 6 Sep 2007 10:11:48 -0600
David Fifield <david () bamsoftware com> wrote:
> You said that you ran all the scans at the same time. This new technique
> ought to approximate TCP congestion control more closely--if four
> concurrent Nmaps are run, each one should run about 1/4 as fast.

This makes sense theoretically.  Practically though, this machine is on
gigabit Ethernet and capable of sending hundreds of thousands of small
packets a second.  I think the limiting factor in sending packets is the
module for the NIC, OS buffers, and syscall servicing rate.

This machine normally has a minimum of 16 nice'd Nmap running, each against
a single host.  So when I run 4 test Nmaps, that brings the total  Nmap
count up to 20.

For the sake of testing, I've killed off all the Nmaps and run these tests
with the box idle.

> That's the best explanation I can think of. Otherwise, your 2048-2048
> result doesn't jibe with the one from a few days ago:
>
> > Nmap done: 186368 IP addresses (15804 hosts up) scanned in 258.023
> > seconds Raw packets sent: 1398404 (55.937MB) | Rcvd: 51659 (2.426MB)
>
> The effect of both the scaled congestion control and the additional drop
> processing should be negligible with such a large minimum parallelism.
> The difference in time should not be that great.
>
> All of the Nmaps are sharing the bandwidth, except that the 2048-2048
> guy is unfairly blasting the network. That's causing the other scans to
> slow down and drop packets.
>
> Please run the NOMIN-NOMAX scan all by itself. And then the 2048-2048
> scan by itself, just as a sanity check. If the time is again over about
> 500 seconds, then I must have changed something I didn't intend to.

Okay, I ran the NOMIN-NOMAX and 2048-2048 scan, one after another with the
box otherwise completely idle:

2048-2048: (david_new_4c.svg)
# Nmap done at Thu Sep  6 20:44:41 2007 -- 186368 IP addresses (11113 hosts
up) scanned in 942.198 seconds

NOMIN-NOMAX: (david_new_4d.svg)
# Nmap done at Thu Sep  6 17:13:47 2007 -- 186368 IP addresses (12902 hosts
up) scanned in 2726.080 seconds

> The only other possibility I can think of is that the additional drop
> processing is slowing it down. I had previously ignored dropped ICMP
> destination unreachables; those are no longer ignored because I thought
> the scaled congestion control would more than make up for the drops. If
> the above tests don't give favorable results, please try again with the
> attached patch (it's just the reverse of r5780). Ignoring those drops
> was something of a compromise to begin with. I think not ignoring them
> is more correct.

Okay I applied this patch.  There was a 31 line offset against the current
SVN so I had to specify a higher fuzz factor.  I think ignoring the
drops (destination net and destination host) is probably the right way to
go.  I'm interested in hearing your reasoning for why they shouldn't be
ignored.

2048-2048: (david_new_5c.svg)
# Nmap done at Thu Sep  6 18:21:49 2007 -- 186368 IP addresses (11217 hosts
up) scanned in 941.418 seconds

NOMIN-NOMAX: (david_new_5d.svg)
# Nmap done at Thu Sep  6 19:07:28 2007 -- 186368 IP addresses (13074 hosts
up) scanned in 2721.055 seconds

> Thanks again for your dedicated testing.

My pleasure.  Thanks for putting up with me -- I feel like I've been Captain
Negativity lately.

> > I can't explain these results.  I've generated graphs these scans in
> > case those will be helpful at all.  I removed the y axis limit of 80
> > before generating these.  The graphs are located at:
> >
> > http://noh.ucsd.edu/~bmenrigh/nmap/
>
> Thanks, those help. (The SVG is good because I can stretch them out and
> get higher resolution.) Although they all looked to be capped at 80, so
> for the 2048-2048 scans the black line was clipped completely (not even
> present in the XML file).

Yep, I screwed up.  Sometime I shouldn't be allowed to use a computer.  I've
re-generated all the graphs with the 80 limit really removed this time.

What SVG viewer do you use?  Everything I've tried has been rather unwieldy.

> David Fifield

Although the standard '.nmap' output of one of these scans is several
hundred megs, I can make them available to you (privately) if you think it
will help you track down what is going on better that your graphs will.

Once you have time to think things over let me know how you'd like me to
proceed.

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFG4GjNqaGPzAsl94IRAlD2AKCfmIzcKuun7gWy5LSMxQAcFBM7QwCeP4U/
CRjUYlydLVWPIBenAS9Mdtk=
=Z+g6
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org