Re: New development in host discovery: response rate scaled congestion control

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 5 Sep 2007 16:36:14 -0600
David Fifield <david () bamsoftware com> wrote:

> Hi all,
>
> You may have been following the development of ultra_scan-based host
> discovery. In the last few weeks, we've been working to improve its
> performance. Some testing reports have shown slow performance when many
> packet drops are detected. I recently hit on an idea that could greatly
> improve the situation in these cases.
>
> The idea is that whenever we make a change to the group congestion
> window, we scale the increment by the inverse of the packet receipt
> ratio; i.e., the ratio of the packets that have been responded to versus
> the number that have been sent. This makes the congestion window vary in
> a healthier manner, as it would with a TCP stream with a steady supply
> of responses coming in. More information and graphs are here:
>
>       http://www.bamsoftware.com/wiki/Nmap/ResponseRateScaledCongestionControl


Excellent page.  You Nmap wiki has been a great read.

> You can get it by running
>
>       svn co
> svn://svn.insecure.org/nmap-exp/david/nmap-massping-migration
>
> I thought this was a bit too experimental to go into the Nmap trunk in
> Subversion, but early results are extremely promising! If you're not
> afraid of building from Subversion, please give it a try.
>
> I recently committed a change that made certain ICMP errors not count as
> drops, in an effort to improve host discovery performance. This new code
> takes that out--many more drops are detected than in the Nmap trunk.
> Despite this, my tests so far have shown the scaled congestion control
> to be faster.

This is a good idea.  Destination host/net unreachable isn't much of a
drop...

> This could probably use some tweaking. For example, it may be better to
> calculate the packet receipt ratio as some kind of moving average rather
> than a lifetime average, so it can react more quickly as network
> conditions change.
>
> This change affects normal port scans too. I hope that it will only
> speed them up, but I haven't tested it much. So remember to try some
> port scans too and report if there's anything out of the ordinary.
>
> Here are some good test scans to run.
>
>       nmap -n -sP --send-ip 192.168.0.0/24
>       nmap -n -sP -PS --unprivileged host
>       nmap -n -sP -PS -T4 host/24
>       nmap -n -sP -PA1 -PS22,80,113 -PE -PM host
>
> Many thanks to all intrepid Nmap testers!
>
> David Fifield

I've been poking at the SVN code for a few hours now and I've gotten some
odd results.

Here is an example:

nmap -sP -P A135,139,445,3389 -n -v -v -d3 -T5 --min-parallelism X
- --max-parallelism Y --excludefile <a file> -oA <file>
a.b.0.0/16 c.d.0.0/16 e.f.0.0/16 --packet-trace

I started three copies of this scan with 64-64, 64-2048, and 2048-2048 for
the parallelism range.  The results all finished within about 1 second of
each other.  When I look at the output, it is apparent that they all
crashed within about 1 second of each other.  The really odd thing is this
happened 33 minutes in for each scan.  Even without a crash, 33 minutes is
a lot longer than these parallelism windows were taking before.

All the outputs ended with similar lines (but not the same hosts):

   c.d.52.164: 0/1/0/0/0/3 2048.00/75/0 116015/-1/-1
   c.d.52.165: 0/1/0/0/0/3 2048.00/__EOF__

Note that __EOF__ is really the end of the file, not text in the file.

I'll get you a stack trace later today if I manage to get out from under
several other projects I have going on.

Also, I love you cc-graph.sh script but it doesn't scale to huge scans.  It
errors out with:

awk: cmd. line:1: (FILENAME=david_new_1a.nmap FNR=1916551) fatal: print to
"standard output" failed (No space left on device)
flexmap/nmap/testing-new/cc-graph.sh: line 16: echo: write error: No space
left on device
awk: cmd. line:1: (FILENAME=david_new_1a.nmap FNR=78162) fatal: print to
"standard output" failed (No space left on device)
libplot error: output stream jammed
graph: error: could not close graphing device
Graph saved to ./david_new_1a.svg.

I suspect that it is trying to write too much to 'one line' of stdout.  I'm
not an awk ninja like you but I'm pretty sure I've deciphered what you're
doing with awk so if you want me to convert it to a short perl script let
me know.

Brandon


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFG3zPJqaGPzAsl94IRApYYAKCd1insFEDxrDfJhbM8TA9+ZW8JHQCgww9h
QGgEJLxvR6917efbrJekR+U=
=jRWF
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org