Re: New option: --min-rate for minimum-rate scanning

[David Fifield <david () bamsoftware com>]

On Mon, Mar 31, 2008 at 04:12:44AM +0000, Brandon Enright wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sun, 30 Mar 2008 20:44:15 -0600 or thereabouts David Fifield
> <david () bamsoftware com> wrote:
> ...snip...
> > That is true, but if the Linux hosts finish faster (for whatever
> > reason) and then the Windows hosts have to finish scanning at a
> > slower rate, that will bring the overall average down. If you run
> > with -d and use the run-time interaction feature by hitting Enter
> > during a scan, you can see a live estimate of the current scanning
> > rate. You might see it really fast at the beginning and slow down at
> > the end.
> >
> > > I'm happy to try any patch, Nmap command, or network size (up to
> > > when Nmap runs out of memory at around /17) so feel free to ask or
> > > patch away.
> >
> > Would you run the tests again with "--max-retries 0"? That will
> > eliminate the doAnyOutstandingRetransmits slowdown.
>
> Here we go again, this time with --max-retries 0 like so:
>
> nmap --min-rate 100000 --min-hostgroup 256 --max-retries 0 -P0 -n -d -v -p- <targets>
>
> These are all local machines.  Multiple scans against other machines
> were consistent with these so I've only included these three scans:
>
> Linux Box:
> Overall sending rates: 89643.44 packets / s, 3944311.23 bytes / s.
> Final times for host: srtt: 165 rttvar: 2  to: 100000
>
> Windows Box:
> Overall sending rates: 18712.29 packets / s, 823340.93 bytes / s.
> Overall sending rates: 18712.29 packets / s, 823340.93 bytes / s.
>
> Nothing:
> Overall sending rates: 14538.09 packets / s, 639675.90 bytes / s.
> Final times for host: srtt: -1 rttvar: -1  to: 1000000
>
> Local /25:
> Overall sending rates: 15573.42 packets / s, 685230.30 bytes / s.
>
> doAnyOutstandingRetransmits was certainly a factor.  It seems something
> else though is taking up most of the time.

There are actually a ton of places where the entire list of outstanding
probes is traversed. This is especially true because there are a lot of
places where list::size is called (grep scan_engine.cc for "listsz ="),
and in libstdc++ list::size is O(n):

http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt07ch16.html#sequences.list.size

Maybe the Linux boxes are sending resets for closed ports, which drops
the probes out of probes_outstanding and keeps the list small. If the
Windows boxes drop the request, the probes have to time out and they
stay in the list a long time, making it longer. That would also make
sense when scanning addresses that aren't connected.

Can you send the output of

nmap --min-rate 100000 --min-hostgroup 256 --max-retries 0 -P0 -n -d3 -p- | grep -E "^(\*\*TIMING|   )"

for scans against a fast Linux host, a slow Windows host, and the
unconnected netblock? (Of course you can just run the grep against a -d4
log.) You could send the raw log file but it's likely to be big.

For me, running such a command against a reset-sending Linux host gives

**TIMING STATS** (1.0050s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 50/*/*/*/*/* 99.05/75/* 100000/70/4
   192.168.0.X: 50/63585/0/50/0/0 99.05/75/0 100000/70/4
**TIMING STATS** (1.0180s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 50/*/*/*/*/* 99.56/75/* 100000/68/0
   192.168.0.X: 50/63535/0/50/0/0 99.56/75/0 100000/68/0
**TIMING STATS** (1.0340s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 50/*/*/*/*/* 100.07/75/* 100000/69/1
   192.168.0.X: 50/63485/0/50/0/0 100.07/75/0 100000/69/1

while running it against a packet-dropping Windows host gives

**TIMING STATS** (1.0040s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 636/*/*/*/*/* 76.01/75/* 100000/372/567
   192.168.0.Y: 636/62535/0/687/51/0 76.01/75/0 100000/231/314
**TIMING STATS** (1.0120s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 650/*/*/*/*/* 76.01/75/* 100000/372/567
   192.168.0.Y: 650/62485/0/686/36/0 76.01/75/0 100000/231/314
**TIMING STATS** (1.0220s): IP, probes active/freshportsleft/retry_stack/outstan
ding/retranwait/onbench, cwnd/ccthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 650/*/*/*/*/* 76.01/75/* 100000/372/567
   192.168.0.Y: 650/62435/0/700/50/0 76.01/75/0 100000/231/314

where the fourth number in the "*/*/*/*/*/*" part of the per-host line
is the number of outstanding probes. You can see it trends much higher
against the Windows host. Be aware that -d4 will probably slow down the
scan too.

I think we can reduce the negative effect of having a lot of outstanding
probes through code changes.

Anyone following this conversation, please note that these issues only
matter at really high packet rates. If you use reasonable arguments to
--min-rate (or don't use that option at all) it won't affect you.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org