Re: massping-migration and other dev testing results

[David Fifield <david () bamsoftware com>]

On Wed, Sep 12, 2007 at 06:21:19AM +0000, Brandon Enright wrote:
> - --min-hostgroup is one of those options that the man explains the high
> level function of but doesn't properly capture how/what it really does.  A
> line in the man something like "this option only affects port scanning
> hosts.  A ping sweep (-sP) and host detection are not affected." might help.

You're right. I've added something to that effect.

> > > Since this value is already so large, using the value from
> > > --min-hostgroup is probably not a good idea.  Perhaps another option
> > > like --min-ping-group.
> >
> > Hmm.
>
> Although the results below are not totally convincing, I still like this
> idea.  Hey, Nmap should let you shoot yourself in the foot if you really
> want to, right? ;-p

Sounds like a good idea, though I won't make it a priority. It would
certainly make testing and experimenting with different ping group sizes
easier.

> > > david_mpm_r5824a.nmap:
> > > # Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11790
> > > hosts up) scanned in 288.591 seconds
> > >
> > > david_nmap_r5824a.nmap:
> > > # Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11936
> > > hosts up) scanned in 287.305 seconds
> > >
> > > david_mpm_r5824b.nmap:
> > > # Nmap done at Wed Sep 12 00:17:31 2007 -- 186368 IP addresses (15628
> > > hosts up) scanned in 2640.259 seconds
> > >
> > > david_nmap_r5824b.nmap:
> > > # Nmap done at Wed Sep 12 00:49:08 2007 -- 186368 IP addresses (15901
> > > hosts up) scanned in 4536.876 seconds
>
> > There's a script called host-list-compare.py in the
> > nmap-massping-migration branch. It takes two .nmap log files and prints
> > out which hosts are up in the first that aren't in the second, and vice
> > versa. I'm curious to know whether the mpm runs just missed a bunch of
> > hosts, or they missed a bunch of hosts but found a bunch of others.
> > Please run that script on your logs for these most recent scans and
> > report back the two lines that say "N extra hosts in X.nmap".
>
> Okay, just the comparisons between MPM and NMAP:
>
> ====
> 176 extra hosts in david_mpm_r5824b.nmap:
> 449 extra hosts in david_nmap_r5824b.nmap:
> ====
>
> Very interesting results.  The 179 is about 1/3 are wireless hosts and the
> rest a random sampling of hosts.  The 449 are almost exclusively a
> particular build/deployment/image of Windows that we have distributed all
> over campus. When I get a chance I'll try to figure out why that build of
> Windows needs a slower scan (or what it is about the networking gear that we
> typically use with this build).

Hopefully that's what the problem is--that the scan just completes too
quickly for all the hosts to respond. That would indicate that something
less than -T5 is needed.

> ====
> 1729 extra hosts in david_mpm_r5824a.nmap:
> 1875 extra hosts in david_nmap_r5824a.nmap:
> ====
> These are a random sampling of hosts all from a large set of the same
> subnets.  If I had to describe the difference, I'd say that hosts in those
> networks have a 50/50 chance of showing up.  Sometimes they show up in one
> scan and not the other.

This I understand. If the parallelism is too high for Nmap to handle,
then the losses should be more or less random, with roughly equal
numbers in both scans.

> > And then, if you wouldn't mind, grep through the logs for a few of the
> > addresses that were missed and see what the differences are. It's
> > significant if, say, a response is received late (after the end of a
> > ping group, which --packet-trace would show) versus not received at all.
>
> I'm happy to do this but it will have to wait until tomorrow.
>
> > I don't know what the hosts on your network are like. During my testing
> > for the migration, I found that there are some weird hosts out there
> > that consistently take 30 seconds or more to respond to a probe. If
> > those 30 seconds don't elapse before the ping group is finished, the
> > host is marked down. There's not much you can do about those except use
> > a larger ping group or less aggressive timing parameters.
> >
> > I'm not suggesting that's what's going on with your scans, it's just one
> > explanation I thought of. Another is that maybe the more robust
> > congestion window actually is oversaturating the line enough that
> > responses are being dropped without being detected.
>
> The following statements ignore wireless:
>
> In the general case, I don't think this is possible.  There are a small
> handful of routers on the network who's CPU can't handle the huge number of
> flows and drop packets.  For the most part though, the network is a
> state-of-the-art multi-gigabit beast with 0 packet loss.
>
> I agree that lose is occurring somewhere, I just don't think it is the
> fault of the network.  I've seen other tools that use libpcap report
> dropped packets once in a while.  Is it possible that Nmap either isn't
> getting the packets out and they are being dropped by libpcap or that the
> responses are getting dropped on the way in?

To investigate this, I added a function to the massping migration branch
that prints the number of dropped packets reported by libpcap. With -d2,
it's called once per invocation of ultra_scan, so roughly once per 4096
hosts during host discovery.

Please run your mpm 'b' scan again with -T5 and see if there are any
drops (the stats lines start with "pcap stats:"). Then run it with -T3
and see if more hosts are detected in the (presumably) longer time the
scan takes.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org