Nmap "port scan ping" algorithm improvements by David, Michael, Fyodor

[Fyodor <fyodor () insecure org>]

Hi.  I have some more svn logs I want to send, both because I want to
link to them in the CHANGELOG and also because they're interesting in
their own right.  I think nmap-dev has about 50 times as many
subscribers as nmap-svn, so many people probably haven't seen these.
They tell the story of an important improvement to Nmap internals
which you wouldn't really notice except that some of your scans may be
a lot faster.  I've pasted the detailed SVN logs by David Michael
Pattrick, and myself below.  Depending on your tastes, you might find
this interesting, or be bored to tears :).

-Fyodor

------------------------------------------------------------------------
r8754 | david | 2008-07-10 17:35:39 -0700 (Thu, 10 Jul 2008) | 9 lines
Changed paths:
   M /nmap/scan_engine.cc

Use rcvdtime, not NULL, as the received time when doing ping or port updates
from ICMP probes during a protocol scan (protoscanicmphack). I don't know why
it was NULL before, but that's wrong. It was probably never noticed because in
the case of a port update, all that happens is a failure to update the timing.
In the case of a ping probe, it would look like a dropped ping probe, but that
woudl be unlikely because protocol scans usually don't take very long. I
discovered it while testing code to allow ping probes to persist between host
discovery and port scanning.


------------------------------------------------------------------------
r8759 | david | 2008-07-10 23:12:38 -0700 (Thu, 10 Jul 2008) | 21 lines
Changed paths:
   M /nmap/Target.cc
   M /nmap/Target.h
   M /nmap/scan_engine.cc
   M /nmap/scan_engine.h

Save timing ping probes between calls to ultra_scan. This allows, for example,
to reuse an ACK ping probe from host detection during a SYN port scan. This can
greatly speed up a scan if the SYN scan finds only filtered ports.

One difficulty with implementing this is that not all ping probes are
appropriate for all scan types.
  nmap -PA -sU scanme.nmap.org
would cache the ACK ping probe and send ACK pings during the UDP scan. But the
pcap filter for the UDP scan doesn't catch TCP packets, so the replies would
not be noticed and they would show up as dropped pings. Likewise,
  nmap -PR -sS 192.168.0.1
would segfault when it tried to use an uninitialized Ethernet descriptor to
send an ARP ping during the SYN scan, which would use raw sockets.

To fix this I added a function pingprobe_is_appropriate that determines whether
a given ping probe is appropriate for the current scan type. If not, the
constructor for HostScanStats just erases the ping probe.

More types of ping probes could be made "appropriate." TCP timing pings work
during a UDP scan if only the pcap filter is expanded to include TCP packets.


------------------------------------------------------------------------
r8764 | fyodor | 2008-07-11 00:33:11 -0700 (Fri, 11 Jul 2008) | 1 line
Changed paths:
   M /nmap/scan_engine.cc

Port scan pings happen against heavily firewalled hosts and the
like when Nmap is not receiving enough responses to normal scan
to properly calculate timing variables and detect packet drops.
This patch increases the maximum per-target frequency of these
probes to one every 1.25 seconds rather than one every five
seconds


------------------------------------------------------------------------
r8784 | david | 2008-07-11 19:18:18 -0700 (Fri, 11 Jul 2008) | 42 lines
Changed paths:
   M /nmap/scan_engine.cc

This commit changes two separate but related things, which I found it
inconvenient to change separately.

The first change fixes a logical error in the storage of timing ping
probes. Each target contains a description of a timing ping probe, which
is stored in the two members
  probespec pingprobe;
  int pingprobe_state;
pingprobe is the probe itself, and pingprobe_state is the state of the
port that the probe was sent to (PORT_OPEN, PORT_CLOSED, etc.). A change
in the state of the port was a criterion used in deciding whether to
replace the current ping probe.

The problem with this was that pingprobe_state was used to hold a host
state, not a port state, during host discovery. Therefore it held a
value like HOST_DOWN or HOST_UP. This was fine as long as host discovery
and port scanning were separate, but now that timing pings are shared
between those phases the states were in confict: HOST_UP = 1 = PORT_CLOSED.
THis was fixed by using a value of PORT_UNKNOWN during host discovery.

The second change redoes how timing ping probes are replaced. There is
now an order of preference for timing ping probe types, defined by the
function pingprobe_score (and pingprobe_is_better, which calls it). The
order I have defined, from highest preference to lowest, is
        ARP
        Raw TCP (not SYN to an open port)
        UDP, IP protocol, or ICMP
        Raw TCP (SYN to an open port)
        TCP connect
        Anything else
The port state is considered only in raw TCP SYN to an open port, which
is given a lower preference because of the possibility of SYN flooding.

Better ping probes supersede worse ping probes. So in
        nmap -PS -sA scanme.nmap.org
the ping probe will be SYN to port 80 after host discovery, but then
will change to ACK to an unfiltered port during port scanning. In
        nmap -PA -sS scanme.nmap.org
the ping probe will be ACK to port 80 after host discovery and will
remain that way during port scanning because SYN to an open port is a
worse ping probe. Run with -d2 to see when timing pings change.


------------------------------------------------------------------------
r8893 | michael | 2008-07-14 14:11:58 -0700 (Mon, 14 Jul 2008) | 18 lines
Changed paths:
   M /nmap/CHANGELOG
   M /nmap/scan_engine.cc

nged the order that pingprobes are favored in, the order is now:
-Raw TCP (not SYN to an open port)
-ICMP information queries (echo request, timestamp request, netmask req)
-ARP
-Raw TCP (SYN to an open port)
-UDP, IP protocol, or other ICMP
-TCP connect
-Anything else

The order used to be
-ARP
-Raw TCP (not SYN to an open port)
-UDP, IP protocol, or ICMP
-Raw TCP (SYN to an open port)
-TCP connect
-Anything else

------------------------------------------------------------------------
r8946 | michael | 2008-07-16 11:51:46 -0700 (Wed, 16 Jul 2008) | 1 line
Changed paths:
   M /nmap/scan_engine.cc

Prevented protocol timing probes from being sent out during udp or tcp scans

------------------------------------------------------------------------
r9244 | david | 2008-07-31 17:08:47 -0700 (Thu, 31 Jul 2008) | 42 lines
Changed paths:
   M /nmap/scan_engine.cc

Simplify and fix the logic surrounding the handling of host discovery
probes, especially IP protocol probes.

Previously if IP protocol ping (-PO) was used anywhere in a host
discovery scan, any response was treated as a protocol response. (The
handlers for other response types had an explicit check for this.) This
means that if you did

nmap -PS -PO

and got back a SYN/ACK in response to the -PS probe, it would be marked
with a reason of proto-response rather than syn-ack. Now, because the IP
protocol response handler matches so broadly, it is given the last
chance at handling a response, only if no interpretation makes sense.
Now the aforementioned scan will give a reason of syn-ack.

The old behavior was not only misleading with respect to reasons, it had
a minor and subtle bug. Consider the following packet trace:

SENT (2.0990s) TCP 192.168.0.21:42205 > target:25 S ttl=40 id=39342 iplen=44  seq=114128202 win=1024 <mss 1460>
SENT (2.2560s) TCP 192.168.0.21:42205 > target:53 S ttl=40 id=51247 iplen=44  seq=114128202 win=1024 <mss 1460>
SENT (2.3280s) TCP 192.168.0.21:42206 > target:25 S ttl=37 id=31111 iplen=44  seq=114062667 win=2048 <mss 1460>
RCVD (2.3530s) TCP target:53 > 192.168.0.21:42205 SA ttl=51 id=0 iplen=44  seq=4159224453 win=5840 ack=114128203 <mss 
1460>
ultrascan_host_probe_update called for machine target state UNKNOWN -> HOST_UP (trynum 1 time: 25123)
Ultrascan DROPPED probe packet to target detected
Changing ping technique for target to tcp to port 25; flags: S

Why is the received packet marked as a drop? And why is the ping
technique change to SYN to port 25 when the response came back from port
53? The reason is that the IP protocol response handler caught the probe
and decided it was in response to one of the sent TCP probes--any of the
TCP probes. It selected the probe to port 25 essentially at random and
used that as the relevant probe. The result is that a drop is wrongly
recorded (slowing down the scan), and a worse than useless ping probe is
used (worse than useless because it will cause another drop any time
it's used).

I found this while trying to emulate PortBunny's default ping scan,
which is
-PS80,25,22,443,21,113,23,53,554,3389,445 -PA3333,11 -PE -PP -PU161,162 -PO51
though not in the same order Nmap uses.




_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org