Top ports and -p switch extension patchset

[doug () hcsw org]

Hi nmap-dev!

This collection of patches I am proposing greatly increases the capabilities
of the -p switch, lays the groundwork for a system that optimises the set
of ports scanned in the default/fast scans, and adds an convenient, flexible way
to trade-off port scan coverage for a reduction in run time. I have been maintaining
parts of this patchset for several years now and this document is largely just an
organised collection of associated documentation.

The remainder of this document is divided up into sections. I have committed the
patches to nmap-exp/soc07/nmap/ and would really appreciate any comments, suggestions,
and/or testing - but please read this entire document carefully first. I especially
stress the "proposed further extension" and "concerns" sections.

Best,

Doug



Top ports and -p switch extension patchset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

o New nmap-services file format
  o PROPOSED FURTHER EXTENSION
o New port list primitives
o -p switch extensions
  o Known ports using []
  o Wildcard port names
o -p switch test suite
o CONCERNS
o Summary




New nmap-services file format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This patch changes how Nmap parses and processes the nmap-services file. If the
services file that Nmap uses is in the traditional format (for instance when using
/etc/services) then the default scanning behaviour is identical to previous Nmap
versions.

In other words, until the nmap-services file is changed, this patch does not change
Nmap with respect to service naming. We still need to collect data and create a
new format nmap-services file.

An example of the old format:

http              80/tcp     # World Wide Web HTTP
http              80/udp     # World Wide Web HTTP

An example of the new format:

http              80/tcp   3080/10083   # World Wide Web HTTP
http              80/udp      1/2788    # World Wide Web HTTP


The new format gives us an arbitrarily granular expression of the relative
frequency of discovering a service on each port and is meant to represent

<Number found open>/<Number surveyed>

For instance, in the above snippet, port 80/tcp was seen open on about 30%
(ratio of about .3) total hosts and as such is much more common, relatively
speaking, than 80/udp.

New format lines can omit an A/B ratio (like old format lines). They default to
having a ratio of 0. If there are no new format lines we consider the file to
be old format

In all cases, protocol scanning behaves as before (it has its own file).



  PROPOSED FURTHER EXTENSION
  ~~~~~~~~~~~~~~~~~~~~~~~~~~

  As version scanning demonstrates, protocols are only very loosely related
  to port numbers. I think it could be very useful to have multiple protocols share
  the same port number. We could do this with multiple lines having the same port
  numbers but different service names and different frequency distributions.

  In the event that 2 common services share a port this could be an excellent
  way to note this information. In fact I suspect that several ports are commonly
  described as having INCORRECT services when they aren't version scanned because of
  out-of-date/incorrect/etc entries in /etc/services and nmap-services.

  When reporting we could opt to show only the most common or possibly show some sort
  of probability distribution. I could even see using this information as a complement
  to the ports directive in the nmap-services file for determining the order of probes
  to be applied.

  This extension suggests we perform service scans against our survey targets when
  creating the new format file instead of using faster SYN scans. Service scans would
  also eliminate the suprisingly large amount of "noise" due to network artifacts,
  misconfigurations, etc. If service scan verifies the service we know that somebody,
  at least superficially, is processing that service on the given port.

  Adopting this extension would mean that we might not always report the IANA reserved
  protocol for a given port, rather what are scans indicate people are actually
  commonly running on that port.



New port list primitives
~~~~~~~~~~~~~~~~~~~~~~~~

In order to allow Nmap users more control over which ports to scan we provide
2 new command line options:

--port-ratio <decimal number between 0 and 1>

  Scans all ports in the nmap-services file with a ratio greater than the
  number specified as the argument.

--top-ports <integer of 1 or greater>

  Scans the N highest-ratio ports found in the nmap-services file.

If the services file is of the new format (contains at least one line in
the new format) then the new default scanning behaviour is as follows:

Default: Scans ports found by our scanning to be open on at least 1% of
         machines: --port-ratio 0.01 (number may need tuning)
Fastscan: Scans the 100 highest ratio ports: --top-ports 100 (again, tuning)

I'm not sure exactly which will end up being the most useful but these 2
primitives should allow us completely arbitrary granularity in specifying
how "common" a port needs to be before we scan it. Notice that the normal scan
vs the fast scan use different primitives. The default scan uses a fixed
probability of the port being open and the fast scan uses a fixed number of
ports to scan. With the current settings we should hope that there are more
than 100 ports with ratio >= 0.01 or else the fast scan will actually scan
more ports than the normal scan!

Notice that you can combine -p and --port-ratio/--top-ports. Especially with
the -p switch extensions described in the next section, this is a powerful and
subtle addition to the -p switch DSL. Nmap will apply the --port-ratio/--top-ports
filter to only the ports specified in the -p argument, not the entire file.

For example the following will scan the 5 most common ports between 1 and 100:

nmap --top-ports 5 -p "1-100" target

You (still) can't use -F with explicit port selection. It makes more sense
to manually set a top-ports or port-ratio value.



-p switch extensions
~~~~~~~~~~~~~~~~~~~~

Specifying which ports to scan using nmap has always felt convenient and
natural. The -p switch is the most common way of doing this although you
might also omit the -p switch and use the default scan or the fast scan (-F).

I always use the -p switch because I prefer to know exactly what I'm scanning
and because the -p switch is just so damn flexible. In fact once you examine and
use enough of these systems you start to recognise them for little programming
languages on their own. The -p switch, along with regular expressions, SQL queries,
unix shells, etc, are examples of "Domain Specific Languages".

The -p switch DSL has a very simple interface to Nmap. The string passed to the
function getpts() (usually from -p) is a "program" in our DSL that returns 3 port
vectors telling Nmap what to scan:

(<TCP ports to scan>, <UDP ports to scan>, <Protocols to scan>)

Here is an example of an expression in the -p switch DSL:

"-20,U:400-403,T:65534-"

I have been trying to get the following 2 -p switch features included in Nmap for
several years. I think this is at least the 3rd port of the code to newer versions
of Nmap. None of the -p switch code is different from my original patch except for
updates to the changing Nmap code base and an enhancement related to the port list
primitives described above.


  Known ports using []
  ~~~~~~~~~~~~~~~~~~~~

  This feature lets you use [] brackets to specify "known ports". That is
  ports appearing in the nmap-services file for old nmap-services file format
  behaviour or ports above your specified ratio/number for new format behaviour
  (see the new port list primitives section). For instance,

  nmap -p [40000-] target

  tells Nmap to scan all ports mentioned in the nmap-services file above 40000
  for an old format nmap-services file. For a new format file this tells Nmap
  to scan all ports that have entries above 40000 - even if their ratio is 0.

  Before this patch, the behaviour of the Nmap default scan is specified in
  a 60 line C function. But with it the traditional behaviour (which we have to
  do if we can't find a new format file or have to use /etc/services) is now
  simply the following string: "1-1024,[1025-]".

  BE CAREFUL ABOUT SHELL EXPANSIONS!!!
  Some shells (like tcsh) treat square brackets as special characters.
  Always quote the argument to -p if uncertain.

  nmap -p '[40000-]' target



  Wildcard port names
  ~~~~~~~~~~~~~~~~~~~

  This new feature of the -p switch DSL lets you specify ports/protocols
  by name and even use wildcards to specify ports/protocols.

  If you can't remember which port, say, snmp runs on, you can use the
  easier to remember command:

  nmap -sSU -p snmp target

  Wildcards (* and ?) are also supported. This tells Nmap to scan all
  services beginning with "http":

  nmap -p http* target

  With a certain nmap-services file it is equivalent to

  nmap -p 80,280,443,591,593,8000,8080,8443 target

  And you can also combine wildcard -p options with --top-ports/--port-ratio.
  For example the following tells Nmap to scan the 3 most popular protocols
  beginning with http:

  nmap -p http* --top-ports 3 target

  BE CAREFUL ABOUT SHELL EXPANSIONS!!!
  If you are trying to match the services nmsp (537/tcp) and nms (1429/tcp)
  and you execute the command
 
  ./nmap -p nm* host
 
  You will see
 
  Found no matches for the service mask 'nmap' and your specified protocols
  QUITTING!
 
  This is because nm* was expanded to the name of the binary file nmap in
  the current directory by your shell. When unsure, quote your port strings:
 
  ./nmap -p 'nm*' host






-p switch test suite
~~~~~~~~~~~~~~~~~~~~

Partly in order to comprehensively test my -p switch additions and partly because
so many subtle bugs have been discovered in the -p switch in the past, last summer
I thought it might be worthwhile to develop a test suite that forces us to make
axiomatic assumptions and instantly verify that further enhancements don't violate
them. I updated and greatly extended the test suite for this patch release.

The test suite is a common lisp program that performs 87 tests that exercise,
to my knowledge, every feature of the -p switch. I chose lisp as the language to
write the suite because of its flexibility. Where other languages are rigid and
specific, lisp is fluid and generic. Other languages are really just skins on top
of lisp with a lot of inconvenient syntax in the way. But I digress. :)

If anyone thinks of any additional tests I'd be happy to add them. To use the test
suite you must have a common lisp environment installed. I use cmucl. You need the CL
"port" library. There are a few variables at the top of the lisp file you need to
update and a line in nmap.cc you need to uncomment:

  // Uncomment the following line to use the common lisp port spec test suite
  //printf("port spec: (%d %d %d)\n", ports->tcp_count, ports->udp_count, ports->prot_count); exit(0);

I will put the test suite into SVN soon (or maybe my blog). Here is its output:

# lisp -quiet -load p-switch-tests.lisp

OLD FORMAT SCANS

---------TCP scans----------
Test #1 'nmap -sS -p - targ' PASSED!
Test #2 'nmap -sS -p 0- targ' PASSED!
Test #3 'nmap -sS -p -20 targ' PASSED!
Test #4 'nmap -sS -p 0-20 targ' PASSED!
Test #5 'nmap -sS -p 823,39,19,99,100 targ' PASSED!
Test #6 'nmap -sS -p -2,10-20,400,65534- targ' PASSED!
Test #7 'nmap -sS -p -2,10-20,T:400,65534-,U:200 targ' PASSED!
Test #8 'nmap -sS -p -2,10-20,U:400,65534- targ' PASSED!
---------UDP scans----------
Test #9 'nmap -sU -p 10-20 targ' PASSED!
Test #10 'nmap -sU -p -20 targ' PASSED!
Test #11 'nmap -sU -p 0-20 targ' PASSED!
Test #12 'nmap -sU -p 0-20,T:100 targ' PASSED!
Test #13 'nmap -sU -p 0-20,U:100 targ' PASSED!
Test #14 'nmap -sU -p U:0-20,100 targ' PASSED!
---------Protocol scans----------
Test #15 'nmap -sO -p - targ' PASSED!
Test #16 'nmap -sO -p 10-20 targ' PASSED!
Test #17 'nmap -sO -p -20 targ' PASSED!
Test #18 'nmap -sO -p 255- targ' PASSED!
---------TCP+UDP scans----------
Test #19 'nmap -sSU -p U:-2,10-20,400,65534- targ' PASSED!
Test #20 'nmap -sSU -p T:-2,10-20,400,65534- targ' PASSED!
Test #21 'nmap -sSU -p -2,10-20,400,65534- targ' PASSED!
Test #22 'nmap -sSU -p -2,10-20,U:400,65534- targ' PASSED!
Test #23 'nmap -sSU -p -2,10-20,T:400,65534- targ' PASSED!
Test #24 'nmap -sSU -p 10-20,U:400-403,T:65534- targ' PASSED!
---------[] scans------------
Test #25 'nmap -sSU -p [-] targ' PASSED!
Test #26 'nmap -sSU -p [-],1234 targ' PASSED!
Test #27 'nmap -sSU -p [-],T:1234 targ' PASSED!
Test #28 'nmap -sSU -p [T:-] targ' PASSED!
Test #29 'nmap -sSU -p [T:-],1234 targ' PASSED!
WARNING:  Duplicate port number(s) specified.  Are you alert enough to be using Nmap?  Have some coffee or Jolt(tm).
Test #30 'nmap -sSU -p T:1234,[-] targ' PASSED!
Test #31 'nmap -sSU -p T:1234,[U:-] targ' PASSED!
Test #32 'nmap -sSU -p [-1024] targ' PASSED!
Test #33 'nmap -sSU -p [1024-] targ' PASSED!
Test #34 'nmap -sSU -p [1024-,1234] targ' PASSED!
Test #35 'nmap -sO -p [-] targ' PASSED!
Test #36 'nmap -sO -p [-],252 targ' PASSED!
WARNING:  Duplicate port number(s) specified.  Are you alert enough to be using Nmap?  Have some coffee or Jolt(tm).
Test #37 'nmap -sO -p [-],136 targ' PASSED!
---------wildcard scans------------
Test #38 'nmap -sSU -p rsync targ' PASSED!
Test #39 'nmap -sS -p http targ' PASSED!
Test #40 'nmap -sU -p http targ' PASSED!
Test #41 'nmap -sSU -p http targ' PASSED!
Test #42 'nmap -sSU -p T:http targ' PASSED!
Test #43 'nmap -sSU -p http* targ' PASSED!
Test #44 'nmap -sSU -p U:http*,T:http*,rsync targ' PASSED!
Test #45 'nmap -sSU -p vnc-http* targ' PASSED!
Test #46 'nmap -sSU -p vnc-http-? targ' PASSED!
Test #47 'nmap -sSU -p *-mgmt targ' PASSED!
Test #48 'nmap -sO -p ipv6* targ' PASSED!
Test #49 'nmap -sSU -p [200-300],T:http,[900,U:1024-],T:f*l*ma?er targ' PASSED!
---------BAD/STRANGE ARGS----------
WARNING:  Duplicate port number(s) specified.  Are you alert enough to be using Nmap?  Have some coffee or Jolt(tm).
Test #50 'nmap -sS -p 1,1 targ' PASSED!
Test #51 'nmap -sU -p T:1-100 targ' PASSED!
Test #52 'nmap -sT -p U:1-100 targ' PASSED!
Test #53 'nmap -sO -p T:50,U:100 targ' PASSED!

NEW FORMAT SCANS

TestNew #54 'nmap --port-ratio 0.01 targ' PASSED!
TestNew #55 'nmap --top-ports 100 targ' PASSED!
TestNew #56 'nmap --port-ratio 0.01 -sU targ' PASSED!
TestNew #57 'nmap --top-ports 100 -sU targ' PASSED!
TestNew #58 'nmap --port-ratio 0.01 -sSU targ' PASSED!
TestNew #59 'nmap --top-ports 100 -sSU targ' PASSED!
TestNew #60 'nmap --top-ports 1 -sSU targ' PASSED!
TestNew #61 'nmap --top-ports 1 -sS targ' PASSED!
TestNew #62 'nmap --top-ports 1 -sU targ' PASSED!
TestNew #63 'nmap --top-ports 20 -sS targ' PASSED!
TestNew #64 'nmap --top-ports 65536 -sS targ' PASSED!
TestNew #65 'nmap --top-ports 8 -sU targ' PASSED!
TestNew #66 'nmap --top-ports 9 -sU targ' PASSED!
TestNew #67 'nmap --port-ratio 0.25 -sSU targ' PASSED!
TestNew #68 'nmap --port-ratio 0.25 -sS targ' PASSED!
TestNew #69 'nmap --port-ratio 0.25 -sU targ' PASSED!
TestNew #70 'nmap --port-ratio 0.8 -sSU targ' PASSED!
TestNew #71 'nmap --port-ratio 0.1 -sU targ' PASSED!
TestNew #72 'nmap --port-ratio 0.01 -sU targ' PASSED!
TestNew #73 'nmap --top-ports 2 -p 80 targ' PASSED!
TestNew #74 'nmap --top-ports 20 -p * targ' PASSED!
TestNew #75 'nmap --top-ports 21 -p * targ' PASSED!
TestNew #76 'nmap --top-ports 16 -p *http* targ' PASSED!
TestNew #77 'nmap --top-ports 100 -p filemaker targ' PASSED!
TestNew #78 'nmap --top-ports 100 -p filemaker -sU targ' PASSED!
TestNew #79 'nmap --top-ports 100 -p filemaker -sSU targ' PASSED!
TestNew #80 'nmap --top-ports 7 -p *http* -sU targ' PASSED!
TestNew #81 'nmap --top-ports 7 -p *http* -sSU targ' PASSED!
TestNew #82 'nmap --top-ports 100 -p *http* -sSU targ' PASSED!
TestNew #83 'nmap --top-ports 2 -p -100 -sS targ' PASSED!
TestNew #84 'nmap --top-ports 2 -p -100 -sU targ' PASSED!
TestNew #85 'nmap --top-ports 2 -p -100 -sSU targ' PASSED!
TestNew #86 'nmap --top-ports 21 -p [-] -sSU targ' PASSED!
TestNew #87 'nmap --port-ratio 0.1 -sSU -p *http* targ' PASSED!


87/87 tests PASSED.




CONCERNS
~~~~~~~~

o This patch, when combined with a new format nmap-services file, will change the
  ports that Nmap scans with a default scan and with a fast scan. Currently deployed
  scripts that rely on certain ports being scanned might have to be changed.

o Collecting meaningful data for this sort of DB (nmap-services) is hard! I am not
  convinced there is such a thing as a "typical port distribution" for hosts on today's
  internet. And what's more, with the pace of technological development as it is, I'm not
  sure that any "typical distribution" will stay "typical" for long.

o Perhaps the biggest issue I see with the new default/fast scan behaviour is that
  it becomes difficult to know if a given port was scanned for when looking at a scan's
  results. Was protocol XYZ included in a fast scan with the services file shipped on
  some given date? It's not easy to know, especially if we plan on doing semi-frequent
  updates to this DB.




Summary
~~~~~~~

In summary I think this collection of patches is ready for inclusion into Nmap. The
-p switch patches are long overdue and are extremely useful on the command line
(quick, what port is NNTP?). The top ports patch is also ready and I don't think it
will interfere with current Nmap installs at all. If we do eventually create a decent
collection of data, then this patch and the port list primitives it offers, are powerful,
generic interfaces to Nmap for using this data.

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org