Nmap Development mailing list archives
Re: Top ports and -p switch extension patchset
From: Luis Martin Garcia <luis.mgarc () gmail com>
Date: Wed, 16 May 2007 12:20:04 +0100
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I am not a nmap developer but as a user I think the proposed ideas are great. I think you did a very good job here Doug, thank you. I hope your proposal goes through and your patches get included in the next release. Regards, Luis. doug () hcsw org wrote:
Hi nmap-dev! This collection of patches I am proposing greatly increases the
capabilities
of the -p switch, lays the groundwork for a system that optimises the set of ports scanned in the default/fast scans, and adds an convenient,
flexible way
to trade-off port scan coverage for a reduction in run time. I have
been maintaining
parts of this patchset for several years now and this document is
largely just an
organised collection of associated documentation. The remainder of this document is divided up into sections. I have
committed the
patches to nmap-exp/soc07/nmap/ and would really appreciate any
comments, suggestions,
and/or testing - but please read this entire document carefully first.
I especially
stress the "proposed further extension" and "concerns" sections. Best, Doug Top ports and -p switch extension patchset ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ o New nmap-services file format o PROPOSED FURTHER EXTENSION o New port list primitives o -p switch extensions o Known ports using [] o Wildcard port names o -p switch test suite o CONCERNS o Summary New nmap-services file format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This patch changes how Nmap parses and processes the nmap-services
file. If the
services file that Nmap uses is in the traditional format (for instance
when using
/etc/services) then the default scanning behaviour is identical to
previous Nmap
versions. In other words, until the nmap-services file is changed, this patch
does not change
Nmap with respect to service naming. We still need to collect data and
create a
new format nmap-services file. An example of the old format: http 80/tcp # World Wide Web HTTP http 80/udp # World Wide Web HTTP An example of the new format: http 80/tcp 3080/10083 # World Wide Web HTTP http 80/udp 1/2788 # World Wide Web HTTP The new format gives us an arbitrarily granular expression of the relative frequency of discovering a service on each port and is meant to represent <Number found open>/<Number surveyed> For instance, in the above snippet, port 80/tcp was seen open on about 30% (ratio of about .3) total hosts and as such is much more common, relatively speaking, than 80/udp. New format lines can omit an A/B ratio (like old format lines). They
default to
having a ratio of 0. If there are no new format lines we consider the
file to
be old format In all cases, protocol scanning behaves as before (it has its own file). PROPOSED FURTHER EXTENSION ~~~~~~~~~~~~~~~~~~~~~~~~~~ As version scanning demonstrates, protocols are only very loosely related to port numbers. I think it could be very useful to have multiple
protocols share
the same port number. We could do this with multiple lines having the
same port
numbers but different service names and different frequency
distributions.
In the event that 2 common services share a port this could be an
excellent
way to note this information. In fact I suspect that several ports
are commonly
described as having INCORRECT services when they aren't version
scanned because of
out-of-date/incorrect/etc entries in /etc/services and nmap-services. When reporting we could opt to show only the most common or possibly
show some sort
of probability distribution. I could even see using this information
as a complement
to the ports directive in the nmap-services file for determining the
order of probes
to be applied. This extension suggests we perform service scans against our survey
targets when
creating the new format file instead of using faster SYN scans.
Service scans would
also eliminate the suprisingly large amount of "noise" due to network
artifacts,
misconfigurations, etc. If service scan verifies the service we know
that somebody,
at least superficially, is processing that service on the given port. Adopting this extension would mean that we might not always report
the IANA reserved
protocol for a given port, rather what are scans indicate people are
actually
commonly running on that port. New port list primitives ~~~~~~~~~~~~~~~~~~~~~~~~ In order to allow Nmap users more control over which ports to scan we
provide
2 new command line options: --port-ratio <decimal number between 0 and 1> Scans all ports in the nmap-services file with a ratio greater than the number specified as the argument. --top-ports <integer of 1 or greater> Scans the N highest-ratio ports found in the nmap-services file. If the services file is of the new format (contains at least one line in the new format) then the new default scanning behaviour is as follows: Default: Scans ports found by our scanning to be open on at least 1% of machines: --port-ratio 0.01 (number may need tuning) Fastscan: Scans the 100 highest ratio ports: --top-ports 100 (again,
tuning)
I'm not sure exactly which will end up being the most useful but these 2 primitives should allow us completely arbitrary granularity in specifying how "common" a port needs to be before we scan it. Notice that the
normal scan
vs the fast scan use different primitives. The default scan uses a fixed probability of the port being open and the fast scan uses a fixed number of ports to scan. With the current settings we should hope that there are more than 100 ports with ratio >= 0.01 or else the fast scan will actually scan more ports than the normal scan! Notice that you can combine -p and --port-ratio/--top-ports. Especially
with
the -p switch extensions described in the next section, this is a
powerful and
subtle addition to the -p switch DSL. Nmap will apply the
--port-ratio/--top-ports
filter to only the ports specified in the -p argument, not the entire file. For example the following will scan the 5 most common ports between 1
and 100:
nmap --top-ports 5 -p "1-100" target You (still) can't use -F with explicit port selection. It makes more sense to manually set a top-ports or port-ratio value. -p switch extensions ~~~~~~~~~~~~~~~~~~~~ Specifying which ports to scan using nmap has always felt convenient and natural. The -p switch is the most common way of doing this although you might also omit the -p switch and use the default scan or the fast scan
(-F).
I always use the -p switch because I prefer to know exactly what I'm
scanning
and because the -p switch is just so damn flexible. In fact once you
examine and
use enough of these systems you start to recognise them for little
programming
languages on their own. The -p switch, along with regular expressions,
SQL queries,
unix shells, etc, are examples of "Domain Specific Languages". The -p switch DSL has a very simple interface to Nmap. The string
passed to the
function getpts() (usually from -p) is a "program" in our DSL that
returns 3 port
vectors telling Nmap what to scan: (<TCP ports to scan>, <UDP ports to scan>, <Protocols to scan>) Here is an example of an expression in the -p switch DSL: "-20,U:400-403,T:65534-" I have been trying to get the following 2 -p switch features included
in Nmap for
several years. I think this is at least the 3rd port of the code to
newer versions
of Nmap. None of the -p switch code is different from my original patch
except for
updates to the changing Nmap code base and an enhancement related to
the port list
primitives described above. Known ports using [] ~~~~~~~~~~~~~~~~~~~~ This feature lets you use [] brackets to specify "known ports". That is ports appearing in the nmap-services file for old nmap-services file
format
behaviour or ports above your specified ratio/number for new format
behaviour
(see the new port list primitives section). For instance, nmap -p [40000-] target tells Nmap to scan all ports mentioned in the nmap-services file
above 40000
for an old format nmap-services file. For a new format file this
tells Nmap
to scan all ports that have entries above 40000 - even if their ratio
is 0.
Before this patch, the behaviour of the Nmap default scan is specified in a 60 line C function. But with it the traditional behaviour (which we
have to
do if we can't find a new format file or have to use /etc/services)
is now
simply the following string: "1-1024,[1025-]". BE CAREFUL ABOUT SHELL EXPANSIONS!!! Some shells (like tcsh) treat square brackets as special characters. Always quote the argument to -p if uncertain. nmap -p '[40000-]' target Wildcard port names ~~~~~~~~~~~~~~~~~~~ This new feature of the -p switch DSL lets you specify ports/protocols by name and even use wildcards to specify ports/protocols. If you can't remember which port, say, snmp runs on, you can use the easier to remember command: nmap -sSU -p snmp target Wildcards (* and ?) are also supported. This tells Nmap to scan all services beginning with "http": nmap -p http* target With a certain nmap-services file it is equivalent to nmap -p 80,280,443,591,593,8000,8080,8443 target And you can also combine wildcard -p options with
--top-ports/--port-ratio.
For example the following tells Nmap to scan the 3 most popular protocols beginning with http: nmap -p http* --top-ports 3 target BE CAREFUL ABOUT SHELL EXPANSIONS!!! If you are trying to match the services nmsp (537/tcp) and nms (1429/tcp) and you execute the command ./nmap -p nm* host You will see Found no matches for the service mask 'nmap' and your specified protocols QUITTING! This is because nm* was expanded to the name of the binary file nmap in the current directory by your shell. When unsure, quote your port
strings:
./nmap -p 'nm*' host -p switch test suite ~~~~~~~~~~~~~~~~~~~~ Partly in order to comprehensively test my -p switch additions and
partly because
so many subtle bugs have been discovered in the -p switch in the past,
last summer
I thought it might be worthwhile to develop a test suite that forces us
to make
axiomatic assumptions and instantly verify that further enhancements
don't violate
them. I updated and greatly extended the test suite for this patch release. The test suite is a common lisp program that performs 87 tests that
exercise,
to my knowledge, every feature of the -p switch. I chose lisp as the
language to
write the suite because of its flexibility. Where other languages are
rigid and
specific, lisp is fluid and generic. Other languages are really just
skins on top
of lisp with a lot of inconvenient syntax in the way. But I digress. :) If anyone thinks of any additional tests I'd be happy to add them. To
use the test
suite you must have a common lisp environment installed. I use cmucl.
You need the CL
"port" library. There are a few variables at the top of the lisp file
you need to
update and a line in nmap.cc you need to uncomment: // Uncomment the following line to use the common lisp port spec test
suite
//printf("port spec: (%d %d %d)\n", ports->tcp_count,
ports->udp_count, ports->prot_count); exit(0);
I will put the test suite into SVN soon (or maybe my blog). Here is its
output:
# lisp -quiet -load p-switch-tests.lisp OLD FORMAT SCANS ---------TCP scans---------- Test #1 'nmap -sS -p - targ' PASSED! Test #2 'nmap -sS -p 0- targ' PASSED! Test #3 'nmap -sS -p -20 targ' PASSED! Test #4 'nmap -sS -p 0-20 targ' PASSED! Test #5 'nmap -sS -p 823,39,19,99,100 targ' PASSED! Test #6 'nmap -sS -p -2,10-20,400,65534- targ' PASSED! Test #7 'nmap -sS -p -2,10-20,T:400,65534-,U:200 targ' PASSED! Test #8 'nmap -sS -p -2,10-20,U:400,65534- targ' PASSED! ---------UDP scans---------- Test #9 'nmap -sU -p 10-20 targ' PASSED! Test #10 'nmap -sU -p -20 targ' PASSED! Test #11 'nmap -sU -p 0-20 targ' PASSED! Test #12 'nmap -sU -p 0-20,T:100 targ' PASSED! Test #13 'nmap -sU -p 0-20,U:100 targ' PASSED! Test #14 'nmap -sU -p U:0-20,100 targ' PASSED! ---------Protocol scans---------- Test #15 'nmap -sO -p - targ' PASSED! Test #16 'nmap -sO -p 10-20 targ' PASSED! Test #17 'nmap -sO -p -20 targ' PASSED! Test #18 'nmap -sO -p 255- targ' PASSED! ---------TCP+UDP scans---------- Test #19 'nmap -sSU -p U:-2,10-20,400,65534- targ' PASSED! Test #20 'nmap -sSU -p T:-2,10-20,400,65534- targ' PASSED! Test #21 'nmap -sSU -p -2,10-20,400,65534- targ' PASSED! Test #22 'nmap -sSU -p -2,10-20,U:400,65534- targ' PASSED! Test #23 'nmap -sSU -p -2,10-20,T:400,65534- targ' PASSED! Test #24 'nmap -sSU -p 10-20,U:400-403,T:65534- targ' PASSED! ---------[] scans------------ Test #25 'nmap -sSU -p [-] targ' PASSED! Test #26 'nmap -sSU -p [-],1234 targ' PASSED! Test #27 'nmap -sSU -p [-],T:1234 targ' PASSED! Test #28 'nmap -sSU -p [T:-] targ' PASSED! Test #29 'nmap -sSU -p [T:-],1234 targ' PASSED! WARNING: Duplicate port number(s) specified. Are you alert enough to
be using Nmap? Have some coffee or Jolt(tm).
Test #30 'nmap -sSU -p T:1234,[-] targ' PASSED! Test #31 'nmap -sSU -p T:1234,[U:-] targ' PASSED! Test #32 'nmap -sSU -p [-1024] targ' PASSED! Test #33 'nmap -sSU -p [1024-] targ' PASSED! Test #34 'nmap -sSU -p [1024-,1234] targ' PASSED! Test #35 'nmap -sO -p [-] targ' PASSED! Test #36 'nmap -sO -p [-],252 targ' PASSED! WARNING: Duplicate port number(s) specified. Are you alert enough to
be using Nmap? Have some coffee or Jolt(tm).
Test #37 'nmap -sO -p [-],136 targ' PASSED! ---------wildcard scans------------ Test #38 'nmap -sSU -p rsync targ' PASSED! Test #39 'nmap -sS -p http targ' PASSED! Test #40 'nmap -sU -p http targ' PASSED! Test #41 'nmap -sSU -p http targ' PASSED! Test #42 'nmap -sSU -p T:http targ' PASSED! Test #43 'nmap -sSU -p http* targ' PASSED! Test #44 'nmap -sSU -p U:http*,T:http*,rsync targ' PASSED! Test #45 'nmap -sSU -p vnc-http* targ' PASSED! Test #46 'nmap -sSU -p vnc-http-? targ' PASSED! Test #47 'nmap -sSU -p *-mgmt targ' PASSED! Test #48 'nmap -sO -p ipv6* targ' PASSED! Test #49 'nmap -sSU -p [200-300],T:http,[900,U:1024-],T:f*l*ma?er targ'
PASSED!
---------BAD/STRANGE ARGS---------- WARNING: Duplicate port number(s) specified. Are you alert enough to
be using Nmap? Have some coffee or Jolt(tm).
Test #50 'nmap -sS -p 1,1 targ' PASSED! Test #51 'nmap -sU -p T:1-100 targ' PASSED! Test #52 'nmap -sT -p U:1-100 targ' PASSED! Test #53 'nmap -sO -p T:50,U:100 targ' PASSED! NEW FORMAT SCANS TestNew #54 'nmap --port-ratio 0.01 targ' PASSED! TestNew #55 'nmap --top-ports 100 targ' PASSED! TestNew #56 'nmap --port-ratio 0.01 -sU targ' PASSED! TestNew #57 'nmap --top-ports 100 -sU targ' PASSED! TestNew #58 'nmap --port-ratio 0.01 -sSU targ' PASSED! TestNew #59 'nmap --top-ports 100 -sSU targ' PASSED! TestNew #60 'nmap --top-ports 1 -sSU targ' PASSED! TestNew #61 'nmap --top-ports 1 -sS targ' PASSED! TestNew #62 'nmap --top-ports 1 -sU targ' PASSED! TestNew #63 'nmap --top-ports 20 -sS targ' PASSED! TestNew #64 'nmap --top-ports 65536 -sS targ' PASSED! TestNew #65 'nmap --top-ports 8 -sU targ' PASSED! TestNew #66 'nmap --top-ports 9 -sU targ' PASSED! TestNew #67 'nmap --port-ratio 0.25 -sSU targ' PASSED! TestNew #68 'nmap --port-ratio 0.25 -sS targ' PASSED! TestNew #69 'nmap --port-ratio 0.25 -sU targ' PASSED! TestNew #70 'nmap --port-ratio 0.8 -sSU targ' PASSED! TestNew #71 'nmap --port-ratio 0.1 -sU targ' PASSED! TestNew #72 'nmap --port-ratio 0.01 -sU targ' PASSED! TestNew #73 'nmap --top-ports 2 -p 80 targ' PASSED! TestNew #74 'nmap --top-ports 20 -p * targ' PASSED! TestNew #75 'nmap --top-ports 21 -p * targ' PASSED! TestNew #76 'nmap --top-ports 16 -p *http* targ' PASSED! TestNew #77 'nmap --top-ports 100 -p filemaker targ' PASSED! TestNew #78 'nmap --top-ports 100 -p filemaker -sU targ' PASSED! TestNew #79 'nmap --top-ports 100 -p filemaker -sSU targ' PASSED! TestNew #80 'nmap --top-ports 7 -p *http* -sU targ' PASSED! TestNew #81 'nmap --top-ports 7 -p *http* -sSU targ' PASSED! TestNew #82 'nmap --top-ports 100 -p *http* -sSU targ' PASSED! TestNew #83 'nmap --top-ports 2 -p -100 -sS targ' PASSED! TestNew #84 'nmap --top-ports 2 -p -100 -sU targ' PASSED! TestNew #85 'nmap --top-ports 2 -p -100 -sSU targ' PASSED! TestNew #86 'nmap --top-ports 21 -p [-] -sSU targ' PASSED! TestNew #87 'nmap --port-ratio 0.1 -sSU -p *http* targ' PASSED! 87/87 tests PASSED. CONCERNS ~~~~~~~~ o This patch, when combined with a new format nmap-services file, will
change the
ports that Nmap scans with a default scan and with a fast scan.
Currently deployed
scripts that rely on certain ports being scanned might have to be
changed.
o Collecting meaningful data for this sort of DB (nmap-services) is
hard! I am not
convinced there is such a thing as a "typical port distribution" for
hosts on today's
internet. And what's more, with the pace of technological development
as it is, I'm not
sure that any "typical distribution" will stay "typical" for long. o Perhaps the biggest issue I see with the new default/fast scan
behaviour is that
it becomes difficult to know if a given port was scanned for when
looking at a scan's
results. Was protocol XYZ included in a fast scan with the services
file shipped on
some given date? It's not easy to know, especially if we plan on
doing semi-frequent
updates to this DB. Summary ~~~~~~~ In summary I think this collection of patches is ready for inclusion
into Nmap. The
-p switch patches are long overdue and are extremely useful on the
command line
(quick, what port is NNTP?). The top ports patch is also ready and I
don't think it
will interfere with current Nmap installs at all. If we do eventually
create a decent
collection of data, then this patch and the port list primitives it
offers, are powerful,
generic interfaces to Nmap for using this data. ---------------------------------------------------------------------- _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGSujkMi/lqD03xGARApgcAKCnULoY/m6CQyUUyNUnjN25qNO0OgCgmhIi m12hOrDe17rW+AFWfA2eOZQ= =C7/E -----END PGP SIGNATURE----- _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- Top ports and -p switch extension patchset doug (May 16)
- Re: Top ports and -p switch extension patchset Luis Martin Garcia (May 16)
- Re: Top ports and -p switch extension patchset Eddie Bell (May 16)
- Re: Top ports and -p switch extension patchset DePriest, Jason R. (May 16)
- Re: Top ports and -p switch extension patchset (fix for list and ping scans) David Fifield (May 22)
- Re: Top ports and -p switch extension patchset Kris Katterjohn (May 23)