Re: Feature Request: --top-ports option for -PS when performing host discovery

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Comments inline:

On Tue, 19 Feb 2008 17:44:51 -0800 or thereabouts doug () hcsw org wrote:

> Hi Nelson,
>
> > Nelson <komseh () gmail com> wrote:
> > > I would like to be able to tell nmap to do host discovery with -PS
> > > and send the TCP Syn Pings to the top X most common ports.
>
> Interesting, I hadn't thought about using top ports data to choose
> the ports used with host discovery. Like Brandon says, the most
> straightforward way would be just to use a regular SYN scan and -P0,
> supplying your top ports data with the -p switch (host discovery
> and -sS even use the same engine in recent Nmap: ultra_scan).

People often forget that as long as a port isn't filtered, it doesn't
matter if it is closed if -PA is used.  My guess is that the most
common ports probably correspond to the most common firewall exceptions
but I don't have any supporting data to that end.

> Of course, this assumes that you *have* top ports data that you trust.
> The backend code to implement such top ports functionality actually
> already exists in Nmap. The only thing holding back the feature is a
> good data-set.

A quality, unbiased data-set is damn hard to make.

> If the Nmap community agrees on a data-set, I would be happy to make
> the top ports feature work. I ran some -iR -p1-65535 scans a year or
> two ago but was never confident enough recommend that the resultsbe
> the official Nmap data.

I did this too.  Scanned at about 80% of my outbound bandwidth (512
kbps at the time) for a month in preparation to provide good data.  I
too found the results to be crap and so didn't report them.

> Unlike my crappy results, Brandon's data looks very promising! Was
> this data collected from full port scans of a large heterogenous
> network without significant firewall interference? If so, and of
> course if Brandon agrees, I could massage the data into a format that
> would allow you to run the following scan:

There was firewall interference bust the vast majority of the firewalls
are "all or nothing".  That is, we either are able to scan all the
ports or none of the ports.  Of course

Here are a few reasons why I think our data shouldn't be the _only_
source of top ports:

* We are an academic network which means we have every version of every
ancient program ever made.  In large quantities too.

* Certain software (Sophos AV for example) is very common here because
of license agreements but is rare "in the wild".  This was the ports
8192-8194.

* The scan was only a single sweep of the campus.  Approximately 1/3 of
our machines are available for any given scan so the data is a small
sample.  The sample is biased because machines running IMAP/HTTP/SSH
are more likely to be on at all times than Windows boxes that come and
go.  This means that server ports have an unfair advantage over Windows
ports.

* Approximately 50% of our Windows boxes are running the Windows
firewall and so are dramatically under-counted.

* It seems like every small group feels a need to buy their own network
printer which makes telnet more common than VNC.

* Common ports and services like 7, 9, 13, 17, 19 are banned and we
block you if you are running them.  This biases our data against
certain services.

* 10k of the scanned computers are ResNet student owned machines and
are _heavily_ biased towards Windows and Mac.  We have more iPhones on
ResNet than Linux boxes (I cry myself to sleep when I think about
that). Here is the breakdown by OS on ResNet:

5427 PC running Windows XP Home
3562 PC running Windows Vista Home Premium
2567 Macintosh running Mac OS X
 465 PC running Windows XP Pro
 297 iPhone/iPod
 290 PC running Windows Vista Home Basic
 212 Access Point
 150 Xbox360
 115 PC running Windows Vista Business
  95 PC running Windows Vista Ultimate
  44 PlayStation3
  44 PC running Linux (Ubuntu)
  41 Nintendo Wii
  32 SmartPhone/Palm/PocketPC
  31 PC running Windows 2000
  23 PSP
  22 Nintendo DS
  18 PC running Linux (other)
  13 Macintosh running something else
  12 Router
  10 PC running Linux
  10 Other
   8 TiVo
   8 PC running Linux (Gentoo)
   5 Other machine running Unix
   4 Xbox
   4 VoIP Phone
   4 PC running Windows Server 2003
   4 PC running Windows Me
   3 PC running Linux (Debian based)
   2 Ruckus Client Simulator
   2 PlayStation2
   2 PC running something else
   2 PC running Windows Vista Enterprise
   2 PC running Linux (Red Hat based)
   1 VideoPhone
   1 Slingbox
   1 PDA Phone
   1 PC running some other Unix
   1 PC running Windows 98
   1 PC running Solaris
   1 Macintosh running Mac OS 9
   1 Macintosh running Mac OS 7


> nmap -P0 -sS --top-ports 50 target
>
> Another option is to create a community submission interface for such
> data similar to how we currently collect OS/service fingerprints. This
> was suggested recently by Lionel Cons:
>
> http://seclists.org/nmap-dev/2008/q1/0162.html

I'd say our data is a decent starting point but that a big
non-academic organization's data is needed to help balance ours out.
Random scans of the Internet are really not the way to go.

> I actually have some perl code that will take Nmap -oG scans and
> strip all information except for port lists, as well as a C program
> that will compress lists of port numbers into a compact ASCII encoding
> suitable for pasting into web text fields:
>
> http://hcsw.org/downloads/portcompress.c


> A potential problem with any top ports data, especially considering
> the rapid, unpredictable pace of internet development, is that data
> might become inaccurate over time. This could happen when a new
> service becomes popular or when an older one becomes obsolete (think
> telnet -- though Brandon's data suggests telnet is still more common
> than, say, VNC.)

Agreed.  We have problems keeping up with current bogon space though and
(thanks to Kris) are still able to manage.  If we really do get large
data-sets provided, we can take weighted averages where as data ages,
it's weight is reduced.

> Anyways, I'm very open to ideas on how tackle this important problem.

How about:

"Dear Nmap-Hackers list.  We are looking for a very large data-set of
popular ports.  If you have a very large network which you think
decently represents other similar organizations please run this Nmap
command....."

I'm sure there are people at a dozen different big companies, schools,
and ISPs that have data or can get data.
> Best,
>
> Doug

Lets make the top ports happen.

Brandon


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHu5CbqaGPzAsl94IRAlNIAJ4rv+KzNxgv2FLouA8YTQYvi94gjwCeJBXM
TAp4vNFj+iiALuBvP7qLbU8=
=iJbi
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org