Nmap Development mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Feature: per-target port specification (with patch!)

From: Robin Wood <robin@digi.ninja>
Date: Mon, 8 Apr 2019 14:38:39 +0100
Hi
I couldn't comment on the patch but I'm trying to understand your use case
and don't quite get what you were explaining, any chance of a bit more
detail on it?

Robin

On Mon, 8 Apr 2019 at 14:15, Jan Gocník <gocnik () dcit cz> wrote:


Hey,

I worked on this some more.

Fixed all of the *memory leaks* that were my fault, and the crash that
was a result of a bug in mergeHostSpecificPorts. I took this opportunity to
rewrite the mergeHostSpecificPorts algorithm, so now *the results are
properly sorted*.
The disrepancy between "Scanning X [max N ports]" and "Completed Connect
Scan at X (0 ports max)" should be fixed as well, the totalprobes value is
now properly initialized.

About the features you mentioned:

The "Not shown: X ports" output for Normal output.

Well, from what I know, this doesn't always allow you to infer information
about all ports anyway. For example, if you get "Not shown: 3995 closed
ports, 514 filtered ports". Therefore, as long as the counts are calculated
correctly, I don't think it's necessary to output information about
additional scanned ports, as you can get these from other more verbose
outputs.


Properly formed XML output, with changes to the DTD and a

"xmloutputversion" number increase.
I simply output another scaninfo tag into each host tag. Updated the DTD
and xmloutputversion to 1.05.
For greppable, I output it in a format similiar to the ports listing.


Combination of this feature with existing --top-ports/port-ratio and -p

options
From my testing, it merges with them properly.


Combination of this feature with CIDR subnetting and IPv4 octet ranges

Should work just fine.


Use of this feature along with advanced features like -O --traceroute

and -sV
I tested it with these options and everything seemed good to me.

On the topic of *memory usage:*
I wrote it so that if you do regular scans, the additional memory usage
should be very small - only a scan_lists per target group, which is <64
bytes, and a few pointers here and there. I can't get reproducible heap
reports form valgrind unfortunately, but in one case the new version even
allocated less memory than latest SVN trunk for "nmap scanme.nmap.org".
The biggest item of all is port_map_rev, which is 65536*sizeof(u16) = 128
kB. This is allocated for each target that has specific ports (but only for
those). Only way I can think of making this better is sharing those for the
whole target group, but I dunno if it's worth the effort.

Now, on the *rationale.*
A very specific (real) example when this feature is useful: We are
scanning a network, which has some kind of network appliance that replies
to SYNs on port 80 on every unleased IP, but doesn't reply to most other
ports at all (not even with a RST). That means that nmap considers every
single IP live,
but then waits for a long time when attempting to scan all the other
ports. I was scanning 521 IPs, of which 11 were really occupied and active
and the 511 others were the "fakes".
I was testing three options: Scan all hosts for all ports that appeared at
least once (marked all_ports), write a simple script that calls nmap for
each target (script), and then used the per-target ports (per_target).

With ping scan, the results were the following:
* all_ports: 102 seconds
* per_target: 80 seconds
* script: 60 seconds
Without ping scan:
* script: 45 seconds
* all_ports: 38 seconds
* per_target: 4 seconds

My conclusion is that this feature is useful when you get filtered ports,
as these take a lot of time. Basically, if you only have open and closed
ports, scanning some additional ones is fast and the "union all ports"
strategy is alright. But once you have filtered ports, the waiting quickly
gets bad.
The script could be upgraded to be parallel, but then you're reinventing
nmap's parallel engine.

Now I know the "proper" solution on this network would be to use a brain
and try to discern real and fake machines, but this already gets much
faster results without a need for thinking, which is always nice.
I understand your concern about losing "new" services, but I see the main
usability of this in manual scans, for example running additional nse
scripts against your last scan results etc. When you are doing automatic
scans (for exmple as a network admin scanning network for new devices each
week), you probably don't need to care about speed anyway...

Updated patch (to svn revision 37611):



Looking forward to further comments and hopefully we can make this work :)

Jan



From:        Jan Gocník/Dcit
To:        "Daniel Miller" <bonsaiviking () gmail com>
Cc:        "Nmap-dev" <dev () nmap org>
Date:        02.04.2019 23:53
Subject:        Re: Feature: per-target port specification (with patch!)
------------------------------


Hey Dan,

thanks for the reply! It's a shame that I didn't find the GitHub issue you
link to before implementing this, as it does raise a lot of valid concerns.

First, let me say that the company I work for wants this feature, so even
if it doesn't end up in upstream, I will try to keep it at least as a fork
- as I'll have to maintain it internally anyway, I wanted to share with the
community, in case others have a need for it as well.

I will go through all the things you mentioned (most of the compatibility
with other options should be taken care of, but memory leaks are a
problem), fix up the code, look at maybe getting the memory footprint
lower, and try to come up with some stronger numbers and rationale.

Jan





From:        "Daniel Miller" <bonsaiviking () gmail com>
To:        "Jan Gocník" <gocnik () dcit cz>
Cc:        "Nmap-dev" <dev () nmap org>
Date:        02.04.2019 21:21
Subject:        Re: Feature: per-target port specification (with patch!)
Sent by:        "dev" <dev-bounces () nmap org>
------------------------------



Some initial notes from building and testing this:

./nmap *scanme.nmap.org* <http://scanme.nmap.org/>^22-80 -d
Starting Nmap 7.70SVN ( *https://nmap.org* <https://nmap.org/> ) at
2019-04-02 18:37 UTC
PORTS: Using top 1000 ports found open (TCP:1000, UDP:0, SCTP:0)
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 18:37
Scanning *scanme.nmap.org* <http://scanme.nmap.org/> (45.33.32.156) [max
2 ports]
Completed Ping Scan at 18:37, 0.09s elapsed (1 total hosts)
Overall sending rates: 24.44 packets / s.
mass_rdns: Using DNS server X.X.X.X
Initiating Parallel DNS resolution of 1 host. at 18:37
mass_rdns: 0.12s 0/1 [#: 3, OK: 0, NX: 0, DR: 0, SF: 0, TR: 1]
Completed Parallel DNS resolution of 1 host. at 18:37, 0.06s elapsed
DNS resolution of 1 IPs took 0.15s. Mode: Async [#: 3, OK: 1, NX: 0, DR:
0, SF: 0, TR: 1, CN: 0]
Initiating Connect Scan at 18:37
Scanning *scanme.nmap.org* <http://scanme.nmap.org/> (45.33.32.156) [max
1059 ports]
Discovered open port 80/tcp
Discovered open port 22/tcp
Discovered open port 9929/tcp
Discovered open port 31337/tcp
nmap: portlist.cc:688: void PortList::mapPort(u16*, u8*) const: Assertion
`mapped_portno < port_list_count[mapped_protocol]' failed.
Aborted (core dumped)


Valgrind identified some memory leaks. These were the ones that are
definitely from this patch: Portlist::setIdStr(), idstr;
PortList::mergeHostSpecificPorts(), new_port_map and new_port_map_rev;

There was also a discrepancy between the "Scanning X [max N ports]" and
"Completed Connect Scan at X (0 ports max)", which valgrind says is due to
an uninitialized value, probably GroupScanStats::totalprobes.

Of course, as you noted before, the results are not sorted by port number,
which we would have to do to ensure predictable output that can be compared
with previous scans.

IPv6 addresses, CIDR, and IPv4 octet ranges seem to work fine with this.

With just my one test scan of localhost and *scanme.nmap.org*
<http://scanme.nmap.org/> with one unique port each and one port in
common, the patched code allocated an additional 383KB of heap memory. I do
not know how this would scale up or down, and I haven't compared it with a
scan that should behave the same in both cases (for instance, "nmap
*192.168.1.0/24* <http://192.168.1.0/24>".

My primary concern remains that repeated use of this feature will result
in missing new services and dropping existing ones if they are missed in
just one scan. This is more about use case than about the code, though, so
I will defer to users on that.

Dan

On Tue, Apr 2, 2019 at 1:30 PM Daniel Miller <*bonsaiviking () gmail com*
<bonsaiviking () gmail com>> wrote:
Jan,

Thanks for this contribution. We've had many requests for this type of
feature in the past, but have elected not to include it for a variety of
reasons. There is an open discussion on our issue tracker that lays out
some of the challenges in correctly implementing such a feature:
*http://issues.nmap.org/1217* <http://issues.nmap.org/1217>

It looks like your patch has tried to handle some of these situations, for
example the "Ports scanned" output for Grepable output (and maybe XML, but
it didn't look complete at first glance). If we are to do an actual code
review and include this new feature, we would have to look for a complete
solution that can handle the following situations:

* The "Not shown: X ports" output for Normal output.
* Properly formed XML output, with changes to the DTD and a
"xmloutputversion" number increase.
* Combination of this feature with existing --top-ports/port-ratio and -p
options
* Combination of this feature with CIDR subnetting and IPv4 octet ranges
* Use of this feature along with advanced features like -O --traceroute
and -sV

Have you done any measurement of scans before and after adding this
feature to determine the actual impact on scan times and bandwidth? Do you
have a bandwidth target for your scans that Nmap is exceeding right now,
and by how much? What does a typical nmap command line look like, and what
performance options have you already tried?

I look forward to hearing more about this from you and our other devs and
users.

Dan

On Tue, Apr 2, 2019 at 8:07 AM Jan Gocník <*gocnik () dcit cz*
<gocnik () dcit cz>> wrote:
Hey,

I would like to propose a feature enabling specifying ports for each
target separately.

Rationale:
It often happens that we already have an nmap scan of 200 machines, and we
want to do a service scan on those same machines. Usually that forces us to
scan the whole network for all the ports that appeared at least once.
That is a big waste of time and bandwidth. What we want to have is
essentially a rescan-like feature, that would rescan just ports that were
found to be open before.

User experience:
Everywhere where you could specify a target (-iL file, command line) you
can supply a "target^ports". It works with all the nmap magic ranges, so
"192.168.1.1-255^22-60" works. The common ports (supplied with -p) are
scanned on all targets.

Implementation details:
I tried to keep it so that if you don't use any "^" in the targets, the
code path should remain largely the same, so there should be no
regressions. However, I had to do some tuning in functions that expected
they can just get the number of probes by multiplying common ports by
targets.
There's a small issue, in that the results of the scan are not sorted
properly, as the target-specific ports get scanned last.

Usage example:
===paste start===
$ nmap -v -Pn -n -p22 "165.227.141.119^80,443" "40.113.73.59^8080"
Starting Nmap 7.70SVN ( *https://nmap.org* <https://nmap.org/> ) at
2019-04-01 19:46 CEST
Initiating SYN Stealth Scan at 19:46
Scanning 2 hosts [max 3 ports/host]
Discovered open port 22/tcp
Discovered open port 80/tcp
Discovered open port 443/tcp
Discovered open port 22/tcp
Completed SYN Stealth Scan at 19:46, 1.45s elapsed (1626388576 total ports
max)
Nmap scan report for 165.227.141.119
Host is up (0.0090s latency).

PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

Nmap scan report for 40.113.73.59
Host is up (0.038s latency).

PORT     STATE    SERVICE
22/tcp   open     ssh
8080/tcp filtered http-proxy

Read data files from: /home/gocnik/nmap
Nmap done: 2 IP addresses (2 hosts up) scanned in 1.52 seconds
           Raw packets sent: 6 (264B) | Rcvd: 4 (176B)
===paste end===

If done the usual way:
$ nmap -v -Pn -n -p22,80,443,8080 165.227.141.119 40.113.73.59
[...]
Raw packets sent: 10 (440B) | Rcvd: 6 (260B)


The patch is against svn trunk at this moment (revision 37608).



Looking forward to all comments!
JaGoTu

P.S.: Sorry if you recieve this e-mail twice, but the previous one
apparently got caught in a moderation queue or something, as it doesn't
show on *seclists.org* <http://seclists.org/>
_______________________________________________
Sent through the dev mailing list
*https://nmap.org/mailman/listinfo/dev*
<https://nmap.org/mailman/listinfo/dev>
Archived at *http://seclists.org/nmap-dev/*
<http://seclists.org/nmap-dev/>
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/


_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/
Previous By Date Next
Previous By Thread Next

Current thread: