Wierd issue: nmap/nping misses (some) icmp replies

[John Robson via dev <dev () nmap org>]

Hi all,

I'm running into a weird issue that I can't quite get my head around.
As part of monitoring devices we use nmap to issue icmp echo requests
to a list of target devices, and obviously the lack of a reply is
worthy of generating an event/alert/alarm.

I am currently seeing sporadic events from a variety of target
devices, which clear on the next polling cycle (i.e. we miss one
response but get the following one sixty seconds later).  Each cycle
is a new nmap process with a list of IP addresses, and the "missed"
responses are just one or two hosts in that list.
Note that this is only affecting one monitoring environment of many,
so my initial suspicion was that it was a network issue, hence the
comparison between the displayed data and a packet capture.

To try and get a simpler test case I used nping to repeatedly poll one
device, and that gave some odd results which sort of correlate with
the above behaviour:
 - Occasionally nping will simply stop seeing responses after some
number (as low as 3) of successful requests (i.e. we just don't get
any RCVD packets printed, and packet loss is 100% after the first
few).
 - More often it will start reporting RCVD packets *far* too late, up
to minutes after the response has been seen in tcpdump, eventually
declaring that it "missed" the last several responses when it finishes
a set of requests.
 - Seems to be "less reliable" with extended delay times (10s is
usually enough to cause the issue).

It's possible that this is an entirely spurious correlation (unless
there are plenty of shared libraries), but it's harder to make a
simple, reproducible, nmap command.

As an example:
(I've trimmed some of the repeated data to make this easier to read)

Running "nping 10.192.1.16 -delay 10s -c30" was fine for the first 25
requests, and then...

SENT (240.2806s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=25] IP [ttl=64]
RCVD (240.2821s) ICMP [10.192.1.16 > 10.206.16.30 Echo reply  seq=25]
IP [ttl=253]
SENT (250.2886s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=26] IP [ttl=64]
SENT (260.2992s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=27] IP [ttl=64]
RCVD (260.2997s) ICMP [10.192.1.16 > 10.206.16.30 Echo reply  seq=26]
IP [ttl=253]
SENT (270.3099s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=28] IP [ttl=64]
SENT (280.3207s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=29] IP [ttl=64]
RCVD (280.3211s) ICMP [10.192.1.16 > 10.206.16.30 Echo reply  seq=27]
IP [ttl=253]
SENT (290.3313s) ICMP [10.206.16.30 > 10.192.1.16 Echo request
seq=30] IP [ttl=64]

Note that the RCVD packets are apparently delayed by longer and longer intervals
Despite the packet trace showing ~1ms response times for all 30 requests.

It looks likely that the printouts of RCVD are in some way tied to the
delay, since they all appear at a 10s interval, just not necessarily
the correct interval.

But after the 30th packet (which has an immediate response in the
packet capture) the summary data shows:
Max rtt: 1.321ms | Min rtt: 0.343ms | Avg rtt: 1.089ms
Raw packets sent: 30 (840B) | Rcvd: 27 (756B) | Lost: 3 (10.00%)
Nping done: 1 IP address pinged in 290.36 seconds

So despite the delay in printing the RCVD packet nmap correctly
*measured* sequence 26 and 27, but then shut down before processing
28/29/30.


The collector system is running ubuntu 20.04, with nmap 7.80, and nping 0.7.80
I appreciate that this isn't quite the latest, but I haven't seen
anything obvious in the changelog.

Trying not to upgrade in place I decided to try a standalone docker
image (instrumentisto/nmap), because that's nice and easy to jump
between versions for testing...
Whilst I continue to get the same issue on the host or in our regular
containers I cannot reproduce it with the standalone docker image
using either version 7.80 or 7.94.

I'm now not sure what to look at next...


--
John Robson
Sr. Customer Support Engineer, Zenoss
jrobson () zenoss com
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at https://seclists.org/nmap-dev/