Service fingerprint integration highlights

[Daniel Miller <bonsaiviking () gmail com>]

> From the 6.47 release, we have added 1098 new service match lines, bringing
us over the 10000 mark! That's a 12% increase, which is huge.

Of those, softmatches increased from 51 to 95, an 86% increase. Soft match
lines help speed up version detection by limiting what probes are sent
after Nmap is sure that a service is using a particular protocol.
Sometimes, though, we use them to match a service that we want more or
better quality submissions for in the future. We now have new or expanded
softmatches for:

adobe-crossdomain
afp
bgp
caldav
domain
epp
gopher
http-proxy
icap
imap
kerberos-sec
ldap
microsoft-ds
netbios-ns
openwebnet
pop3
postgresql
ppp
rtsp
sieve
slp-servreg
smpp
smtp
svrloc
telnet
tftp
upnp
webdav
ws-discovery

A few open-source software projects have been fingerprinted based on source
code analysis. A comment in the nmap-service-probes file for postgresql
matches describes the technique of checking out the source at every git tag
(version) and grepping for a particular output string, noting which
versions change. Using this technique, the Postgres fingerprints were
extended, and new fingerprints for Varnish Cache CLI and amavisd-new were
created without receiving any submissions for those versions! There are
probably other services that would benefit from this sort of analysis. For
example, "openssl s_server -WWW" gives a nice error message including line
numbers:

# TODO: hunt down line number/version number correlations
match http m|^HTTP/1\.0 200 ok\r\nContent-type: text/plain\r\n\r\nError
opening ''\r\n\d+:error:[A-F\d]+:system library:fopen:No such file or
directory:bss_file\.c:169:fopen\('','r'\)\n\d+:error:[A-F\d]+:BIO
routines:BIO_new_file:no such file:bss_file\.c:172:\n| p/OpenSSL s_server
-WWW httpd/

Several fingerprints are for various USB-over-network protocols:
# Need to figure out what this is and how to structure the match
match ipmi-usb m|^IUSB
\0\0\0\x007\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xf1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0$|
p/IPMI USB redirection/ d/remote management/
match ipmi-usb m|^IUSB
\0\0\0\x007\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xf1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x0210\.6\.11\.29\0\0\0\0\0\0\0\0\0\0\0\0\0\0$|
p/IPMI USB CD redirection/ d/remote management/
# TRENDnet NetUSB - 4-byte-length-prefixed null-terminated strings
# USB-over-network:
https://www.trendnet.com/kb/kbp_viewquestion.asp?ToDo=view&questId=1350&catId=516
match netusb m|^\0\0\0. connect success c\d+ \n\0\0\0\0. NetUSB ([\w._-]+),
2\d\d\d, \d+ \n\0\0\0\0\x0c AUTH ISOC\n\0\0\0\0\x18 MOVE_HEAD
filterAudio\n\0\0\0\0\0| p/TRENDnet NetUSB/ v/$1/
# virtualhere 2.2.5, port 7575
match virtualhere
m|^\0\0\0\0%\0\0\0\x0c\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x04\0\0\0.\xca\xc0T|
p/VirtualHere USB Server/

Some things are truly random and hard to match. This one has had a couple
false-positives in the submissions we've received after it was put in, but
it seems worthwhile to keep:

# Port 9535: http://community.landesk.com/support/docs/DOC-1591
# This is 264 random bytes, probably some sort of shared-key encryption
match landesk-rc m|^.{264}$|s p/LANDesk remote management/

Similarly, the SSH server on Fortinet devices uses a randomized banner.
Instead of adding these every time, we made this match line but commented
it out due to false positives:

# FortiSSH uses random server name
#match ssh m|^SSH-([\d.]+)-[\w._-]{5,14}\n| p/FortiSSH/ i/protocol $1/

Finally got to the bottom of this odd one that was frequently submitted.
It's some demo software that Netgear puts into their OpenWrt-based firmware
for home routers. I think that the MAC address may be filled out in some
cases, but I didn't have any examples in this round of submissions. Also,
I'm not sure whether it gets filled out with the requester's MAC or the
router's MAC.

# /usr/sbin/potval
# https://github.com/elvanderb/TCP-32764/issues/98
match pot m|^0NTP00-00-00MAC00-00-00-00-00-00| p|Netgear POT-(Get/Set)
Demo| d/broadband router/

Another enigma partly solved. The public key always starts with this
sequence, but as the comment states, we don't know what the name portion
refers to:

# Not sure what this name is. Have seen XenVMMXenVMM, @\x03, and NOTFOUND
match r1soft-cdp m|^\0\0\x01.R.\x02\n.\x08\xa3\x80\x04\x10.\x18\0
[\0\x01]\*.(.*?)\x10\0\x1a\x90\x02-----BEGIN PUBLIC
KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQ|s p/R1Soft Continuous Data
Protection Agent/ i/name: $P(1)/

This fun service is used to remote-control a Roku set-top box. On another
note, should we continue to call these set-top boxes if the "TV set" is
more likely to be hanging on the wall than having something set on top of
it?

# port 8080, accepts commands like "press up" "press mute"
match roku-remote m|^([0-9A-Z]{5}[A-Z]\d{6})\r\n>| p/Roku remote API/ i/SN
$1/ d/media device/

Added a bunch of fingerprints to match unauthenticated shells listening
directly on TCP ports:

match shell m|^(?:ba)?sh-\d\.\d+\w?# $| p/ROOT SHELL/ i/**BACKDOOR**/
o/Unix/
match shell m|^(?:ba)?sh-\d\.\d+\w?\$ $| p/bind shell/ i/**BACKDOOR**/
o/Unix/
match shell m|^(?:ba)?sh: no job control in this
shell\n(?:ba)?sh-\d\.\d+\w?\$ $| p/bind shell/ i/**BACKDOOR**/ o/Unix/
match winshell m=^Microsoft Windows (2000|XP|NT 4\.0) \[Version
([\d.]+)\]\r\n\(C\) Copyright 1985-20\d\d Microsoft Corp\.\r\n\r\n=
p/Microsoft Windows cmd.exe/ v/$2/ i/**BACKDOOR**/ o/Windows $1/
match winshell m|^Microsoft Windows \[Version ([\d.]+)\]\r\n\(C\) Copyright
1985-20\d\d Microsoft Corp\.\r\n\r\n| p/Microsoft Windows cmd.exe/ v/$1/
i/**BACKDOOR**/ o/Windows/
match shell m|^bash: line 1: \$'\\r': command not found\nbash: line 2:
\$'\\r': command not found\n| p/Bash shell/ i/**BACKDOOR**/
match shell m|\r: bad character in file name: '/bin/\r'\n$| p/Plan 9 rc
shell/ i/**BACKDOOR**/ o/Plan 9/

What was that? Plan 9? Yes, lots of Plan 9 services were added, thanks to a
valuable contributor:

match imap m|^\* BYE imap4 connections must use ssl\n$| p/Plan 9 imapd/
i/must use ssl/ o/Plan 9/
match rexec m|^/bin/ip/rexexec: auth_proxy: auth_proxy rpc: negotiation
failed, no common protocols or keys\n| p/Plan 9 rexexec/ o/Plan 9/
match ssh m|^SSH-([\d.]+)-Plan9\r?\n| p/Plan 9 sshd/ i/protocol $1/ o/Plan
9/
match venti m|^venti-02-libventi\n| p/Plan 9 venti storage system/ o/Plan 9/
match ident m|^rc \(tcp113\): null list in concatenation\n| p/Plan 9
identd/ o/Plan 9/
match shell m|\r: bad character in file name: '/bin/\r'\n$| p/Plan 9 rc
shell/ i/**BACKDOOR**/ o/Plan 9/
match http m|^HTTP/1\.1 \d\d\d .*\r\nServer: Pegasus/Plan9\r\n|s p/Pegasus
httpd/ o/Plan 9/
match exportfs m|^(?:p9sk1@[\w._-]+ )*p9sk1@([\w._-]+)\0/bin/exportfs:
auth_proxy: auth_proxy rpc write: : invalid argument\n| p/Plan 9 exportfs/
o/Plan 9/ h/$1/
match smtp m|^220 ([\w._-]+) ESMTP\r\n250 2\.0\.0 See
http://www\.ietf\.org/rfc/rfc2821\r\n|
p|Plan 9 upas/smtpd| o/Plan 9/ h/$1/
match cpu m|^unsupported auth method\0| p/Plan 9 cpu/ o/Plan 9/

More on the subject of backdoors, we have a couple new fingerprints for
malware:

match darkcomet m|^[0-9A-F]{12}$| p/DarkComet RAT/ i/**BACKDOOR**/
match ftp m|^220 220 RMNetwork FTP\r\n$| p/Ramnit worm ftpd/ i/malware/
match rmmd m|^100 Rmmd version ([\w._ -]+?)\. *\r\n101 [\da-f]{32}\r\n|
p/Rmmd trojan/ v/$1/

Plus, the infamous TCP-32764 backdoor:

# https://github.com/elvanderb/TCP-32764
match scmm m|^MMcS\xff\xff\xff\xff\0\0\0\0| p/SerComm manufacturer
backdoor/ d/broadband router/

Replaced a bunch of haphazard Apache match lines with some
intentionally-ordered matches to cover possible future edge-cases and grab
as much correct info as possible:

match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache[/
](\d[-.\w]+)\r.*\nX-Powered-By: PHP/([\w._-]+)\r\n|s p/Apache httpd/ v/$1/
i/PHP $2/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache\r.*\nX-Powered-By:
PHP/([\w._-]+)\r\n|s p/Apache httpd/ i/PHP $1/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache[/
](\d[-.\w]+)\r.*\nX-Powered-By: ([^\r\n]+)\r\n|s p/Apache httpd/ v/$1/ i/$2/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache\r.*\nX-Powered-By:
([^\r\n]+)\r\n|s p/Apache httpd/ i/$1/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache[/ ](\d[-.\w]+)
([^\r\n]+)|s p/Apache httpd/ v/$1/ i/$2/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache[/
](\d[.\w-]+)\s*\r?\n|s p/Apache httpd/ v/$1/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache\r\n|s p/Apache httpd/
match http m|^HTTP/1\.[01] \d\d\d .*\r\nServer: Apache
+\(([^\r\n\)]+)\)\r\n|s p/Apache httpd/ i/$1/

These print servers had many different matches, but are all the same thing
so I consolidated them:

# Branded as Longshine, TRENDnet, TP-LINK, IOGear, Hawking
# Date is usually (always?) Mon, 24 Sep 2001 18:00:00 GMT
match http m|^HTTP/1\.0 \d\d\d .*\r\nDate: .* GMT\r\nMIME-version:
1\.0\r?\nServer: ZOT-PS-(\d+)/([\w._-]+)\r?\n| p/Zero One Technology $1
httpd/ v/$2/ d/print server/

Apple Filing Protocol (AFP) matches are tricky to determine what variance
is due to OS or application version, and what is due to configuration. I
was able to consolidate a lot of these matches by capturing the machine
type. Possible improvement would be to also capture the hardware version
(comma-separated digits after the hardware type which a user could use to
solidly pin down a range of OS X versions supported). Example:

match afp
m=^\x01\x03\0\0........\0\0\0\0........\x8f\xfb.([^\0\x01]+)[\0\x01].*?(iMac|Mac(?:mini|Pro|Book(?:Air|Pro)?))\d+,\d+\x04\x06AFP3\.3\x06AFP3\.2\x06AFP3\.1\x06AFPX03.\tDHCAST128.*[\x04\x05]([\w.-]+)\x01.afpserver=s
p/Apple AFP/ i/name: $1; protocol 3.3; Mac OS X 10.5 - 10.6; $2/ o/Mac OS
X/ h/$3/

PROTIP: If you're developing a honeypot, copy the response from a real
service, don't write your own:

match honeypot m|^HTTP/1\.0 200 OK\r\nAllow: OPTIONS, GET, HEAD,
POST\r\nContent-Length: 0\r\nConnection: close\r\n\r\n| p/Dionaea Honeypot
httpd/
match honeypot m|^SIP/2\.0 200 OK\r\nContent-Length: 0\r\nVia: SIP/2\.0/TCP
nm;branch=foo\r\nFrom: sip:nm@nm;tag=root\r\nAccept: application/sdp\r\nTo:
sip:nm2@nm2\r\nContact: sip:nm2@nm2\r\nCSeq: 42 OPTIONS\r\nAllow: REGISTER,
OPTIONS, INVITE, CANCEL, BYE, ACK\r\nCall-ID: 50000\r\nAccept-Language:
en\r\n\r\n| p/Dionaea Honeypot sipd/
match ms-sql-s
m|^\x04\x01\x00\x2b\x00\x00\x00\x00\x00\x00\x1a\x00\x06\x01\x00\x20\x00\x01\x02\x00\x21\x00\x01\x03\x00\x22\x00\x00\x04\x00\x22\x00\x01\xff\x08\x00\x02\x10\x00\x00\x02\x00\x00|
p/Dionaea honeypot MS-SQL server/

Used MS Windows version table lookups to generate detailed fingerprints for
ms-sql-s. These include service pack and major update info. Example:

match ms-sql-s
m|^\x04\x01\x00\x25\x00\x00\x01\x00\x00\x00\x15\x00\x06\x01\x00\x1b\x00\x01\x02\x00\x1c\x00\x01\x03\x00\x1d\x00\x00\xff\x0a\x32\x06\x51|
p/Microsoft SQL Server 2008 R2/ v/10.50.1617; RTM+ MS11-049/ o/Windows/

Additionally, we have vastly improved detection of SSL and SSL-wrapped
services by expanding the port lists for the SSL probes, marking more
standard ssl-wrapped ports as sslports for the associated services, and
matching more TLS versions and handshake alerts. We also have a new probe,
TLSSessionReq, which adds some TLS features, ciphersuites, and extensions
that could cause the older SSLSessionReq request to fail to get a response
from a real TLS service. Discussion: http://seclists.org/nmap-dev/2015/q2/17

And finally, David Fifield did some great work on detecting Tor, including
a new tor-versions probe: http://seclists.org/nmap-dev/2015/q1/39

Happy scanning!
Dan

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/