RE: Nmap Service Detection Proposal

["Jay Freeman \(saurik\)" <saurik () saurik com>]

As for the XML library, I would suggest using Xerces-C.  Xerces is Apache's
XML library (-C being the C version, they also have a Java version).  Unless
there is some hope that a large number of people out there already have
libxml, Xerces is a complete implementation of DOM and SAX and all sorts of
other XML goodies, plugs into an XSLT container well (Xalan, another Apache
library).  Code from a large number of projects has been contributed to
Xerces and Xalan, including Sun's and IBM's parsing libraries (much of the
original code base is actually from IBM's XML4C, I believe).

I do like the idea of using XML as the base file library, as it solves all
of the escaping issues (most, if not all, of the escaping is done by the XML
parser).

I'm going to sit down sometime tomorrow (assuming I have some time, I think
I do... have to work on a document with my partner, but that shouldn't take
_that_ long) think of different ways to handle the jumping issues (if we
think its HTTP, and fail, but now know it is some other protocol, but have
to start over again, we know what kind of connection to jump to), and ways
of using the ports for sorting help without being locked in by them at the
same time.

A few issues that need to be dealt with, however, are stuff like timeouts.
Some services are just slower than others.  I noticed this while building
nmap+V (which I can modify into using an XML configuration file, although it
still wouldn't be parallel, but it is a start).  I can't assume that within
4 seconds I am going to get a reply, an example (I think, was a while ago)
was sending a HELP command to a mail server to get more accurate/further
version information seems to take some daemons a while (and also returns a
lot of data, which also has to be dealt with).  This is why I up the data
wait to 3 seconds (w 3000000) while waiting for a qmail reply from HELP.

Even if you were to try to give the daemon as much time as possible and go
on to other things, the data length issue is still a problem.  In the case
of FTP I use a 4096 byte receive buffer (< 4096) to get data because
sometimes I couldn't get the version string until long after the initial
connection as people were putting long connection banners on their FTP
servers.  At the same time, ports like chargen (or just really long web
pages) need to be dealt with.  A regular expression alone isn't rich enough
to describe how much data is needed before the particular item can match.

Sincerely,
Jay Freeman (saurik)
saurik () saurik com

-----Original Message-----
From: Fyodor [mailto:fyodor () insecure org]
Sent: Sunday, August 27, 2000 3:29 AM
To: nmap-dev () insecure org
Subject: Nmap Service Detection Proposal


Hi guys,

Here is a proposed grammar for Nmap service detection.  The main idea is
that the nmap-service-magic file contains a list of "probes".  Each probe
contains the following information:

1) A list of common ports for the services detected by the probe (for
   implementation optimization only -- an open port will first be tested
   with probes that list that port number)).

2) A string to be sent to the port right after connection establishment
   (if TCP).  The string can include escaped binary chars.

3) A list of (case insensitive) regular expressions to match against the
   response and the protocol name the regexp relates to.  The regexps. can
   contain escaped binary chars as well.

What I like about this solution is that it leaves the probe definitions
very simple and easy to read, modify, maintain, add to, etc.  I also
believe that it can be implemented efficiently.

The question is whether it is powerful enough.  I suspect that the vast
majority of protocols could be detected via a sufficiently clever probe
string and regex match.  Can anyone think of any protocols that could not
be detected by method but could with a more powerful (think
"C") syntax?  And RPC services don't count because we can get them with
the existing RPC grinder.

Lets look at a few example of services that might be challenging:

Netbios-ssn // NetBIOS Session Service
netbios-ns  // NetBIOS Name Service
x11
lpd
pcanywhere
orasrv      // Oracle

Could anyone he knows one of these protocols well send to the list an
example of a probe that could be sent & regexp that would recognize the
response as belonging to the given service?  Keep in mind that inducing
errors may be the easiest detection method.  Of course, composing a binary
"wire" example and testing against the given service makes the best demo.
Or if you have an English suggestion (eg a certain X11 or Netbios-ns call
that should always elicit a distinct respnose), chime in!

Or if you want to suggest another protocol that could present a serious
challenge, let us know.  Maybe someone can think of a way to ID it.  Its
important that we identify any fatal problems before we spend a lot of
time implenting this and creating service-specific "probes".

As far as the actual nmap-service-magic file grammer, we could either do
something custom or use XML.  Here is a custom example:

# The catch-all HTTP probe (which leads to distinctive error msgs from
# many services
Probe TCP=21,22,23,25,80,110,118,1080,8080 SEND="GET / HTTP/1.0\r\n\r\n"
ftp  220.*ftp
pop3 +OK
ssh  ssh-
smtp smtp
nntp posting ok
http http/1

# Probe X11.  I made up the hext values, presumably they would
# correspond to xome sort of X request
PROBE TCP=6000-6010 SEND="\x32\x28\x14\x29\x71\xB4"
x11  \x31\x72\x98


A possible XML alternative would be:

<SERVICEPROBE TCP="21,22,23,25,80,110,1080,8080" SEND="GET
/HTTP/1.0\r\n\r\n">
<SERVICE NAME="ftp" REGEX="220.*ftp" />
<SERVICE NAME="pop3" REGEX="+OK" />
...
</SERVICEPROBE>
<SERVICEPROBE TCP="6000-6010" SEND="\x32\x28\x14\x29\x71\xB4" />
<SERVICE NAME="x11" REGEX="\x31\x72\x98" />
</SERVICEPROBE>

Of course if we did use XML we would need a high quality free XML parsing
library.  Is libxml the way to go or are there better ones that any of you
recommend?

Cheers,
-F




---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).


---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).