RE: Nmap Service Detection Proposal

["Jay Freeman \(saurik\)" <saurik () saurik com>]

Your point "a" is one of the main issues I was trying to deal with while
writing the current versions of nmap+V.  If you do an -sT type scan I am
even reusing the existing connections that nmap created to minimize
reconnects.  The real problem with this method (that people are apt to point
out, and I very much agree with) is that you can't detect, as you mentioned
in your e-mail, non-responsive protocols that are on very incorrect ports.
Here are a few examples of problems you run into:

If you just connect to the port, not get a banner, and send an HTTP client
request, you will find a lot of servers.  You will not find PostgreSQL
servers, CVS pservers, most IRC servers, and you will make Finger servers
slightly more difficult to detect (as many will then try to find users
"GET", "/", and "HTTP/1.0", instead of returning a user list (difficult, but
not impossible to notice) or "No one logged on", which is at least a de
facto standard for most daemons).  There are all sorts of other protocols
that don't do anything when confronted with an HTTP client response, or even
close the connection.

Instead you can send "\r\n\r\n", which will root out the PostgreSQL, CVS,
and many Finger servers, but not IRC servers, or most of the other protocols
out there.

If your goal is just to detect non-responsive daemons on common ports, and
responsive daemons on any port (which is luckily most of the really useful
ones, such as SMTP, POP, etc.) as accurately as I can, and maybe even send
an HTTP client response as a last ditch effort and look for a web server
(not currently in nmap-versions, but just requires a well placed "g 101"
command right after "p 106 8009"), all while doing it stealthily, you might
as well be using nmap+V.  If all you don't like is my language, that's very
easy to change, I was just looking for something that let's you look at it
in a non-syntax highlighting text editor, and still not have the total line
noise experience (at least the numbers and commands and starts of most
arguments line up on the left for easy browsing).

Nmap+V, with a single connection, is just as powerful as the snippet of an
example of the language Fyoder came up with, and if used at that level of
power doesn't even look THAT much different even with my existing language:

http http/1

Becomes:

! 0 {http} {} ^http/1

While offering much _more_ ability to root deeper into the protocol and get
more information from the single first connection, such as the application
serving the request, the version of the daemon, extraneous information such
as the <title/> of a web page or the network of an IRC server (the two
examples of which are already implemented as of nmap+V-2.1 from a few months
ago using multiple "-sV"'s on the command-line).  I could even write a very
simple shell script to take that configuration file and translate it into my
existing one.  I also find case sensitivity to be extremely important, if a
server responds with "+ oK", it isn't POP3.  It looks like POP3, but it
isn't, and clients aren't likely to work with the server anyway, it is more
likely to be a badly written prank.

It also isn't going to be as easy as the example.  You are still going to
have to look for complex matches such as:

220 ArGoSoft Mail Server, Version
201 Microsoft Exchange Internet News Service Version
502 Permission Denied - Permission Denied - Powered by Typhoon -

As many protocols don't have a strict standard for how to offer up banners,
or even what status codes need to be sent immediately after connection.  The
HTTP extended status codes are actually listed in the RFC as rough
guidelines that can be modified to fit the site's needs, and most clients
will just look for the really common ones, and then the first digit for the
general idea: "5xx? Yep... error!".  The way most of the older protocols
(also read as: most commonly used) work, as is also illuminated by my
explanation of Finger in my previous post, is to not include a good way to
immediately verify that the server you are talking to is this protocol and
not some other protocol.  You simply write the client and server with
certain assumptions in mind, connect them together, get them talking, and if
one does something the other doesn't expect, they bail (or in the case of
Finger, return the data to the user and call it a success :) ).

As for multiple connections (which you seem to be against anyway) and using
the ports as hints as to what to try first, I can modify my parser to
include that functionality (by keeping track of 'p' commands that are used,
and then going back through the file again and passing over the branch) or
make it slightly more efficient than that with a slight change in the file
format.

Is this simplistic model of approximate detection what people are looking
for?  Personally, I don't think it is....  If so I will stop working on the
XML versions of the configuration files I started last night, which should
give a multi-level protocol detection scheme, easily handle multiple
connections to root out every last protocol that could be sitting on a port,
and then go arbitrarily deep into the protocol to get any information
requested, all while maintaining separation between protocols and the hints
files.  The real problem with this is the obvious "but this is going to
require lots of connections to the server, isn't it?"  Loss of stealth is
the price paid for more information....

Luckily, the way I am building the new configuration files for nmap+V should
allow for a command-line option that tells nmap to "forsake extra/precise
information in exchange for using a single connection.", and hopefully still
be able to reuse a -sT connection (although this is more iffy), so using
more connections shouldn't decrease the overall ability of the system (as
you seemed to say in your FTP example), but has the possibility of making it
more powerful.

The general idea is partially separating protocol detection from I am just
starting porting some of the protocols over from nmap-versions so I can see
what the requirements on the language will be (after I have a few of them
over and I like the system I will start writing the parsing code), but here
is some scattered pieces out of _.xml (the main startup file), HTTP.xml, and
RFB.xml:

++++++
    <match regex="220[ -]"> <!-- last ditch effort for 220 protocols -->
      <try xml="SMTP"/>
      <try xml="FTP"/>
    </match>
++++++

++++++
  <prod data="\r\n\r\n">
    <match exact="cvs \\[pserver aborted\\]: bad auth protocol start:">
      <try xml="CVS"/>
    </match>
++++++

++++++
<protocol short="RFB" long="Remote Framebuffer">
  <fork>
    <require length="12" keep="1" regex="RFB
([0-9][0-9][0-9].[0-9][0-9][0-9])\n">
      <set var="PROT-VER-MIN" sub="1"/>
      <set var="PROT-VER-MAX" sub="1"/>
    </require>
++++++

++++++
<protocol short="HTTP" long="Hypertext Transfer Protocol">
  <fork>
    <send data="GET / HTTP/1.0\nUser-Agent: Mozilla/4.0 (compatible; MSIE
5.01;
Windows NT 5.0)\n\n"/>
    <require until=" " max="24" keep="1" regex="HTTP/([0-9]+\\.[0-9]+) ">
      <set var="PROT-VER-MIN" sub="1"/>
      <set var="PROT-VER-MAX" sub="1"/>
    </require>
++++++

And I realize I am using \ escaping instead of XML entity escaping (my XML
syntax highlighter highlighted \n, so I thought I would use it for now and
"rpl" it out later).

I'm going to allow for forking a connection (so you got some information out
of this connection, now we need to fork, start a new connection, get
something, and return back to the previous connection), as well as totally
ending the current one and starting a new fork.  I'm using sub-expression
regular expression parsing for the basis of data acquisition.

My goal is that the individual protocol files will allow for totally
stand-alone protocol verification and information gathering and will
hopefully go a few levels deep into the protocol: the RFB one is going to
send the appropriate replies, and get up to the authorization section, and
the HTTP one is going to read more of the headers returned to decide if it
is a web server, a Servlet engine, VNC, or something else sitting on the
opposite side of the connection.

I want to be able to use the protocol itself to fingerprint the server as
opposed to requiring the server to explain itself: RFB doesn't tell me what
is on the other side, but knowing what option flags, different
authentication types, and encoding methods the different versions support I
should give a good idea as to the version of VNC (or some other application)
that is being run.

I also want (but am still thinking about the best ways of doing this) to use
the connections to gather as much information as possible, and present the
extra information back to the calling application in an XML format designed
for that protocol.  I'm going to try to work this into a command line switch
much like my current one (as extended protocol information gathering could
require more and more connections).  If an application doesn't know what the
protocol is, it likely doesn't care about the information anyway, so it
isn't important that these extended information bits follow a standard to be
followed across services.

Sincerely,
Jay Freeman (saurik)
saurik () saurik com

-----Original Message-----
From: Paul Tod Rieger [mailto:prie () abl com]
Sent: Tuesday, August 29, 2000 1:13 AM
To: nmap-dev () insecure org
Subject: Re: Nmap Service Detection Proposal

In support of the need for service detection,

http://www.sunworld.com/sunworldonline/swol-08-2000/swol-0818-unixsecurity.h
tml

points out that, in order to bypass those "pesky" corporate
firewalls,  "developers are more and more frequently building
applications that run via port assignments that are well known
and commonly used -- the HTTP and HTTPS ports (80 and 443,
respectively)."


Back to the subject:  Fyodor <fyodor () insecure org> proposed:

> that the nmap-service-magic file contains a list of "probes".  Each probe
> contains the following information:
>
> 1) A list of common ports for the services detected by the probe (for
>   implementation optimization only -- an open port will first be tested
>   with probes that list that port number).
>
> 2) A string to be sent to the port right after connection establishment
>   (if TCP).  The string can include escaped binary chars.
>
> 3) A list of (case insensitive) regular expressions to match against the
>   response and the protocol name the regexp relates to.  The regexps. can
>   contain escaped binary chars as well.


and gave an example of the nmap-service-magic file grammer:

> # The catch-all HTTP probe (which leads to distinctive error msgs from
> # many services
> Probe TCP=21,22,23,25,80,110,118,1080,8080 SEND="GET / HTTP/1.0\r\n\r\n"
> ftp  220.*ftp
> pop3 +OK
> ssh  ssh-
> smtp smtp
> nntp posting ok
> http http/1
>
> # Probe X11.  I made up the hext values, presumably they would
> # correspond to xome sort of X request
> PROBE TCP=6000-6010 SEND="\x32\x28\x14\x29\x71\xB4"
> x11  \x31\x72\x98


and in XML as well.  All of this looks good to me, but, of course,
I have a couple of questions:

a) "an open port will first be tested" -- does this mean a port may
be tested multiple times?  Will this be stealthy?  For instance, if
port 21 really is FTP but wrapped under tcpd, multiple tests won't
be able to identify it -- and they might look like aggressive/lame
cracking attempts.  (Also, tcpd may slow down response times even
when connections are permitted....)

Instead, if the service can't be identified from a single test,
maybe it could just be flagged for closer inspection by the user.


b) for ftp, pop3, ssh, and smtp -- if nmap grabs the line that
contains the regexp match, aren't chances pretty good that you also
have the product & version?  (Also, your "220.*ftp" seems to take
care of Saurik's concerns re: uniquely identifying FTP....)


Tod
abl.com*


---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).