Re: Nmap-NSE Release candidate 2

[Fyodor <fyodor () insecure org>]

On Wed, Aug 09, 2006 at 05:05:51AM -0700, doug () hcsw org wrote:
> I'm glad to hear that except I'm afraid you might be doing a lot of jumping
> up and down as this is a very long piece of feedback. :)

Indeed it is :).  You make many good points.  I think Diman is mostly
focused on stabilization and bug fixing right now, but planning for
the future is definitely valuable.  In the same way, the suggestions I
make in this email may also happen post-integration.

> I don't know if there are any plans to add user-specifiable parameters
> into NSE but I have a feeling that eventually we will run into this sort of need.

I agree that we may need that eventually.  Though the vast majority of
our scripts should not have any required parameters, it may be OK to
offer options (like your FTP bounce attack anon ftp password example).

> The other method used in your scripts, Diman, will probably be more effective
> because it will be applied depending on the service detected. For instance:
>
> showHTMLTitle.lua:portrule = function(host, port)
> showHTMLTitle.lua-      if
> showHTMLTitle.lua-              (       port.number == 80
> showHTMLTitle.lua-                      or port.service == "http")
> showHTMLTitle.lua-              and port.protocol == "tcp"

Yeah, I agree that this "OR" format is usually superior to the type
which requires the canonical port number AND service name.

> But, by the same token, if port.number is 80 and port.service is known to *not* be http,
> do we really want to run this script? I think it might be better to always ignore
> port numbers and only depend on the service name + transport protocol.

I suppose an issue with that is whether we want to support -sC without
-sV.  If so, the scripts won't have a version-detected port.service
(though I suppose they could fall back on nmap-services).  And I can
also see strong arguments for making -sC imply -sV.  After all,
version detection is a sort of basic scripting.  In fact, I also think
-sV should probably imply -sC for just the version detection category
of scripts.  That's where we can put the skyp2 test, for example.

> One of the biggest benefits of version detection is that it makes people realise
> just how little security benefit they get from obfuscating port numbers and runnning
> services on non-standard ports.

Hehe, here is an amusing quote about that from the FD list:

  "One application had all the ports reassigned to all non standard
   ports. When I asked why such a brain dead thing was done, they said
   it was for security, and that "it would be too much work to find
   these ports". Then I showed them nmap with the port identification
   option. Their jaw dropped to the floor. They had *NO*
   security. Anonymous ftp world writable, ..."
     --http://seclists.org/fulldisclosure/2006/Mar/1533.html

A risk of relying on only version detection is that it causes
cascading failures if VD fails to identify the service.  So for
well-known ports like 22,25,53, etc, it probably makes sense to leave
the port # there as an option.  Or maybe just ensuring that the
nmap-services name is populated in that field in the case VD fails to
identify it (which is what normal Nmap output does, with the low
confidence rating causing the question mark after the service name).

> Perhaps a useful abstraction that could be made is to have a shortcut
> in portrule. For instance, what if portrule could be either a function
> OR a string? First of all, thank lua's dynamic typing system. Then
> consider being able to specify portrules like the following:

Good point.  And I like Diman's suggested solution to this (the
"portrule = protorule("http", "tcp") stuff).

> I know lua provides a fairly complete regular expression library but I still
> wonder if it would be worthwhile creating some sort of lua binding for
> PPCRE. Nmap requires libppcre for version detection so we can always count on
> it being linked.

Heh, I asked Diman the same thing as soon as I learned about LUA
"patterns".  They are somewhat powerful, but certainly no match for
PCRE!  I think they will work for most of our needs, but I also would
like to see us bind to libpcre at some point.

> (Interestingly, Nmap will soon have 3 distinct, incompatible
> regular expression libraries available - POSIX, PPCRE, and lua's)

Nmap would have probably just used POSIX instead of adding pcre
(though pcre is better), except that I don't think the POSIX regular
expressions are available on all of the platforms Nmap runs on.

> One of the best things about designing a scripting language extension is that we
> can hunt for and discover common scripting idioms and make them conveniently
> available to the scripter.

Absolutely!

> If we take this to a further level of abstraction, we could make use of
> this same tactic in NSE. Imagine a function
>
> get_contents_until_match = function(regex)

Great idea!

> I can't seem to find any use of the 'description values in the C source or
> any of the lua scripts. Does the description serve any purpose other than
> being a comment in the source code?

Besides being a useful comment in the source code, I would like to see
them put in (at least) the XML output at some point.  Maybe they could
be included in normal Nmap too, with a list at the end of the entire
(all targets) run with the description of each script which produced
output against at least one host during the scan.

> Auto-documentation like this is a very powerful concept and could perhaps
> be developed further. Consider quickly being able to see and choose exactly
> which scripts will be applied against a discovered service.

Yeah, though we can't require too much interactivity in Nmap itself.

> In a program I started work on a while ago, which I call nuff, I make
> extensive use of scheme macros and this sort of auto-documentation concept so
> that a utility can automatically be well-documented. The parameters, their types
> and acceptable values, whether this utility requires root privileges, etc etc,
> can all be automatically determined and documented. I hope to someday muster
> up the time and energy to complete this set of macros for publication.

Yeah, that can be valuable stuff.  Look at the auto-generated Nessus
NASL script web archive, for example.

> Out of curiosity, I'm wondering why you chose NSE for some of your scripts
> instead of version detection. For example, malware/kibuvDetection.lua seems
> to accomplish the same thing that adding a match line or 2 to the probes file
> would.

Hmm ... a very fascinating test would be to run Nmap version detection
and Nmap NSE with just the kibuvDetection script against the same
large group of IPs and compare the scan times and accuracy.  I'd love
to hear those results.

Cheers,
-F



_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev