Re: Nmap Attack Scripting Language (NASL)

[doug () hcsw org]

Hello nmap-dev,

On Mon, May 22, 2006 at 06:20:26PM -0700 or thereabouts, Fyodor wrote:

> For what it is worth, we're currently looking at LUA as the embedded
> scripting language of choice.  Anyone have experience in this area?

I've looked into Lua a bit and I'm fairly impressed with what I see. The
system is compact, well designed, and well documented.

Lua was obviously *heavily* inspired by the Scheme language and contains
many of Scheme's features that I consider vital for any modern scripting
language:

* Lexical scoping, functions as first class citizens, etc.

* Incremental, user controllable Garbage Collection:
  http://www.lua.org/manual/5.1/manual.html#2.10

* Proper tail call optimisation:
  http://www.lua.org/manual/5.1/manual.html#2.5.8

* User customisable control flow. Lua calls these "coroutines" and places emphasis
  on their use in a cooperative multithreading system but really these coroutines
  are just slightly limited Scheme "continuations". (Lua's coroutines aren't
  re-invocable like Scheme continuations are):
  http://www.lua.org/manual/5.1/manual.html#2.11


I do have a couple small concerns with Lua, however:

* Internally it looks as though *all* numbers are stored as double
  precision floating point values - even, for instance, array indices:

  http://www.lua.org/pil/2.3.html

  Using floating point values makes bitwise operations much more complex;
  see my next point.

  It does seem as though Lua's primary number type might be changable by
  modifying this line in the Lua source:

  typedef double lua_Number;

  Although changing double to, for example long, leaves us with no floating
  point capabilities whatsoever!

* And Lua itself doesn't provide much functionality for dealing with binary
  data. Shifts, bitwise and/or/xor/complement, and other primitives will
  probably have to be added.

* Considering the fact that Lua is very simple and doesn't use some advanced
  byte code optimisation techniques found in decent Scheme compilers, I would
  expect a Scheme implementation to have a performance advantage over Lua in
  most cases. However, with Nmap and, in general, any network bound program,
  the way network I/O is handled is *BY FAR* the most important determinate
  of performance.


Most of these aren't major problems and some would likely have to be overcome
with Scheme as well. The fact that I'm concerned about some of these at all
might give away the fact that normally I'm a common lisp programmer and am
used to the most powerful system available. ;)


As Fyodor touches upon in his requirements document, it is vital that any
scripting language handle many different types of network I/O asynchronously.
This is definitley the most difficult part of the scripting language design
and will have to be carefuly thought out beforehand. There are couple
fundamentally different ways to do it:

* "By hand". Most of Nmap uses carefuly designed C data structures and never
  does blocking calls until a main select() loop or some such (done inside Nsock,
  for example).  Nsock, Nmap's library for doing exactly this, works very well and
  was what I used for the rDNS system. In theory, we could just give a script
  Nsock bindings but that might not make for easy-to-write scripts!

* A "callback" interface would also be possible. The user could define a number
  of standard named functions in Lua or whatever and a callback engine would have
  to be created in C that takes care of all blocking I/O and calls the user-defined
  Lua functions as appropriate. This is, sort of, the way that version detection works.

* In theory, C threads could be used. This has the advantage that scripts
  can be written simply and directly using normal blocking connect(), read(),
  and write() (whatever their counterparts in Lua will be), calls. The disadvantage
  is that C threads are usually very "heavy" and require lots of memory and
  CPU time overhead. This is a problem since we might be creating, destroying,
  and switching between, hundreds of probes per second. Also, decent threading
  can be be tough to do portably; decent thread scheduling even more so.
  (GNU pth is, IMO, the best implementation and I usually use it over POSIX).

* Finally, a continuation style could be used that makes it feel, to the
  Lua (or whatever) programmer that they are using direct blocking calls.
  connect, read, write, etc would have to be defined so that when
  they are called they put a descriptor into a "read queue" or a "connect queue",
  store the current continuation and resume another "scheduling continuation"
  which continues processing continuations until its "run queue" is empty
  then do a big select() call on all collected descriptors. Then, once
  select() returns, the offending descriptors would have to be found and
  their associated continuations, well, continued.

  I have done empirical tests with Scheme continuations in the past and I've
  found that each continuation takes only a few hundred memory cells (typically
  8 to 32 bytes each) and creation/destruction/context switching is negligible.
  I can't speak for Lua "coroutines" but suspect they are roughly comparable.


In my personal opinion, Lua's biggest flaw is that it doesn't use S-expression
scheme/lisp syntax. This makes some things that Scheme/lisp programmers take for
granted impossible: macros, symbolic computation, "code as data", etc, etc, etc.

That said, I think Lua is the best tool available for this task because
of exactly one feature: its lack of S-expression scheme/lisp syntax. :)

If we want this scripting interface to be as widely accesible as possible (which
I believe we do) we want a straightforward easy programmer interface. Do
we really want to be fielding questions on nmap-dev like
"lol buds what's a cadr!??! k thxbye"?

Besides, as any scheme/lisp programmer knows, when you're dealing
with S-expressions, syntax is irrelevant! If you really want to code in
S-expressions (who can blame you?), writing a compiler is trivial:

;; Simple Scheme to Lua compiler. Only supports basic
;; arithmetic, '=, and 'if for now. Infix limitations also apply.

(define (scm->lua f)
  (cond
    ((pair? f)
      (case (car f)
        ((set!) (string-append (scm->lua (cadr f)) " = " (scm->lua (caddr f))))
        ((+ - / *) (string-append "(" (scm->lua (cadr f))
                                  " " (atom->string (car f)) " "
                                  (scm->lua (caddr f)) ")"))
        ((if) (string-append "if " (scm->lua (cadr f)) " then\n"
                               (scm->lua (caddr f))
                             "\nend"))
        ((=) (string-append (scm->lua (cadr f)) " == " (scm->lua (caddr f))))))
    (else (atom->string f))))



Now we can compile scheme forms like so:

> (scm->lua '(if (= a 1) (set! b (+ (- b 2) (* a 3)))))
"if a == 1 then\nb = ((b - 2) + (a * 3))\nend"


Doug



_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev