luaffi for Nmap

[Patrick Donnelly <batrick () batbytes com>]

In effort to clear up any confusion I may have caused in the IRC
meeting [1], I want to lay out the situation more clearly. Especially,
I want to make clear the Lua ecosystem:

Lua is maintained by a committee of 3 people at PUC-Rio in Rio de
Janeiro, Brazil [2]. It is a small development team that keeps
development closed but source open. They do communicate quite
frequently on the lua-l mailing list though. For the purposes of
discussing different implementations of Lua, many use PUC-Lua to
distinguish (as many use CPython for Python's main implementation).
Still, PUC-Lua is the official Lua source and so plain "Lua" is more
frequently used.

I also want to make clear that Lua is "ANSI C" only with a very
limited number of exceptions. These exceptions are only in libraries,
not in the language core. The require function and dynamic shared
library loading is one example of breaking ANSI C (in terms of using
functions not offered by ANSI C). Lua's core design philosophy is to
build the mechanisms, let the downstream providers do the rest. This
is particularly true for libraries.

LuaJIT [3] is a one-man project by the illustrious Mike Pall. LuaJIT
is renowned for being the fastest (trace) JIT compiler for dynamic
languages in existence. Besides being a great JIT compiler and
interpreter (the interpreter alone is about 2-5 times faster than
PUC-Lua), LuaJIT also offers a Foreign Function Interface (FFI) [4].
This was a solo effort Mike Pall took on that is unique for LuaJIT.
The #1 goal of the project was to eliminate the overhead of Lua->C
calls for bindings (e.g. NSE's nsock is one such binding).

I emphasize there is no official ties between LuaJIT and PUC-Lua. In
fact, LuaJIT has decided to forge it's own hybrid path with the
release of Lua 5.2 by cherry picking features that Mike Pall decides
are worth having while discarding the rest. This isn't necessarily a
bad thing as Mike is 100% focused on Lua 5.1 support and speed. It
does introduce fragmentation, however.

LuaJIT's FFI library has been very well received by the Lua community.
It has also been discussed widely in other forums. Because of its
success, there have been efforts to create a fork [5, 6] to be used by
PUC-Lua. [Keep in mind, as with all JIT compilers and third-party
implementations of a language, not everyone is able to or willing to
make the switch from the official language source. Nmap, in my
opinion, has little to gain, currently, from switching to LuaJIT.]
This library was, like most Lua bindings and LuaJIT, developed outside
of the Lua team's influence (the 3 person committee). luaffi will
probably never be incorporated into PUC-Lua.

Still, we may be able to use luaffi in Nmap for our own purposes. I
brought this topic up in IRC in order to expose another avenue to add
support for ssh (via libssh). The three possible options initially
discussed were (from IRC):

1) Integrate a 3rd party C library with acceptable license such as libssh2
2) Implement it in Lua ourselves or with Lua SSH library (which we
haven't really been able to find, I don't think)
3) Do external call out to ssh binary, which avoids bloating Nmap with
a library, but then it only works if they have ssh installed, and I
guess on Windows we'd have to include an ssh.exe probably

The first option is the writing of a C binding for NSE. We do this for
nsock, pcrelib, and others.
The second option is the usual approach (where feasible) of
implementing the protocol in plain Lua.
The third option is very unfavorable as it requires the presence of
some executable (e.g. "ssh") and a common command line interface.
Windows makes this option particularly difficult.

I later brought up the fourth option which is luaffi. I believe it's a
possible mechanism for binding libraries to Nmap using plain Lua code.
It would definitely help with generating difficult scripts more
quickly as the libraries for a particular protocol, e.g. ssh, can be
quickly bound and used so the actual task (whatever that may be) can
be achieved. [For example, the binding for the compression library
zlib is quite small [7].] Best of all, there is no bloating of Nmap as
these libraries can be conditionally bound at runtime. In fact, if the
library is not available, a smart script will abort running at all
without causing a fuss.


Anyway, I hope I cleared up the confusion about this so people can
make a more educated decision about what to do or not do with luaffi.
There are still things to explore, such as stability of this luaffi
fork.

[1] http://seclists.org/nmap-dev/2012/q2/799
[2] http://www.lua.org/about.html
[3] http://www.luajit.org/
[4] http://luajit.org/ext_ffi.html
[5] http://lua-users.org/lists/lua-l/2011-07/msg00467.html
[6] https://github.com/jmckaskill/luaffi
[7] http://luajit.org/ext_ffi_tutorial.html#zlib

-- 
- Patrick Donnelly
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/