Ncat-Lua: a complete status report for development and discussion

[d33 tah <d33tah () gmail com>]

Hi guys,

TL;DR:

This e-mail is about the Ncat-Lua GSoC project. If you care about its
development, jump to any section you find interesting (or, most
preferably, just read the whole message) and comment on it in an
appropriate mailing list thread. If you don't want to bother looking
for the thread (it might not be there), e-mail me and I'll read the
message and forward it where it belongs. I really need some feedback,
so... thanks in advance!

The long version:

This e-mail is an update on my Ncat-Lua project, for the ones
interested in the development of Lua scripting support for Ncat. I'll
try to explain here what the new Ncat can do and what it still can't.
Should you find any part of this e-mail worth commenting on, don't
hesitate to! Implementing new Ncat Lua features without feedback or
use cases feels quite like walking in the dark, which is why I would
definitely appreciate your comment.

The current Nmap's trunk includes the Ncat version with --lua-exec
<filename> feature merged in. For those who hear about it for the
first time, it's basically an equivalent of --sh-exec "lua <filename>"
- the script given as the parameter will be ran with its standard
input and standard output redirected to the connection. It proved to
be enough to implement some simple demo services like ECHO, DAYTIME or
DISCARD. It can do much more though - as I showed in
ncat/scripts/conditional.lua, you can write a bit more complex service
as well, making decisions based on what user types in. This approach
has a few shortcomings though.

While playing with --lua-exec's capabilities, I also figured out a way
to print to the Ncat's output streams. The trick is to use the
io.stderr from inside of the Lua script, which isn't modified in the
forking process, so it still effects in printing the message to the
terminal - this works in Windows as well. It's cool, but it pretty
much calls for the access to Ncat's standard input as well.

**ncat-lua-script branch**

Since I already had quite a few use cases/user stories to justify the
feature, I set out to experiment with an implementation prototype for
reading Ncat's STDIN in ncat-lua-script branch. I added this feature -
at the moment to *nix systems only - by creating another pipe between
the parent and child process and adding it as "io.ncatin" to the
--lua-exec scripts. This is only available if you specify
--lua-extensions in the command-line.

You can see some new shiny demos there - I hacked up a demo that can
connect to echo.websocket.org via a stub of Websocket protocol, there
are ROT13 and Base64 filtering demos as well. I also added an active
filter example - telnet.lua, which transparently responds to the
Telnet's control commands and cuts them out of the output (which is an
improvement over the -t command-line switch in "vanilla" Ncat).
There's also a shiny IRC demo that chooses a random nickname and joins
the #bots channel, allowing you to speak there (tested on Freenode)!

While implementing these, I quickly started to need some kind of
select() to watch over io.stdin and io.ncatin. I implemented this as a
function that takes a table of file instances and removes the values
that don't have any data to read. This dramatically cut the CPU
footprint of the scripts and allowed to make the IRC demo (and not
only) feel real-time. Unfortunately, it also led to the question of
how should this be implemented (see the "Open issues" section).

Oh, by the way, thanks to restoring the default SIGPIPE handler inside
of the child process, I could finally implement the CHARGEN demo - the
scripts will simply die when the connection closes.

**Open issues:**

Below is the current status of the discussion on Ncat-Lua. It's my
attempt to summarize the issues the project has and point to the
relevant threads on this mailing list. If you'd like to speak up on
any of these topics and I hadn't mentioned a link to a seclists thread
here, please consider starting a new one. This way we could avoid the
mess that started to be forming here:
http://seclists.org/nmap-dev/2013/q2/276

The following list will include some features that are lacking use
cases / user stories. Implementing them without these is pretty much
pointless, because it's hard to figure out a good design to something
that isn't well specified. Thus the question I'd like to ask you -
would you find any of these features useful? Or maybe do you think the
opposite way - that a particular feature is completely redundant?
Those would be very valuable signals for me.

* Stdlib

While looking for a good way to implement io.select(), Patrick
suggested me to mimic the interfaces from Luasocket. I actually
stumbled upon the very same problem a few times - websockets have
their own implementation in Lua as well
(https://github.com/lipp/lua-websockets/), same goes for the eof
function I removed in r31245. I'm starting to think that instead of
reinventing the wheel, it might be a good idea to copy/fork some
useful libraries and bundle them with Ncat. Or make Ncat work with
Luarocks/Luadist package management systems.

* Interfacing with Ncat - o.target, reacting to SIGCHLD

I believe that Lua scripts would benefit from a bit more interfacing
with Ncat. For example, the Websocket script could be interested in
the particular hostname that "o.target" Ncat variable contains so that
it would handshake with a valid domain. The port would probably be
useful as well, not to mention the information on whether the SSL is
enabled or not. There's quite a lot of things that could be added, but
the question is - which of them do you find actually useful, as a
future Ncat script developer?

* State sharing, server facilities

Although the current implementation already lets you code up pretty
sophisticated service servers, you can't for example create even a
simple chat server that keeps track of its users in order to broadcast
them messages. I think it could also be useful to be able to decide on
whether to accept the connection or not.

* Parameter passing and invocation

--lua-extensions is a playground feature and some of its sub-features
might be dropped or renamed. It's obvious though that “ncat
--lua-extensions --lua-exec scripts/ws.lua echo.websockets.org 80” is
ugly and has to be replaced with something like “ncat --with ws
echo.websockets.org 80”. Discussion on this topic can be found here:
http://seclists.org/nmap-dev/2013/q2/501 and
http://seclists.org/nmap-dev/2013/q2/461 .

* Stacking

Websockets, ROT13/Base64, Telnet filters... at the moment they limit
the user to using one particular script. It would definitely be useful
though to be able to stack scripts, so that you could for example
encode your session with Base64 while sending it over Websockets.

* Making new connections

httplookup, one of the use cases on the secwiki page I created
(https://secwiki.org/w/Ncat-Lua) seems to imply that Ncat's scripting
engine is capable of creating another connections. It's yet to be
discussed if such functionality would actually be used. This is the
current description of the use case - comments are of course welcome:

> I want a service I can connect to and would let me perform HTTP
> queries the same way nslookup lets me peform DNS ones. After
> connecting, I will be greeted with '> ' command prompt. I first start by
> setting the remote host using the command "server nmap.org" and
> send the newline character. Then I'd like to type in the first line of the
> request (without the HTTP version), send the newline and get the reply
> headers, like in this example session:
>
> > server nmap.org
> > GET /
>
> HTTP/1.1 200 OK
> Date: Fri, 28 Jun 2013 20:28:42 GMT
> Server: Apache/2.2.15 (CentOS)
> Accept-Ranges: bytes
> Transfer-Encoding: chunked
> Content-Type: text/html; charset=UTF-8

* Read-write loop layout

The read-write loop in Ncat filter scripts (the ones using io.ncatin)
is starting to follow a quite clear pattern. At the beginning of the
script there is some initialization phase code, with the definitions
of functions and variables that will be used later. Then there's an
infinite loop, in which the script performs a select on both io.ncatin
and io.stdin and then performs actions depending on the contents of
the array returned by select(). As utility functions, I kept adding
table.find so often that I created a special library for it
(ncatlib.lua).

Prior to my work started in r31287, base64.lua and rot13.lua shared so
much code that I finally thought it might be useful to create
run_filter, a function that takes “encode” and “decode” functions,
reads lines from the socket and ncatin in an infinite loop and parses
the input it with the functions from the parameters. It's not the
perfect solution though - I can't wrap telnet.lua with the run_filter
as it looks now. Despite that, I believe it might be the good
direction.

** Conclusion **

So, well, that's it. As you can see, there are some awesome features
on the horizon, but in order to reach that stage, we need more use
cases and discussion. That's why - again - feel free to comment on
anything you find interesting here, any feedback is welcome. Thanks!

Yours,
Jacek Wielemborek
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/