Re: Ncat + Lua - GSOC feedback request

[David Fifield <david () bamsoftware com>]

On Sun, May 12, 2013 at 05:51:27PM +0200, Jacek Wielemborek wrote:
> 2. Ideas for the use cases
>
> So, we've got this well-documented shiny tool that speaks Lua now.
> Apart from all the joy of starting a script with #!/usr/bin/ncat -L
> header line, what else is there? I already mentioned: 1) NSE protocol
> libraries being used for more personal needs and 2) better NSE testing
> facilities.

Here are my three use cases. May I suggest that you create a SecWiki
page containing a list of use cases?

lua-exec option. This is perhaps the least ambitious, least
controversial, and easiest notion of Lua integration. I think you should
do it first before anything else. What it means is that you can do
        ncat --lua-exec program.lua
Ncat will fork and exec program.lua as if it were a shell script with
--sh-exec. In fact, the above is the same as
        ncat --sh-exec "lua program.lua"
except that you don't need a separate installation of Lua. (And also,
maybe Ncat's Lua interpreter will have libraries available by default or
something.)

Implement telnet negotiation and CRLF replacement. The -t and -C options
are implemented as special-purpose C code. (Grep for "dotelnet" and
"fix_line_endings".) It would be nice if these along with other things
could be implemented as little Lua snippets. What I'm picturing here is
not separate script files, but actual Lua source code as strings within
the Ncat source code. Each little program would be like a filter that
does CRLF substitution or whatever. This use case seems straightforward
but I think the implementation is not trivial (but neither is it super
hard). CRLF substitution needs to keep state across calls in case a CRLF
crosses a buffer boundary. Telnet negotiation needs to send data
in-band in response to information it receives. So neither is a simple
block-at-a-time byte filter.

WebSocket mode. See RFC 6455. It would be nice to do
        ncat --websocket ws://example.com:8000/
        ncat --listen --websocket ws://localhost:8000/
What this involves is first sending a special HTTP request asking for a
protocol change to WebSocket. Then you have to receive a response and do
some SHA1 calculations and do error handling. After that you can send
and receive data on the same TCP connection, but not directly: each
little block of data needs to be encapsulated in WebSocket messages and
frames. This is quite a heavyweight use case. An implementation that can
do it, can probably do most of what we would want Lua integration to do.
You should also be able to use SSL WebSocket in the obvious way:
        ncat --websocket wss://example.com:8000/

Chat mode for me is pretty much meh. If it's easy to do with some inline
Lua, fine. I don't think it's worth adding code to allow custom
nicknames or anything fancy.

I don't find the use of NSE libraries compelling. You'll have to
convince me. I think I don't yet understand the better NSE testing idea.

> 3. Hard requirements
>
> * Quite a lot of planning
>
> I definitely consider it a requirement number one for this task. I
> don't really believe that it's possible to integrate Lua with Ncat
> properly without prior planning, especially if we're heading for NSE
> compatibility. The planning phase is when I expect to need my mentor
> the most, so that the code would be written well and once, with no
> further rewriting ever needed.

If I may offer a different perspective, it may be that lots of
pre-planning (like http://c2.com/cgi/wiki?BigDesignUpFront) isn't right
for this job. Partly because we don't even have a list of requirements
yet, and only a few use cases. You would be trying to design something
that is under-specified. I am fine with building some prototypes and
throwing away some designs.

Lots of planning up front also increases the risk that there will be
nothing useful at the end of the summer. I suggest you implement
--lua-exec first, and from there you will have a better idea of the
requirements.

> 4. Open questions
>
> How much compatibility with NSE is possible for Ncat without getting
> too bloated?

I think that you should not aim for NSE compatibility unless you already
have a worked-out design for it. What would it mean for Ncat to be
compatible with http-date.nse, for example? With http.lua? I don't know
what that even means. I fear that without a design or use cases in mind,
we'll have an implementation that won't be used.

> 5. Another option – some thoughts on Ncat+Lua without NSE support
>
> There were a few design problems that I had come across. For example –
> how should the scripts be invoked and interfaced with the original
> Ncat functionality? At first, I had the idea of adding –lua-file
> command line switch that should be used along with the old-style
> connection destination specification like this:
>
> ncat localhost 2233 --lua-file hello.c
>
> Once run, the Ncat would connect to the destination in a traditional
> manner and then run all the hooks defined in the lua file – for
> example, if the file has “on_connect” function defined, it would be
> run once the connection to the remote host has been established.

Here are some examples of the callback-based model:
        https://github.com/d33tah/nmap/compare/3550b77f8e...c63a309b0d#L3R1
        https://github.com/d33tah/nmap/compare/3550b77f8e...c63a309b0d#L4R781
I worry that having callbacks for a few specific events won't be general
enough to be useful. Check your proposed design against the use cases
and see that the use cases are served. Even if we imagine on_send and
on_receive callbacks, I don't think you can even implement the CRLF and
telnet ideas correctly. Certainly a simple on_connect isn't going to
suffice for WebSocket.

What if I need something called back for every received line? What if I
need something called back every 100 ms? What if I want to count the
number of 0xff bytes that are received, after base64 decoding? I think
you will get lost trying to provide enough callbacks to be interesting.

I'm thinking of an alternative model where you have little Lua programs
that either are aware of the network and communicate using sockets, or
else read their stdin and stdout which Ncat will convert to socket
operations.

> After a bit of thinking, I thought it would be cool to add an -L
> switch that would read the lua file from standard input and run it
> once it hits an EOF.

I don't like this idea. stdin is kind of sacred to Ncat. It's one end of
the network pipe. After you read a script from stdin, does that mean you
can no longer type into Ncat? Does the script need to read from its own
stdin in order to allow that? I think the idea is confused.

David Fifield
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/