Re: Ncat + Lua - GSOC feedback request

[Jacek Wielemborek <wielemborekj1 () gmail com>]

On Fri, June 14, 2013 at 10:43:42 David Fifield wrote:
> On Sun, May 12, 2013 at 05:51:27PM +0200, Jacek Wielemborek wrote:
> > 2. Ideas for the use cases
> >
> > So, we've got this well-documented shiny tool that speaks Lua now.
> > Apart from all the joy of starting a script with #!/usr/bin/ncat -L
> > header line, what else is there? I already mentioned: 1) NSE protocol
> > libraries being used for more personal needs and 2) better NSE testing
> > facilities.
>
> Here are my three use cases. May I suggest that you create a SecWiki
> page containing a list of use cases?

Unfortunately I can't. I reported my problems here:
http://seclists.org/nmap-dev/2013/q2/449

> lua-exec option. This is perhaps the least ambitious, least
> controversial, and easiest notion of Lua integration. I think you should
> do it first before anything else. What it means is that you can do
>       ncat --lua-exec program.lua
> Ncat will fork and exec program.lua as if it were a shell script with
> --sh-exec. In fact, the above is the same as
>       ncat --sh-exec "lua program.lua"
> except that you don't need a separate installation of Lua. (And also,
> maybe Ncat's Lua interpreter will have libraries available by default or
> something.)

I like this idea as a starting point - I understand that all the the calls to 
io.write()/print() should direct to the socket instead of the stdout and all 
calls to io.read() should read from the string? (oh, by the way, is the 
forking really necessary?)

> Implement telnet negotiation and CRLF replacement. The -t and -C options
> are implemented as special-purpose C code. (Grep for "dotelnet" and
> "fix_line_endings".) It would be nice if these along with other things
> could be implemented as little Lua snippets. What I'm picturing here is
> not separate script files, but actual Lua source code as strings within
> the Ncat source code. Each little program would be like a filter that
> does CRLF substitution or whatever. This use case seems straightforward
> but I think the implementation is not trivial (but neither is it super
> hard). CRLF substitution needs to keep state across calls in case a CRLF
> crosses a buffer boundary. Telnet negotiation needs to send data
> in-band in response to information it receives. So neither is a simple
> block-at-a-time byte filter.

I'll look into that, at the moment I don't really know much about the 
negotiation protocol (but I'll catch up). I'm a bit worried about hardcoding 
Lua code in the Ncat core though - what if the user wants to drop Lua 
functionality? Fallback to C routines? Or should we make Lua a requirement for 
Ncat to compile?

> WebSocket mode. See RFC 6455. It would be nice to do
>       ncat --websocket ws://example.com:8000/
>       ncat --listen --websocket ws://localhost:8000/
> What this involves is first sending a special HTTP request asking for a
> protocol change to WebSocket. Then you have to receive a response and do
> some SHA1 calculations and do error handling. After that you can send
> and receive data on the same TCP connection, but not directly: each
> little block of data needs to be encapsulated in WebSocket messages and
> frames. This is quite a heavyweight use case. An implementation that can
> do it, can probably do most of what we would want Lua integration to do.
> You should also be able to use SSL WebSocket in the obvious way:
>       ncat --websocket wss://example.com:8000/

I love this one - it would add some really new functionality to Ncat and be a 
great demo of what the Lua engine could (and should) be be capable of. The 
only thing that I don't exactly understand is the usage - do you mean that 
every Lua script should be capable of messing with argv interpretation (sounds 
dangerous) or WebSocket should be a special script? Option three that comes to 
my mind is invocation like:

ncat --lua-exec "websocket,url=ws://example.com:8000"
ncat --lua-exec "websocket" example.com 8000
ncat --lua-exec "websocket" --ssl example.com 8000

ncat --lua-exec "websocket,url=ws://example.com:8000,mode=listen"
ncat --lua-exec "websocket" -l -p 8000 

These examples raise questions about how much of the the Ncat internal 
interface should be exposed to the user - my initial plans suggested to allow 
the scripts to read (and perhaps change) the "options" structure, here I also 
introduce a special script parameter list. That's probably going to be quite 
some work for me to do.

> Chat mode for me is pretty much meh. If it's easy to do with some inline
> Lua, fine. I don't think it's worth adding code to allow custom
> nicknames or anything fancy.

I pretty much considered it a nice server demo - from the beginning I assumed 
that Ncat-Lua should cover all the Ncat's functionality, but when I now think 
of it, perhaps it's better to focus on the client side first.

> I don't find the use of NSE libraries compelling. You'll have to
> convince me. I think I don't yet understand the better NSE testing idea.

(I'll reply to that below, where you answered to the section 4 of my original 
e-mail, which is also about NSE)

> > 3. Hard requirements
> >
> > * Quite a lot of planning
> >
> > I definitely consider it a requirement number one for this task. I
> > don't really believe that it's possible to integrate Lua with Ncat
> > properly without prior planning, especially if we're heading for NSE
> > compatibility. The planning phase is when I expect to need my mentor
> > the most, so that the code would be written well and once, with no
> > further rewriting ever needed.
>
> If I may offer a different perspective, it may be that lots of
> pre-planning (like http://c2.com/cgi/wiki?BigDesignUpFront) isn't right
> for this job. Partly because we don't even have a list of requirements
> yet, and only a few use cases. You would be trying to design something
> that is under-specified. I am fine with building some prototypes and
> throwing away some designs.
>
> Lots of planning up front also increases the risk that there will be
> nothing useful at the end of the summer. I suggest you implement
> --lua-exec first, and from there you will have a better idea of the
> requirements.

Hm, we already talked about this on IRC and I agree that might be the better 
option. So, instead of strictly following the timeline I wrote for my 
proposal, I'll be focusing on --lua-exec for now.

> > 4. Open questions
> >
> > How much compatibility with NSE is possible for Ncat without getting
> > too bloated?
>
> I think that you should not aim for NSE compatibility unless you already
> have a worked-out design for it. What would it mean for Ncat to be
> compatible with http-date.nse, for example? With http.lua? I don't know
> what that even means. I fear that without a design or use cases in mind,
> we'll have an implementation that won't be used.

Honestly, I don't have any design for the NSE part (and the IRC log from my 
last e-mail suggests it would be a really difficult task). My idea of 
compability with http-date.nse meant that the user runs something like: 

ncat --script http-date localhost 80

And the action method would be run against the server. The compability with 
http.lua would mean that it would be possible to use the library inside the 
Ncat scripts.

I wouldn't mind dropping NSE support for now though; it was just an idea I 
meant to pass to you guys and find out if I it's possible to do in three months 
along with a Lua interface to the rest of Ncat's functionality (with some 
mentoring of course). 

> > 5. Another option – some thoughts on Ncat+Lua without NSE support
> >
> > There were a few design problems that I had come across. For example –
> > how should the scripts be invoked and interfaced with the original
> > Ncat functionality? At first, I had the idea of adding –lua-file
> > command line switch that should be used along with the old-style
> > connection destination specification like this:
> >
> > ncat localhost 2233 --lua-file hello.c
> >
> > Once run, the Ncat would connect to the destination in a traditional
> > manner and then run all the hooks defined in the lua file – for
> > example, if the file has “on_connect” function defined, it would be
> > run once the connection to the remote host has been established.
>
> Here are some examples of the callback-based model:
>       https://github.com/d33tah/nmap/compare/3550b77f8e...c63a309b0d#L3R1
>       https://github.com/d33tah/nmap/compare/3550b77f8e...c63a309b0d#L4R781
> I worry that having callbacks for a few specific events won't be general
> enough to be useful. Check your proposed design against the use cases
> and see that the use cases are served. Even if we imagine on_send and
> on_receive callbacks, I don't think you can even implement the CRLF and
> telnet ideas correctly. Certainly a simple on_connect isn't going to
> suffice for WebSocket.
>
> What if I need something called back for every received line? What if I
> need something called back every 100 ms? What if I want to count the
> number of 0xff bytes that are received, after base64 decoding? I think
> you will get lost trying to provide enough callbacks to be interesting.
>
> I'm thinking of an alternative model where you have little Lua programs
> that either are aware of the network and communicate using sockets, or
> else read their stdin and stdout which Ncat will convert to socket
> operations.


Now that I think of it, it might be really hard to implement stuff with 
callbacks (probably would involve lots of hacks using global variables to 
share states in weird ways in order to do anything I hadn't thought of). 
Stacking callbacks could probably help, but I'll be happy to try your idea 
first.

> > After a bit of thinking, I thought it would be cool to add an -L
> > switch that would read the lua file from standard input and run it
> > once it hits an EOF.
>
> I don't like this idea. stdin is kind of sacred to Ncat. It's one end of
> the network pipe. After you read a script from stdin, does that mean you
> can no longer type into Ncat? Does the script need to read from its own
> stdin in order to allow that? I think the idea is confused.
>
> David Fifield

The whole point of -L was to allow running scripts with #!/usr/bin/ncat -L 
shebang. The interpreter mode would be a "bonus" and it's definitely not my 
priority. You raised an interesting point there though - that it should be 
carefully thought over what do my Ncat-Lua functionality do with stdin.

Jacek Wielemborek
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/