Re: Ncat + Lua - GSOC feedback request

[Jacek Wielemborek <wielemborekj1 () gmail com>]

NOTE: this post is over three A4 pages long, so in order to make it
easier for the eyes I prepared a printer-friendly PDF/ODT version
available here:


http://kolos.math.uni.lodz.pl/~d33tah/gsoc/ncat_lua/

Hi,


I'm writing this e-mail to let you guys know that I'm still interested
in the Google Summer of Code and I'm not waiting idly for the proposal
selection results. During the last week I had spent a lot of time
collecting and reading as many resources on Nmap, Ncat, Lua and NSE as
I could possibly find. Among these, I had found a few talks of David
and Fyodor from FOSDEM and DEFCON, read Fyodor's “Nmap Security
Scanning” book and “Programming in Lua” as well.


I also played with Nmap's code a bit. I read quite a lot of it while
doing a bit of research on how much work would be needed to port it
for my Sony PSP (just for fun; AFAIK it only has application-level
networking interface anyway, so I'd have to drop a lot of features).
In the meantime, I thought of two ideas on how to improve the Nmap
usability; I'll post them on nmap-dev once I'll polish them a bit.


I also gave a lot of thinking to my proposal and decided to ask you
for comments on my idea to extend Ncat with an NSE-like API that will
make NSE script programming and testing even easier. I talked with
bonsaiviking via IRC and he pointed out that my proposal lacks use
cases for my project. So here's the update to my proposal:


1. The way I see it – “the now” and the deliverables.


(feel free to skip over the first three paragraphs if you don't feel
like reading my marketing insights about the project)


Ncat is IMHO a hacker's must-have, the tools from netcat family have
the reputation of being the „swiss-army knife” (pardon the Wikipedia
quote, but I agree with it) for being both very tiny and offering a
pretty useful functionality. Generally, it's used for making outbound
TCP/UDP/STCP connections as well as opening ports using these
protocols (with presentation-level support for SSL).


Combined with Unix piping capabilities it gives a lot of space for
creative use – you can basically turn any CLI application into an
online version of it. You can wrap anything with the SSL support
without putting any effort into it. You can set up a bridge between
different hosts and protocols. And you can do a lot more.


By releasing Ncat under Nmap's umbrella, it has a guaranteed large
userbase. Its additional advantages include: SOCKS support, connection
brokering and STCP support. With so many features, here's where Lua
comes into play.


Lua is the mother tongue of NSE. At the very least it means that there
would be quite a lot of Lua programmers which already had something to
do with Lua socket programming – I mean the ones who have experience
writing NSE scripts. I believe, though, that there's much more to it –
I am thinking of adding NSE support for Ncat (see the questions at the
end of this post). If done properly, this allows the NSE hackers to
make use of the libraries from the already existing Lua codebase for
their own needs, taking advantage of the enormous variety of protocols
that are already supported (and the ones that will be in the future!).


This exchange of code works both ways. The Nmap devs will also be able
to test their NSE scripts more easily, writing both client scripts and
– if they would like to – tiny proof-of-concept servers for unit
testing of their projects (see the next section).


While considering the project, I wasn't at first completely convinced
if it wouldn't be easier to do the scripting in another language with
extensive socket programming support in the standard library, like
Ruby or Python. Though, as I also mentioned below, Ncat+Lua will have
the advantage of fitting on an embedded system while providing full
server/client functionality with SSL support. I haven't heard of any
such environment so far – and I do believe it's worth making it
happen.


2. Ideas for the use cases


So, we've got this well-documented shiny tool that speaks Lua now.
Apart from all the joy of starting a script with #!/usr/bin/ncat -L
header line, what else is there? I already mentioned: 1) NSE protocol
libraries being used for more personal needs and 2) better NSE testing
facilities.


There's more to it. Ncat with its tiny resource requirements fits
nicely on embedded devices. When you combine the tool that can open
sockets with an extensive scripting language support, you get
networking power that no other scripting language that I know of can
boast about.


And there's another upside – chat functionality could finally be
removed from the C code. And moved to the Lua code. And extended more
easily, if anyone wishes to. I read somewhere the complains about
being unable to change nicknames in Ncat. Imagine size of a C patch
needed to add the support for that. Now compare it to the size of the
Lua implementation of this feature. Doesn't it kind of suggest where
all the logic should go?


3. Hard requirements


* Quite a lot of planning


I definitely consider it a requirement number one for this task. I
don't really believe that it's possible to integrate Lua with Ncat
properly without prior planning, especially if we're heading for NSE
compatibility. The planning phase is when I expect to need my mentor
the most, so that the code would be written well and once, with no
further rewriting ever needed.


* Low-level work


Before I wrote the tiny “proof of concept” code I published on Github,
I tried to write it as a stand-alone application that makes use of
nsock... and I noticed that it's easy to make mistakes that forced me
to use strace and a debugger extensively. Taking into account that
nsock isn't yet used for ncat listening mode and that it doesn't
really have a lot of documentation... well, I wouldn't be surprised to
be “forced” to some low-level networking work to make the Lua
scripting functionality done. I feel prepared for the challenge
though.

* Documentation


Definitely a lot of work needs to be put into documenting the
interfaces and I strongly stressed in my proposal schedule that I'm
planning to spend at least one, probably two weeks spend entirely on
preparing the documentation that is user-friendly. I'm planning to
keep writing the manual systematically to avoid getting into the trap
of zipping it all up at the deadline. As “documentation” I also mean
the code comments and examples/test cases that illustrate the typical
usage of the feature.


4. Open questions



How much compatibility with NSE is possible for Ncat without getting
too bloated?


I kept mentioning NSE in this document quite a few times, but I have
to admit I hadn't yet done all the research needed to estimate how
much work would need to be put to make Ncat a testing platform for
Nmap Lua scripts. As I said in my proposal, I can work hard to make it
as shiny as possible, but it might be risky to bloat the Ncat codebase
too much if we care about the „lightweight” reputation. If I made the
build process configurable (and I find it critical at the moment), it
would hopefully not harm the size of the resulting binary.


For the amount of work, I would love to get a bit of realistic
feedback on whether you think that one person can integrate NSE nicely
into Ncat in three months of work. I guess you can already tell that
I'm very motivated though. Even if whole NSE infrastructure wouldn't
fit into the project all at once, I already outlined the deliverables
that can be achieved without full success on it – there's the unit
testing (for NSE programmers and not only), there's the fast coding of
servers on embedded devices. And there's a lot more of potential.


5. Another option – some thoughts on Ncat+Lua without NSE support


In this section I'll extend the original idea from my proposal, where
I was talking about adding Lua support to Ncat without speaking of NSE
(at least not as a priority). First I'd like to give an update on my
Ncat fork you can find on Github. By now my „proof of concept” should
easily compile and work with the bundled liblua. I left a few hacks,
documented them all on my wiki
(https://github.com/d33tah/nmap/wiki/Hacks-and-bugs).


There were a few design problems that I had come across. For example –
how should the scripts be invoked and interfaced with the original
Ncat functionality? At first, I had the idea of adding –lua-file
command line switch that should be used along with the old-style
connection destination specification like this:


ncat localhost 2233 --lua-file hello.c


Once run, the Ncat would connect to the destination in a traditional
manner and then run all the hooks defined in the lua file – for
example, if the file has “on_connect” function defined, it would be
run once the connection to the remote host has been established.


This approach had the advantage of being fairly easy to implement – it
took me just a few hours to get the proof of concept code working. On
the other hand, it felt ugly – even if I changed --lua-file switch to
-L, it still wasn't too comfortable for everyday use; I believe that
every feature needs a package and if the package isn't cool enough,
the feature won't be used.


After a bit of thinking, I thought it would be cool to add an -L
switch that would read the lua file from standard input and run it
once it hits an EOF. That would make it easy to write powerful scripts
that, for example, establish more than one connection in the same Ncat
instance. The hacky way to implement it would be to introduce two new
functions, connect() and listen() that expect a port number (a host
for the connect() as well) and a dictionary containing Ncat options
that would override the defaults – such as the protocol and callbacks,
like this:


#!/usr/bin/ncat -L
options = {}
options[“protocol”] = “UDP”
options[“on_connect”] = function() end
connect(„localhost”, 2233, options)


That introduces a few interesting issues – we would need a separate
copy of struct options for each socket for example. I believe it would
be the most fun way to do it though. For the testing purposes, I'd
also add another way to turn on “lua stdin” mode – by detecting the
case when the argv[0] is ncat-lua, which would be an alias to ncat
executable. That would make it easier in the testing stage – since
Linux shebangs don't accept more than one parameter, adding the
current directory to PATH would let me run the scripts by writing a
header that contains “#!/usr/bin/env ncat-lua”.


6. Summary


That's pretty much it. I hope I convinced you that introducing Ncat
into Lua will be an exciting new feature that makes an interesting
Google Summer of Code project. I'd love to find out what do you think
of my idea. I'm looking forward to hearing some feedback from you guys
:).


Yours,

Jacek “d33tah” Wielemborek


2013/5/2 Arturo 'Buanzo' Busleiman <buanzo () buanzo com ar>:
> My two cents: no swig please :D
>
> Cheers,
> Buanzo.
>
> On May 1, 2013 2:44 PM, "David Fifield" <david () bamsoftware com> wrote:
> > On Wed, May 01, 2013 at 06:44:49PM +0200, Jacek Wielemborek wrote:
> > > Today I finished reading through the lists of organisations and
> > > projects that interested me and decided that Ncat is going to be my
> > > top priority for this year's GSOC's application.
> > >
> > > The proposal I simply fell in love with is "Bringing lua to ncat"
> > > which can be found here:
> > >
> > > https://secwiki.org/w/GSoC_community_ideas#Bringing_lua_to_ncat
> > >
> > > I'd like to ask if there is anyone else that already expressed
> > > interest in this task (hopefully not) so I'd know if I can keep
> > > hacking some code for my application. Also, what do you think of SWIG
> > > instead of Lua?
> >
> > As far as I know, noone has expressed interest in the project so far.
> >
> > SWIG instead of Lua is probably not an option. We already package and
> > ship Lua and people know how to use it from NSE.
> >
> > David Fifield
> > _______________________________________________
> > Sent through the dev mailing list
> > http://nmap.org/mailman/listinfo/dev
> > Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/