Nmap Development mailing list archives
Re: Ncat + Lua - GSOC feedback request
From: Jacek Wielemborek <wielemborekj1 () gmail com>
Date: Sun, 12 May 2013 17:51:27 +0200
NOTE: this post is over three A4 pages long, so in order to make it easier for the eyes I prepared a printer-friendly PDF/ODT version available here: http://kolos.math.uni.lodz.pl/~d33tah/gsoc/ncat_lua/ Hi, I'm writing this e-mail to let you guys know that I'm still interested in the Google Summer of Code and I'm not waiting idly for the proposal selection results. During the last week I had spent a lot of time collecting and reading as many resources on Nmap, Ncat, Lua and NSE as I could possibly find. Among these, I had found a few talks of David and Fyodor from FOSDEM and DEFCON, read Fyodor's “Nmap Security Scanning” book and “Programming in Lua” as well. I also played with Nmap's code a bit. I read quite a lot of it while doing a bit of research on how much work would be needed to port it for my Sony PSP (just for fun; AFAIK it only has application-level networking interface anyway, so I'd have to drop a lot of features). In the meantime, I thought of two ideas on how to improve the Nmap usability; I'll post them on nmap-dev once I'll polish them a bit. I also gave a lot of thinking to my proposal and decided to ask you for comments on my idea to extend Ncat with an NSE-like API that will make NSE script programming and testing even easier. I talked with bonsaiviking via IRC and he pointed out that my proposal lacks use cases for my project. So here's the update to my proposal: 1. The way I see it – “the now” and the deliverables. (feel free to skip over the first three paragraphs if you don't feel like reading my marketing insights about the project) Ncat is IMHO a hacker's must-have, the tools from netcat family have the reputation of being the „swiss-army knife” (pardon the Wikipedia quote, but I agree with it) for being both very tiny and offering a pretty useful functionality. Generally, it's used for making outbound TCP/UDP/STCP connections as well as opening ports using these protocols (with presentation-level support for SSL). Combined with Unix piping capabilities it gives a lot of space for creative use – you can basically turn any CLI application into an online version of it. You can wrap anything with the SSL support without putting any effort into it. You can set up a bridge between different hosts and protocols. And you can do a lot more. By releasing Ncat under Nmap's umbrella, it has a guaranteed large userbase. Its additional advantages include: SOCKS support, connection brokering and STCP support. With so many features, here's where Lua comes into play. Lua is the mother tongue of NSE. At the very least it means that there would be quite a lot of Lua programmers which already had something to do with Lua socket programming – I mean the ones who have experience writing NSE scripts. I believe, though, that there's much more to it – I am thinking of adding NSE support for Ncat (see the questions at the end of this post). If done properly, this allows the NSE hackers to make use of the libraries from the already existing Lua codebase for their own needs, taking advantage of the enormous variety of protocols that are already supported (and the ones that will be in the future!). This exchange of code works both ways. The Nmap devs will also be able to test their NSE scripts more easily, writing both client scripts and – if they would like to – tiny proof-of-concept servers for unit testing of their projects (see the next section). While considering the project, I wasn't at first completely convinced if it wouldn't be easier to do the scripting in another language with extensive socket programming support in the standard library, like Ruby or Python. Though, as I also mentioned below, Ncat+Lua will have the advantage of fitting on an embedded system while providing full server/client functionality with SSL support. I haven't heard of any such environment so far – and I do believe it's worth making it happen. 2. Ideas for the use cases So, we've got this well-documented shiny tool that speaks Lua now. Apart from all the joy of starting a script with #!/usr/bin/ncat -L header line, what else is there? I already mentioned: 1) NSE protocol libraries being used for more personal needs and 2) better NSE testing facilities. There's more to it. Ncat with its tiny resource requirements fits nicely on embedded devices. When you combine the tool that can open sockets with an extensive scripting language support, you get networking power that no other scripting language that I know of can boast about. And there's another upside – chat functionality could finally be removed from the C code. And moved to the Lua code. And extended more easily, if anyone wishes to. I read somewhere the complains about being unable to change nicknames in Ncat. Imagine size of a C patch needed to add the support for that. Now compare it to the size of the Lua implementation of this feature. Doesn't it kind of suggest where all the logic should go? 3. Hard requirements * Quite a lot of planning I definitely consider it a requirement number one for this task. I don't really believe that it's possible to integrate Lua with Ncat properly without prior planning, especially if we're heading for NSE compatibility. The planning phase is when I expect to need my mentor the most, so that the code would be written well and once, with no further rewriting ever needed. * Low-level work Before I wrote the tiny “proof of concept” code I published on Github, I tried to write it as a stand-alone application that makes use of nsock... and I noticed that it's easy to make mistakes that forced me to use strace and a debugger extensively. Taking into account that nsock isn't yet used for ncat listening mode and that it doesn't really have a lot of documentation... well, I wouldn't be surprised to be “forced” to some low-level networking work to make the Lua scripting functionality done. I feel prepared for the challenge though. * Documentation Definitely a lot of work needs to be put into documenting the interfaces and I strongly stressed in my proposal schedule that I'm planning to spend at least one, probably two weeks spend entirely on preparing the documentation that is user-friendly. I'm planning to keep writing the manual systematically to avoid getting into the trap of zipping it all up at the deadline. As “documentation” I also mean the code comments and examples/test cases that illustrate the typical usage of the feature. 4. Open questions How much compatibility with NSE is possible for Ncat without getting too bloated? I kept mentioning NSE in this document quite a few times, but I have to admit I hadn't yet done all the research needed to estimate how much work would need to be put to make Ncat a testing platform for Nmap Lua scripts. As I said in my proposal, I can work hard to make it as shiny as possible, but it might be risky to bloat the Ncat codebase too much if we care about the „lightweight” reputation. If I made the build process configurable (and I find it critical at the moment), it would hopefully not harm the size of the resulting binary. For the amount of work, I would love to get a bit of realistic feedback on whether you think that one person can integrate NSE nicely into Ncat in three months of work. I guess you can already tell that I'm very motivated though. Even if whole NSE infrastructure wouldn't fit into the project all at once, I already outlined the deliverables that can be achieved without full success on it – there's the unit testing (for NSE programmers and not only), there's the fast coding of servers on embedded devices. And there's a lot more of potential. 5. Another option – some thoughts on Ncat+Lua without NSE support In this section I'll extend the original idea from my proposal, where I was talking about adding Lua support to Ncat without speaking of NSE (at least not as a priority). First I'd like to give an update on my Ncat fork you can find on Github. By now my „proof of concept” should easily compile and work with the bundled liblua. I left a few hacks, documented them all on my wiki (https://github.com/d33tah/nmap/wiki/Hacks-and-bugs). There were a few design problems that I had come across. For example – how should the scripts be invoked and interfaced with the original Ncat functionality? At first, I had the idea of adding –lua-file command line switch that should be used along with the old-style connection destination specification like this: ncat localhost 2233 --lua-file hello.c Once run, the Ncat would connect to the destination in a traditional manner and then run all the hooks defined in the lua file – for example, if the file has “on_connect” function defined, it would be run once the connection to the remote host has been established. This approach had the advantage of being fairly easy to implement – it took me just a few hours to get the proof of concept code working. On the other hand, it felt ugly – even if I changed --lua-file switch to -L, it still wasn't too comfortable for everyday use; I believe that every feature needs a package and if the package isn't cool enough, the feature won't be used. After a bit of thinking, I thought it would be cool to add an -L switch that would read the lua file from standard input and run it once it hits an EOF. That would make it easy to write powerful scripts that, for example, establish more than one connection in the same Ncat instance. The hacky way to implement it would be to introduce two new functions, connect() and listen() that expect a port number (a host for the connect() as well) and a dictionary containing Ncat options that would override the defaults – such as the protocol and callbacks, like this: #!/usr/bin/ncat -L options = {} options[“protocol”] = “UDP” options[“on_connect”] = function() end connect(„localhost”, 2233, options) That introduces a few interesting issues – we would need a separate copy of struct options for each socket for example. I believe it would be the most fun way to do it though. For the testing purposes, I'd also add another way to turn on “lua stdin” mode – by detecting the case when the argv[0] is ncat-lua, which would be an alias to ncat executable. That would make it easier in the testing stage – since Linux shebangs don't accept more than one parameter, adding the current directory to PATH would let me run the scripts by writing a header that contains “#!/usr/bin/env ncat-lua”. 6. Summary That's pretty much it. I hope I convinced you that introducing Ncat into Lua will be an exciting new feature that makes an interesting Google Summer of Code project. I'd love to find out what do you think of my idea. I'm looking forward to hearing some feedback from you guys :). Yours, Jacek “d33tah” Wielemborek 2013/5/2 Arturo 'Buanzo' Busleiman <buanzo () buanzo com ar>:
My two cents: no swig please :D Cheers, Buanzo. On May 1, 2013 2:44 PM, "David Fifield" <david () bamsoftware com> wrote:On Wed, May 01, 2013 at 06:44:49PM +0200, Jacek Wielemborek wrote:Today I finished reading through the lists of organisations and projects that interested me and decided that Ncat is going to be my top priority for this year's GSOC's application. The proposal I simply fell in love with is "Bringing lua to ncat" which can be found here: https://secwiki.org/w/GSoC_community_ideas#Bringing_lua_to_ncat I'd like to ask if there is anyone else that already expressed interest in this task (hopefully not) so I'd know if I can keep hacking some code for my application. Also, what do you think of SWIG instead of Lua?As far as I know, noone has expressed interest in the project so far. SWIG instead of Lua is probably not an option. We already package and ship Lua and people know how to use it from NSE. David Fifield _______________________________________________ Sent through the dev mailing list http://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
_______________________________________________ Sent through the dev mailing list http://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Ncat + Lua - GSOC feedback request Jacek Wielemborek (May 01)
- Re: Ncat + Lua - GSOC feedback request David Fifield (May 01)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (May 02)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (May 02)
- Re: Ncat + Lua - GSOC feedback request Arturo 'Buanzo' Busleiman (May 02)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (May 12)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (Jun 13)
- Re: Ncat + Lua - GSOC feedback request David Fifield (Jun 14)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (Jun 16)
- Re: Ncat + Lua - GSOC feedback request Henri Doreau (Jun 16)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (Jun 17)
- Re: Ncat + Lua - GSOC feedback request David Fifield (Jun 17)
- Re: Ncat + Lua - GSOC feedback request Daniel Miller (Jun 17)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (Jun 17)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (Jun 17)
- Re: Ncat + Lua - GSOC feedback request Jacek Wielemborek (May 02)
- Re: Ncat + Lua - GSOC feedback request David Fifield (May 01)