Nmap Development mailing list archives
Ncat-Lua: a complete status report for development and discussion
From: d33 tah <d33tah () gmail com>
Date: Mon, 8 Jul 2013 19:10:38 +0200
Hi guys, TL;DR: This e-mail is about the Ncat-Lua GSoC project. If you care about its development, jump to any section you find interesting (or, most preferably, just read the whole message) and comment on it in an appropriate mailing list thread. If you don't want to bother looking for the thread (it might not be there), e-mail me and I'll read the message and forward it where it belongs. I really need some feedback, so... thanks in advance! The long version: This e-mail is an update on my Ncat-Lua project, for the ones interested in the development of Lua scripting support for Ncat. I'll try to explain here what the new Ncat can do and what it still can't. Should you find any part of this e-mail worth commenting on, don't hesitate to! Implementing new Ncat Lua features without feedback or use cases feels quite like walking in the dark, which is why I would definitely appreciate your comment. The current Nmap's trunk includes the Ncat version with --lua-exec <filename> feature merged in. For those who hear about it for the first time, it's basically an equivalent of --sh-exec "lua <filename>" - the script given as the parameter will be ran with its standard input and standard output redirected to the connection. It proved to be enough to implement some simple demo services like ECHO, DAYTIME or DISCARD. It can do much more though - as I showed in ncat/scripts/conditional.lua, you can write a bit more complex service as well, making decisions based on what user types in. This approach has a few shortcomings though. While playing with --lua-exec's capabilities, I also figured out a way to print to the Ncat's output streams. The trick is to use the io.stderr from inside of the Lua script, which isn't modified in the forking process, so it still effects in printing the message to the terminal - this works in Windows as well. It's cool, but it pretty much calls for the access to Ncat's standard input as well. **ncat-lua-script branch** Since I already had quite a few use cases/user stories to justify the feature, I set out to experiment with an implementation prototype for reading Ncat's STDIN in ncat-lua-script branch. I added this feature - at the moment to *nix systems only - by creating another pipe between the parent and child process and adding it as "io.ncatin" to the --lua-exec scripts. This is only available if you specify --lua-extensions in the command-line. You can see some new shiny demos there - I hacked up a demo that can connect to echo.websocket.org via a stub of Websocket protocol, there are ROT13 and Base64 filtering demos as well. I also added an active filter example - telnet.lua, which transparently responds to the Telnet's control commands and cuts them out of the output (which is an improvement over the -t command-line switch in "vanilla" Ncat). There's also a shiny IRC demo that chooses a random nickname and joins the #bots channel, allowing you to speak there (tested on Freenode)! While implementing these, I quickly started to need some kind of select() to watch over io.stdin and io.ncatin. I implemented this as a function that takes a table of file instances and removes the values that don't have any data to read. This dramatically cut the CPU footprint of the scripts and allowed to make the IRC demo (and not only) feel real-time. Unfortunately, it also led to the question of how should this be implemented (see the "Open issues" section). Oh, by the way, thanks to restoring the default SIGPIPE handler inside of the child process, I could finally implement the CHARGEN demo - the scripts will simply die when the connection closes. **Open issues:** Below is the current status of the discussion on Ncat-Lua. It's my attempt to summarize the issues the project has and point to the relevant threads on this mailing list. If you'd like to speak up on any of these topics and I hadn't mentioned a link to a seclists thread here, please consider starting a new one. This way we could avoid the mess that started to be forming here: http://seclists.org/nmap-dev/2013/q2/276 The following list will include some features that are lacking use cases / user stories. Implementing them without these is pretty much pointless, because it's hard to figure out a good design to something that isn't well specified. Thus the question I'd like to ask you - would you find any of these features useful? Or maybe do you think the opposite way - that a particular feature is completely redundant? Those would be very valuable signals for me. * Stdlib While looking for a good way to implement io.select(), Patrick suggested me to mimic the interfaces from Luasocket. I actually stumbled upon the very same problem a few times - websockets have their own implementation in Lua as well (https://github.com/lipp/lua-websockets/), same goes for the eof function I removed in r31245. I'm starting to think that instead of reinventing the wheel, it might be a good idea to copy/fork some useful libraries and bundle them with Ncat. Or make Ncat work with Luarocks/Luadist package management systems. * Interfacing with Ncat - o.target, reacting to SIGCHLD I believe that Lua scripts would benefit from a bit more interfacing with Ncat. For example, the Websocket script could be interested in the particular hostname that "o.target" Ncat variable contains so that it would handshake with a valid domain. The port would probably be useful as well, not to mention the information on whether the SSL is enabled or not. There's quite a lot of things that could be added, but the question is - which of them do you find actually useful, as a future Ncat script developer? * State sharing, server facilities Although the current implementation already lets you code up pretty sophisticated service servers, you can't for example create even a simple chat server that keeps track of its users in order to broadcast them messages. I think it could also be useful to be able to decide on whether to accept the connection or not. * Parameter passing and invocation --lua-extensions is a playground feature and some of its sub-features might be dropped or renamed. It's obvious though that “ncat --lua-extensions --lua-exec scripts/ws.lua echo.websockets.org 80” is ugly and has to be replaced with something like “ncat --with ws echo.websockets.org 80”. Discussion on this topic can be found here: http://seclists.org/nmap-dev/2013/q2/501 and http://seclists.org/nmap-dev/2013/q2/461 . * Stacking Websockets, ROT13/Base64, Telnet filters... at the moment they limit the user to using one particular script. It would definitely be useful though to be able to stack scripts, so that you could for example encode your session with Base64 while sending it over Websockets. * Making new connections httplookup, one of the use cases on the secwiki page I created (https://secwiki.org/w/Ncat-Lua) seems to imply that Ncat's scripting engine is capable of creating another connections. It's yet to be discussed if such functionality would actually be used. This is the current description of the use case - comments are of course welcome:
I want a service I can connect to and would let me perform HTTP queries the same way nslookup lets me peform DNS ones. After connecting, I will be greeted with '> ' command prompt. I first start by setting the remote host using the command "server nmap.org" and send the newline character. Then I'd like to type in the first line of the request (without the HTTP version), send the newline and get the reply headers, like in this example session:server nmap.org GET /HTTP/1.1 200 OK Date: Fri, 28 Jun 2013 20:28:42 GMT Server: Apache/2.2.15 (CentOS) Accept-Ranges: bytes Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8
* Read-write loop layout The read-write loop in Ncat filter scripts (the ones using io.ncatin) is starting to follow a quite clear pattern. At the beginning of the script there is some initialization phase code, with the definitions of functions and variables that will be used later. Then there's an infinite loop, in which the script performs a select on both io.ncatin and io.stdin and then performs actions depending on the contents of the array returned by select(). As utility functions, I kept adding table.find so often that I created a special library for it (ncatlib.lua). Prior to my work started in r31287, base64.lua and rot13.lua shared so much code that I finally thought it might be useful to create run_filter, a function that takes “encode” and “decode” functions, reads lines from the socket and ncatin in an infinite loop and parses the input it with the functions from the parameters. It's not the perfect solution though - I can't wrap telnet.lua with the run_filter as it looks now. Despite that, I believe it might be the good direction. ** Conclusion ** So, well, that's it. As you can see, there are some awesome features on the horizon, but in order to reach that stage, we need more use cases and discussion. That's why - again - feel free to comment on anything you find interesting here, any feedback is welcome. Thanks! Yours, Jacek Wielemborek _______________________________________________ Sent through the dev mailing list http://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Ncat-Lua: a complete status report for development and discussion d33 tah (Jul 08)