Re: Ideas for GSoC 2010

[Joao Correa <joao () livewire com br>]

Thanks for the comment Ron,

Funny that you have mentioned this email address harvesting script. In
2009's GSoC I was thinking about developing this script, but due to
priority changes and the lack of spidering support, we decided to go
on with other scripts.

I'm certain that we can write it with that as soon as spidering
support is ready!

On Mon, Apr 5, 2010 at 11:20 PM, Ron <ron () skullsecurity net> wrote:
> Hey Joao,
>
> On the topic of HTTP spidering, another script I'd use a lot is one that could harvest email addresses from a site 
> (while spidering). In fact, those email addresses could be saved and used for bruteforcing in other scripts -- I 
> already do that, somewhat manually.
>
> Ron
>
> On Mon, 5 Apr 2010 23:09:32 -0300 Joao Correa <joao () livewire com br>
> wrote:
> > Hi Guys,
> >
> > I'm considering applying again as a student on 2010 GSoC with nmap.
> > Last year I had a great experience with NSE, so I'm thinking about
> > applying for the NSE script developer idea.
> >
> > So far, I had some ideas that can be developed during the summer, but
> > I'm sure that I can use some ideas and discussion with you. Part of
> > them were taken from the TODO list.
> >
> > I would like to thank any opinion in advance.
> >
> > Here follows some of my ideas:
> >
> > o HTTP Spidering feature:
> >
> > The HTTP Spidering feature was recently mentioned by Ron on nmap-dev
> > as "desperately needed". It is known that this feature would allow
> > efficient development of many interesting scripts. Patrick have worked
> > a lot on it for one year now, but it is not ready yet. Some issues,
> > such as results storage, availability and bandwidth consumption (as
> > mentioned on TODO list), need to figured out and implemented.
> > My first objective is to finish writing the HTTP Spidering feature,
> > making it reliable and available.
> >
> > o HTTP Related scripts:
> >
> > With the Spidering feature working, the second objective is to develop
> > a few scripts that might make good use of it.
> >
> > * Write a high-speed brute force script for HTTP authentication
> > (mentioned in TODO file).
> > This script might use NSE multiple thread parallelism features (also
> > mentioned in TODO).
> >
> > * Write a script to map all the asynchronous requests (XMLHttpRequest
> > and similar resources) made by a file, allowing API discovery for web
> > 2.0 applications. This script should output every page called by these
> > requests, together with the parameters used. It will allow pentesters
> > to map a web application API, revealing points that can be target of
> > exploitation.
> >
> > * Write a script to parse javascript inside HTML. This script is going
> > to check every javascript function and external file included,
> > outputting function signatures. Some verbose parameters may be
> > considered, because sometimes outputs can get very large.
> > This script will help pentesters to map javascript functions. This
> > would help them to search for and understand asynchronous requests
> > API, form validation mechanisms and everything else on a web
> > application that might be relying on scripts.
> > Output example:
> > | External file: /scripts/jslib.js
> > | - function foo()
> > | - function bar(foovar)
> > |
> > | Internal: index.html
> > | - function checkInput(input)
> >
> > * Write a script to search for password fields in HTML forms. This
> > script can also be extended to, with proper parameters, display other
> > types of fields (maybe all), including hidden ones.
> > This script will allow mapping forms that might be target of brute
> > forces or more sophisticated sql injections.
> >
> > o Other Tasks:
> >
> > Doing the above mentioned tasks might not take all Summer. For this
> > reason I'm thinking about a few more features that I can handle during
> > GSoC. Time might not be enough to accomplish everything below, but,
> > accordingly to mentor's decision, I can go on with what is more
> > important.
> >
> > o Work on random NSE related issues from nmap-dev
> >
> > Last year I had the opportunity to, during GSoC, help fixing bugs and
> > solving problems that were not directly related to the scripts I was
> > writing. Once again I expect to be available and help with random NSE
> > related issues.
> >
> > o Support routing http requests through proxies (mentioned in TODO
> > file).
> >
> > In 2009's GSoC I've written proxy.lua lib
> > (http://nmap.org/nsedoc/lib/proxy.html) to be used with the open proxy
> > detection scripts. I believe that a big part of this code can be
> > reused by http.lua to open and use proxy connections, since it already
> > implements the whole handshake.
> > Of course that part of the proxy.lua code will need some work, mainly
> > to fit the needs that may arise while editing http.lua.
> >
> > o Web application fingerprinting script (mentioned in TODO file).
> >
> > This is well described in TODO.
> >
> > o Secondary script ideas.
> >
> > I also have a few more ideas for scripts. These ideas need more
> > brainstorming to figure out the best way to implement.
> > - Script to brute force forms with password fields: A problem would be
> > recognizing correct/wrong password submitted.
> > - Improve SQL Injection script with spidering: This script uses a very
> > simple spidering engine to walk through and parse the web pages. I
> > believe that this can (and certainly will) be reworked for better
> > performance after spidering support is complete.
> > - File inclusion path traversal: Check urls for arguments with
> > filenames, if one is found, try to use path traversal to include a
> > different file. One problem here is how to identify filenames on urls
> > (regular expressions considering extensions?)
> >
> > o IDS Detection Feature
> >
> > While discussing with David about IDS detection scripts
> > (http://seclists.org/nmap-dev/2010/q1/814), I had the idea of writing
> > a feature instead of a script. The idea is basically checking for
> > scripts that started running (portrule was true), but that, during
> > execution, started receiving timeouts. If this happens, and the port
> > is no longer open (changed to closed or filtered), than the script
> > probably triggered an IDS rule.
> > Performance issues might be considered for this feature. Maybe it
> > should not be a default feature, or should only be default to "vuln",
> > "intrusive", "exploit" and related script categories.
> >
> > Thanks again,
> > João.
> > _______________________________________________
> > Sent through the nmap-dev mailing list
> > http://cgi.insecure.org/mailman/listinfo/nmap-dev
> > Archived at http://seclists.org/nmap-dev/
>
>
> --
> Ron Bowes
> http://www.skullsecurity.org
> http://www.twitter.com/iagox86
>
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/