Re: Built-in authentication for http.lua

[David Fifield <david () bamsoftware com>]

On Sat, Jul 31, 2010 at 10:42:14AM +0200, Patrik Karlsson wrote:
> I implemented a quick brute script to test the new authentication feature and found two problems.
> One is related to caching and the other to calling the http lua with multiple worker threads.
>
> As my brute framework uses multiple worker threads to perform brute-forcing I noticed the following
> error popping up at times:
>
> ./nselib/http.lua:874: do not have a lock on this mutex
> stack traceback:
>       [C]: in function 'mutex'
>       ./nselib/http.lua:874: in function 'insert_cache'
>       ./nselib/http.lua:1040: in function 'get'
>       ./scripts/http-brute.nse:62: in function 'login'
>       ./nselib/brute.lua:405: in function 'doAuthenticate'
>       ./nselib/brute.lua:442: in function 'main'
>       ./nse_main.lua:584: in function <./nse_main.lua:584>
>
> The error may be in how I'm calling the http lib from my code, I don't know. 
> Anyway, when running with a single thread it works great. In order to test this change the line       
> engine:setMaxThreads(1) in http-brute to 10.

I see this too. It seems to only happen at the end of the script. If I
set unpwdb.timelimit=10s, it happens after 10 seconds, and with
unpwdb.timelimit=20s, it happens after 20.

I did some looking into this. The problem is that the http cache mutex
is being freed twice. First by mutex_aux_done, which is defined in
nse_nmaplib.cc and is "called when a thread ends but failed to
unlock the mutex." It is then freed again in insert_cache by the http
library.

Patrick, do you have an idea why this is happening? I attached the patch
that I used to diagnose it. The telltale parts are

NSE: Finished http-brute against 192.168.0.1:80.
mutex done aux_mutex_done
0x90c4990 mutex("done")
mutex done insert_cache
0x90c3b20 mutex("done")
NSE: Finished http-brute against 192.168.0.1:80.
mutex done aux_mutex_done
0x90c25a8 mutex("done")
mutex done insert_cache
0x90c4990 mutex("done")
NSE: http-brute against 192.168.0.1:80 threw an error!
./nselib/http.lua:877: 0x90c4990 do not have a lock on this mutex

To test this, it's reasonably easy to set up thttpd locally to require
authentication, without installing anything.

wget http://www.acme.com/software/thttpd/thttpd-2.25b.tar.gz
tar xzvf thttpd-2.25b.tar.gz
cd thttpd-2.25b
perl -p -i -e 's/getline/_getline/g' extras/htpasswd.c
./configure && make
./extras/htpasswd -c .htpasswd test
sudo ./thttpd -D

Then you can reproduce this error by running

./nmap --datadir . localhost -p 80 -Pn -n --script http-brute -d --script-args http-brute.path=/,unpwdb.timelimit=5s

> The cache problem occurs when a page is protected by authentication.
> As the cached result is not tied to the authenticated user any subsequent calls, regardless if they were 
> authenticated or not will return the result.
> I don't know if this is a big problem for anything else, but when testing authentication it sure is, as all login 
> attempts after the first successful will appear as correct.
> I worked around this in my script by issuing the no_cache option when calling the http.get method.

I hadn't thought of this. I had to make it so that 401 replies would not
be cached, otherwise even the correct credentials would get the cached
401. I don't know what the right thing to do here is. Caching the
authenticated response might be what someone wants. I think your idea of
using no_cache is the best.

David Fifield

Attachment: mutex-done-debug.diff
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/