Re: [NSE] apache-userdir-enum

[jah <jah () zadkiel plus com>]

On 27/07/2009 22:56, David Fifield wrote:
> I like this script. It's a good idea. Could it be combined with the
> recently added http-enum.nse script? I like the idea of checking the
> version detection results and only continuing if it matches "apache".
>   
I envisioned the script being used for discovering usernames as a
precursor to a brute-force attempt over the same or other protocols
(e.g. telnet, ssh) and to better focus such an attempt on usernames that
are highly likely to exist on the target.  Key to this would be limiting
the number of false positives.
I think that there would be distinct reasons for running the http-enum
and apache-userdir scripts - one to find directories to explore and one
to find usernames - and in the latter case, it might be overkill to test
for all of the directories in http-enum (especially those not usually
found on apache servers).
If running the apache-userdir script against a wider range of services
doesn't result in (more) false positives then it doesn't sound
unreasonable to combine the scripts - but are there any advantages in
doing so and would they outweigh the advantage of being able to run the
scripts separately?
> http-enum uses HEAD when possible. It also does a false positive check
> using "/Nmap404Check", but I think the random one in apache-userdir-enum
> is better.
The reason I didn't go for HEAD requests is mainly due to some server
configurations which result in different responses for HEAD and GET
requests for the same resource (without regard for the HTTP standard). 
It would be possible to test whether a server responds incorrectly for a
HEAD request for a single resource and then to make an assumption about
how it will handle requests for other resources, but I think it's safer
(and less hassle) not to bother.  There also seems to be very little
difference in speed when making HEADs compared to GETs - I've only done
limited testing in this regard, but what I have done has resulted in
less than 1% difference and not always in favour of HEADs.

On 27/07/2009 23:18, Fyodor wrote:
> Thanks Jah!  I'm wondering how server-specific this ~username behavior
> is?  In particular, I'm wondering how many other web servers have
> copies that approach.  It would be fascinating to do a big scan
> (without the "apache" check) and determine:
>
> 1) How many of the servers have one of the 11 tested ~usernames
> 2) How many of those pass the "apache" check.
>
> For example, you note in your comments that Tomcat Coyote apparently
> exhibits this behavior sometimes.  Also, some Apache admins remove the
> "Apache" banner.  And some Apache-derived servers might still support
> the behavior while not advertising Apache in their server line.
I've done some testing.  I modified apache-userdir-enum.nse to exclude
the apache check and to stuff it's results into the registry so that
they could be dumped by the attached userdir-results.nse which runs at a
higher runlevel - I thought this would be easier than trying to parse
the results.  I wasted quite a lot of time trying to find services to
test by generating random IPs, but eventually wrote the attached ryl.nse
which uses Yahoo!s random link generator and writes hostnames to a file
for use by nmap (the filename is hardcoded).

If you would like to repeat the test you can run ryl.nse to generated a
list of webserver hosts like so:

nmap -sP -PN -n <any_target>

don't forget to change the hardcoded filenames
and then run the other scripts something like this:

nmap -n -sSV -PS80,443,8000,8443 -p 80,443,8000,8443 --script
apache-userdir-enum,userdir_results -iL <host_list>

and then look for the host script result (per hostgroup) from
userdir_results.

I did a fairly small test agaist 3746 services on ports 80, 443, 8000,
8080 and 8443. Spreadsheet anyone? [1]
Of those, 3446 yielded zero usernames.  300 yielded at least one username.

Of the 300:
292 were some version of Apache
Nmap was unable to determine version info for 3 services.
5 were other known services.

Of the 5 other:
2 (on the same host) were detected as "IBM HTTP Server (Based on
Apache)" and resulted in HTTP 403 for /~root/
1 was detected as "Microsoft IIS webserver 6.0" and yielded HTTP 200 for
/~admin/ and HTTP 403 for /~root/ - more on this in a minute.
2 (on the same host) were detected as "Microsoft IIS httpd" and resulted
in HTTP 200 for /~root/ but on closer inspection this turned out to be a
weird error page and a very minimal header in the response - certainly
not a userdir.

Of the 3 unknowns:
1 resulted in HTTP 200 for /~admin/ and I was unable to decide whether
or not this is likely to be a userdir - it's certainly possible.  The
header contained Server=Squeegit/1.2.5 (3_sir).  url:
http://209.202.252.41/~admin/
2 (on the same host) resulted in HTTP 403 for /~root/ and this looks
like a real userdir.  The error pages on these services report that the
service is "WebServerX" and look exactly like the supposed "Microsoft
IIS httpd" service noted above.  I believe they are both based on Apache
and it seems common to hide this fact using the headers.

This small test doesn't really persuade me either way in terms of
scanning non-apache servers.  On the one hand we have a couple IBM HTTP
Server (Based on Apache), possibly Squeegit and the probably
Apache-based WebServerX/IIS, but on the other we have a couple of false
positives.

Sven noted that lighttpd uses mod_userdir and it looks like Nginx does
too [2]. <http://wiki.nginx.org/NginxUserDir>
Perhaps the portrule could check for an extended list of services which
would have the benefit that when we know the service is something like
an ADSL router or IIS we won't bother trying something which is unlikely
to yield useful info.  The script would, of course, execute against all
http-like services when version detection is not performed.
Perhaps a bigger scan is in order!

Regards,

jah

[1] -
http://spreadsheets.google.com/pub?key=tSa60W2OW62a9KMDyF_EP-g&gid=1
<http://spreadsheets.google.com/pub?key=tSa60W2OW62a9KMDyF_EP-g&gid=1>
[2] - http://wiki.nginx.org/NginxUserDir

id=""
author=""
runlevel="1"
description = ""
categories = {}

hostrule = function( host )
  return true
end

local url    = require"url"
local http   = require"http"
local srdnse = require"stdnse"

action = function( host )
  
  local num_uhosts = 512
  local todo_file = "U:\jah\webservers_todo.list"
  local done_file = "U:\jah\webservers_done.list"
  
  
  -- http://random.yahoo.com/bin/ryl
  local ryl_host = "random.yahoo.com"
  local ryl_path = "/bin/ryl"
  
  local l, u = {}, {}
  local loc
  local req_count = 0
  
  local o = {}
  o.header = { ['User-Agent'] =
  'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12 (.NET CLR 3.5.30729)' }
  
  -- read already tested hosts from file
  local df = io.open(done_file,"r")
  assert(df)
  for line in df:lines() do
    u[line:lower()] = true
  end
  df:close()
  
  -- use yahoo to get num_uhosts unique random hosts
  while #l < num_uhosts do
        local r = http.get(ryl_host, 80, ryl_path, o)
        req_count = req_count+1
        if type(r) ~= 'table' or r.status ~= 302 or type(r.header) ~= 'table' or type(r.header.location) ~= 'string' 
then
          error("I don't think Yahoo! wants to play with you anymore...")
        else
                loc = url.parse(r.header.location)
                assert(type(loc.host) == 'string' and loc.host ~= '')
                -- l[#l+1] = ("%s:%s"):format(loc.host, loc.port or (loc.scheme == 'https' and 443) or 80)
                local rhost = loc.host:lower()
                if not u[rhost] then
                  u[rhost] = true
                  l[#l+1] = rhost
                end
    end
    stdnse.sleep(1) -- being polite
  end
  
  -- write unique hosts to file
  df = io.open(done_file,"a")
  local tf  = io.open(todo_file,"w")
  assert(tf and df)
  for _, line in ipairs(l) do
          line = ("%s\n"):format(line:lower())
          df:write(line)
          tf:write(line)
        end
        df:close()
        tf:close() 
        
        return ("%s requests were sent"):format(req_count)
end

id=""
author=""
runlevel="2"
description = ""
categories = {}

local tab = require"tab"

hostrule = function( host )
  return true
end

action = function( host )
  if not next(nmap.registry.apacheuserdirresults) then return nil end
  
  local r = nmap.registry.apacheuserdirresults
  local c = nmap.registry.apacheuserdircounts
  
  local rs = tab.new(3)
  
  tab.add(rs, 2, "Not Founds")
  tab.nextrow(rs)
  tab.add(rs, 1, "Count")
  tab.add(rs, 2, "Product")
  tab.nextrow(rs)
  for prod, count in pairs(c) do
    tab.add(rs, 1, tostring(count))
    tab.add(rs, 2, prod)
    tab.nextrow(rs)
  end 
  tab.nextrow(rs)
  tab.add(rs, 2, "--------------------")
  tab.nextrow(rs)
  
  tab.add(rs, 2, "Found")
  tab.nextrow(rs)
  tab.add(rs, 1, "Port")
  tab.add(rs, 2, "Product")
  tab.add(rs, 3, "Usernames")
  tab.nextrow(rs)
  
  for ip,s in pairs(r) do
    tab.add(rs, 2, ip)
    tab.nextrow(rs)
    for port, t in pairs(s) do
      tab.add(rs, 1, tostring(port))
      tab.add(rs, 2, t.ver)
      tab.add(rs, 3, t.dirs)
      tab.nextrow(rs)
    end
  end
  nmap.registry.apacheuserdirresults = {}
  local results = tab.dump(rs)
  return ("  \n%s"):format(results)
end

author = "jah <jah () zadkiel plus com>"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html";
categories = {"discovery"}
description = [[
Attempts to enumerate valid usernames on an apcahe server running with mod_userdir.

The Apache mod_userdir module allows user-specific directories to be accessed using the http://example.com/~user/ 
syntax.
This script makes http requests in order to discover valid user-specific directories and infer valid usernames.
By default, the script will use Nmaps nselib/data/usernames.lst
An http response status of 200 or 403 means the username is likely a valid one and the username will be output in the
script results along with the status code (in parentheses).

This script makes an attempt to avoid false positives by requesting a directory which is unlikely to exist.  If the
server responds with 200 or 403 then the script will not continue testing it.

Refs: CVE-2001-1013 - http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2001-1013
]]

---
-- @args
-- u=path/to/custom/usernames.list or
-- apacheuserdir.u=path/to/custom/usernames.list
--
-- @output
-- 80/tcp open  http    syn-ack Apache httpd 2.2.9
-- |_ apache-userdir-enum: Valid Users: root (403), user (200), test (200)

local http      = require 'http'
local stdnse    = require 'stdnse'
local datafiles = require 'datafiles'



---
-- The script will run against http[s] and http[s]-alt ports.  If version detection
-- is performed and discovers a service which does not contain 'apache' the script
-- will not run.
portrule = function(host, port)
  local svc = { std = { ["http"] = 1, ["http-alt"] = 1 },
                ssl = { ["https"] = 1, ["https-alt"] = 1 } }
  if port.protocol ~= 'tcp' or not
  ( svc.std[port.service] or svc.ssl[port.service] ) then
    return false
  end
  -- Don't bother running on SSL ports if we don't have SSL.
  if (svc.ssl[port.service] or port.version.service_tunnel == 'ssl') and not
  nmap.have_ssl() then
    return false
  end
  --[[ Don't bother running if version detection says something other than Apache
  if port.version and type(port.version.product) == 'string' and not
  port.version.product:lower():match('apache') then -- TODO also matches tomcat coyote - has mod_userdir or similar?
    return false
  end ]]--
  return true
end



action = function(host, port)

  if not nmap.registry.apacheuserdir then init() end
  local usernames = nmap.registry.apacheuserdir
  if #usernames == 0 then return nil end  -- speedy exit if no usernames

  local filename = filename:match( "[\\/]([^\\/]+)\.nse$" )
  local found = {}
  local data

  for i, uname in ipairs(usernames) do

    data = http.get( host, port, ("/~%s/"):format(uname) )
    if data and type(data.status) == 'number' then
      if (data.status == 403 or data.status == 200) and i == 1 then
        -- This server is unlikely to yield useful information since it returned
        -- 200 or 403 to a request for a directory which is highly unlikely to exist.
        stdnse.print_debug( 1, "%s detected false positives at %s:%d - status was %d", filename, host.ip, port.number, 
data.status)
        break
      elseif data.status == 403 or data.status == 200 then
        found[#found+1] = ("%s (%d)"):format(uname, data.status)
      -- else we didn't get an interesting status code
      end
    else
      stdnse.print_debug( 2, "%s got a bad or zero response from %s:%d", filename, host.ip, port.number)
    end

  end

  -- store result in registry
        local v, vs = {}, ""
        if port.version then
                v[#v+1] = type(port.version.product) == 'string' and port.version.product ~= '' and 
port.version.product or nil
                v[#v+1] = type(port.version.version) == 'string' and port.version.version ~= '' and 
port.version.version or nil
                vs = table.concat(v, " ")
                vs = vs:match("^%s*$") and 'unknown' or vs
  else
    vs = 'unknown'
  end
  
  if #found == 0 then
    nmap.registry.apacheuserdircounts[vs:lower()] = nmap.registry.apacheuserdircounts[vs:lower()] and 
nmap.registry.apacheuserdircounts[vs:lower()] + 1 or 1
    stdnse.print_debug( 2, "%s found Zero users at %s:%d", filename, host.ip, port.number)
    return nil
  end
        
        nmap.registry.apacheuserdirresults[host.ip] = nmap.registry.apacheuserdirresults[host.ip] or {}
        nmap.registry.apacheuserdirresults[host.ip][port.number] = 
        { ['ver'] = vs,
          ['dirs']= table.concat(found, ", ")
  }

  return ("Valid Users: %s"):format(table.concat(found, ", "))

end



---
-- Parses a file containing usernames (1 per line), defaulting to "nselib/data/usernames.lst"
-- and stores the resulting array of usernames in the registry for use by all threads of this
-- script.  This means file access is done only once per Nmap invocation.
-- init() also adds a random string to the array (in the first position) to attempt to catch
-- false positives.
-- @return nil

function init()
  nmap.registry.apacheuserdirresults = {}
  nmap.registry.apacheuserdircounts  = {}
  local filename = filename and filename:match( "[\\/]([^\\/]+)\.nse$" ) or ""
  local customlist = nmap.registry.args.u or
    (nmap.registry.args.apacheuserdir and nmap.registry.args.apacheuserdir.u) or
    nmap.registry.args['apacheuserdir.u']
  local read, usernames = datafiles.parse_file(customlist or "nselib/data/usernames.lst", {})
  if not read then
    stdnse.print_debug(1, "%s %s %s", filename, usernames and "" or "had issues reading usernames:",
      usernames or "No Error message")
    nmap.registry.apacheuserdir = {}
    return nil
  end
  -- random dummy username to catch false positives
  if #usernames > 0 then table.insert(usernames, 1, randomstring()) end
  nmap.registry.apacheuserdir = usernames
  stdnse.print_debug(1, "%s Testing %d usernames.", filename, #usernames)
  return nil
end



---
-- Uses openssl.rand_pseudo_bytes (if available, os.time() if not) and base64.enc
-- to produce a randomish string of at least 11 alphanumeric chars.
-- @return String

function randomstring()
  local bin    = require"bin"
  local base64 = require"base64"
  local rnd, s, l, _
  if pcall(require, "openssl") then
    rnd = openssl.rand_pseudo_bytes
  end

  s = rnd and rnd(8) or tostring( os.time() )
  -- increase the length of the string by 0 to 7 chars
  _, l = bin.unpack(">C", s, 8) -- eighth byte should be safe for os.time() too
  s = l%8 > 0 and s .. s:sub(1,l%8) or s
  -- base 64 encode and replace any non alphanum chars (with 'n' for nmap!)
  s = base64.enc(s):sub(1,-2):gsub("%W", "n")
  return s
end


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org