Re: NSE: http.identify_404 follows redirects

[Tom Sellers <nmap () fadedcode net>]

The change did not appear to be disruptive in my tests. Unless someone objects I will commit the update to http.lua and 
as well as updates to the related scripts to standardize the call to identify_404.

Tom


On 11/30/2015 7:46 PM, Tom Sellers wrote:
> All,
>
>   I was going to open a git issue on this, but I decided to toss it at the list for discussion.
>
> References:
>   https://nmap.org/nsedoc/lib/http.html#identify_404
>   https://svn.nmap.org/nmap/nselib/http.lua
>
> http.identify_404 is a function that can be used to determine how an HTTP server responds to unknown pages. It can be 
> used, for example, to detect when an HTTP server responds 200 OK to everything
> which can break a script if it is merely checking the status code when requesting something like  /MyAppsSpecialPage.
>
>
> http.identify_404 follows HTTP redirects which may result in unexpected behavior.  I noticed this while testing some 
> changes to a script against a ethernet switch that generates a 302 redirect
> response for a request to /.  http.identify_404 follows the redirect and then the 'data' variable contains the 
> results for the new location.  The identify_404 function has code to deal with redirects
> and other errors but this won't be triggered if the call to http.get follows it first.
>
>
> Relevant code is at line 2476 in nselib/http.lua
>
>
> function identify_404(host, port)
>   local data
>   local bad_responses = { 301, 302, 400, 401, 403, 499, 501, 503 }
>
>   -- The URLs used to check 404s
>   local URL_404_1 = '/nmaplowercheck' .. os.time(os.date('*t'))
>   local URL_404_2 = '/NmapUpperCheck' .. os.time(os.date('*t'))
>   local URL_404_3 = '/Nmap/folder/check' .. os.time(os.date('*t'))
>
>   data = get(host, port, URL_404_1)
>
>
>
> I performed a review of the scripts where identify_404 is being used and I did not find any place where it looked 
> like following redirects would be desirable.
>
>
> grep -i 'identify_404' /usr/local/share/nmap/scripts/*.nse
>
> /usr/local/share/nmap/scripts/hnap-info.nse:  local status_404, result_404, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-avaya-ipoffice-users.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-backup-finder.nse:  local res, res404, known404 = http.identify_404(host, port)
> /usr/local/share/nmap/scripts/http-cakephp-version.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-default-accounts.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-default-accounts.nse:  local result, result_404, known_404 = 
> http.identify_404(host, port)
> /usr/local/share/nmap/scripts/http-enum.nse:  local result, result_404, known_404 = http.identify_404(host, port)
> /usr/local/share/nmap/scripts/http-huawei-hg5xx-vuln.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-malware-host.nse:  local result, result_404, known_404 = http.identify_404(host, 
> port)
> /usr/local/share/nmap/scripts/http-userdir-enum.nse:  local result, result_404, known_404 = http.identify_404(host, 
> port)
> /usr/local/share/nmap/scripts/http-vuln-cve2010-0738.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/http-wordpress-enum.nse:  local status_404, result_404, body_404 = 
> http.identify_404(host, port)
> /usr/local/share/nmap/scripts/http-wordpress-plugins.nse:  local status_404, result_404, body_404 = 
> http.identify_404(host, port)
> /usr/local/share/nmap/scripts/membase-http-info.nse:  local _, http_status, _ = http.identify_404(host,port)
> /usr/local/share/nmap/scripts/riak-http-info.nse:  local _, http_status, _ = http.identify_404(host,port)
>
>
> I recommend that we disable following redirects...
>
>   data = get(host, port, URL_404_1,{redirect_ok=false})
>
> but that means that we will have to do something intelligent with 301, 302, etc. in the context of the scripts above.
>
> Currently, for certain HTTP response codes  http.identify_404 is returning success, http status code, nil
>
>
> local bad_responses = { 301, 302, 400, 401, 403, 499, 501, 503 }
>
> .....<snip>......
>
>   -- Loop through any expected error codes
>   for _,code in pairs(bad_responses) do
>     if(data.status and data.status == code) then
>       stdnse.debug1("HTTP: Host returns %s instead of 404 File Not Found.", get_status_string(data))
>       return true, code
>     end
>   end
>
>
> This may change the logic in the calling scripts.  I think http.identify_404 should return false and let the debug 
> message from the code above be displayed if the user has enabled debugging. If there
> is approval I will implement the change and update the scripts above with a standard block of code.  This will be 
> tested against a python HTTP server that responds 200 OK to all requests as well as a
> server that responds to most default requests with a 302.  Some of the scripts, like http-cakephp-version.nse, need 
> to have the logic adjusted anyway to address some false positives.
>
> Thoughts?
>
>
> Tom
>
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/