Re: NSE: http.identify_404 follows redirects

[Johanna Curiel <johannapcuriel () gmail com>]

Hi Tom

If you commit this change if no one objects, I would like to test it and
feedback my results to you

Also you can make available the scripts you changed  and I can feedback my
results


regards

Johanna

On Thu, Dec 3, 2015 at 8:56 AM, Tom Sellers <nmap () fadedcode net> wrote:

> The change did not appear to be disruptive in my tests. Unless someone
> objects I will commit the update to http.lua and as well as updates to the
> related scripts to standardize the call to identify_404.
>
> Tom
>
>
> On 11/30/2015 7:46 PM, Tom Sellers wrote:
> > All,
> >
> >   I was going to open a git issue on this, but I decided to toss it at
> the list for discussion.
> > References:
> >   https://nmap.org/nsedoc/lib/http.html#identify_404
> >   https://svn.nmap.org/nmap/nselib/http.lua
> >
> > http.identify_404 is a function that can be used to determine how an
> HTTP server responds to unknown pages. It can be used, for example, to
> detect when an HTTP server responds 200 OK to everything
> > which can break a script if it is merely checking the status code when
> requesting something like  /MyAppsSpecialPage.
> > http.identify_404 follows HTTP redirects which may result in unexpected
> behavior.  I noticed this while testing some changes to a script against a
> ethernet switch that generates a 302 redirect
> > response for a request to /.  http.identify_404 follows the redirect and
> then the 'data' variable contains the results for the new location.  The
> identify_404 function has code to deal with redirects
> > and other errors but this won't be triggered if the call to http.get
> follows it first.
> > Relevant code is at line 2476 in nselib/http.lua
> >
> >
> > function identify_404(host, port)
> >   local data
> >   local bad_responses = { 301, 302, 400, 401, 403, 499, 501, 503 }
> >
> >   -- The URLs used to check 404s
> >   local URL_404_1 = '/nmaplowercheck' .. os.time(os.date('*t'))
> >   local URL_404_2 = '/NmapUpperCheck' .. os.time(os.date('*t'))
> >   local URL_404_3 = '/Nmap/folder/check' .. os.time(os.date('*t'))
> >
> >   data = get(host, port, URL_404_1)
> >
> >
> >
> > I performed a review of the scripts where identify_404 is being used and
> I did not find any place where it looked like following redirects would be
> desirable.
> > grep -i 'identify_404' /usr/local/share/nmap/scripts/*.nse
> >
> > /usr/local/share/nmap/scripts/hnap-info.nse:  local status_404,
> result_404, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-avaya-ipoffice-users.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-backup-finder.nse:  local res,
> res404, known404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-cakephp-version.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-default-accounts.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-default-accounts.nse:  local result,
> result_404, known_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-enum.nse:  local result, result_404,
> known_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-huawei-hg5xx-vuln.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-malware-host.nse:  local result,
> result_404, known_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-userdir-enum.nse:  local result,
> result_404, known_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-vuln-cve2010-0738.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/http-wordpress-enum.nse:  local
> status_404, result_404, body_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/http-wordpress-plugins.nse:  local
> status_404, result_404, body_404 = http.identify_404(host, port)
> > /usr/local/share/nmap/scripts/membase-http-info.nse:  local _,
> http_status, _ = http.identify_404(host,port)
> > /usr/local/share/nmap/scripts/riak-http-info.nse:  local _, http_status,
> _ = http.identify_404(host,port)
> > I recommend that we disable following redirects...
> >
> >   data = get(host, port, URL_404_1,{redirect_ok=false})
> >
> > but that means that we will have to do something intelligent with 301,
> 302, etc. in the context of the scripts above.
> > Currently, for certain HTTP response codes  http.identify_404 is
> returning success, http status code, nil
> > local bad_responses = { 301, 302, 400, 401, 403, 499, 501, 503 }
> >
> > .....<snip>......
> >
> >   -- Loop through any expected error codes
> >   for _,code in pairs(bad_responses) do
> >     if(data.status and data.status == code) then
> >       stdnse.debug1("HTTP: Host returns %s instead of 404 File Not
> Found.", get_status_string(data))
> >       return true, code
> >     end
> >   end
> >
> >
> > This may change the logic in the calling scripts.  I think
> http.identify_404 should return false and let the debug message from the
> code above be displayed if the user has enabled debugging. If there
> > is approval I will implement the change and update the scripts above
> with a standard block of code.  This will be tested against a python HTTP
> server that responds 200 OK to all requests as well as a
> > server that responds to most default requests with a 302.  Some of the
> scripts, like http-cakephp-version.nse, need to have the logic adjusted
> anyway to address some false positives.
> > Thoughts?
> >
> >
> > Tom
> >
> > _______________________________________________
> > Sent through the dev mailing list
> > https://nmap.org/mailman/listinfo/dev
> > Archived at http://seclists.org/nmap-dev/
>
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/