Re: [NSE] http.lua and delimiters

[jah <jah () zadkiel plus com>]

On 02/10/2008 05:00, David Fifield wrote:
> I have to say, those are compelling numbers. I may have been too quick
> to judge your patch earlier. I'm looking forward to seeing your updated
> patch.
>   
Useful input - as usual, David.
>   
> > I'm pretty much ready to submit an updated patch as used for this test,
> > but there's just one thing I'm wondering about adding.  The header value
> > containing the status code (Status-Line) is currently discarded after
> > the code itself is captured, but I'm tending toward keeping it to be
> > more complete.  Also, sometimes they're almost interesting:
> > HTTP/1.1 403 Forbidden ( The server denied the specified Uniform
> > Resource Locator (URL). Contact the server administrator.  )
> >     
>
> That's fine by me. I don't it should be part of the header, rather a
> separate table entry, because it appears it's not really considered part
> of the header:
>
>         generic-message = start-line
>                           *(message-header CRLF)
>                           CRLF
>                           [ message-body ]
>         start-line      = Request-Line | Status-Line
>
>         message-header = field-name ":" [ field-value ]
>   
The attached patch adds to the previous one with the following:

The library returns status-line: {status, status-line, header, body} if
the status code is successfully matched - otherwise it's nil, the same
as status.  I've updated the scripts nsedoc with this info and attached
a possible patch for docs/scripting.xml.

The test for the shortest header when the response matches both \r\n\r\n
and \n\n.  This was present during the testing I did yesterday.

A new pattern to split the header from the body: \n\r\n.  I managed to
hit (once) the code which catches responses where there is no clear
separation between header and body and this pattern was introduced in
reaction to that one response caught by this code  (The same code I was
thinking of removing).  The new pattern wasn't present during yesterdays
testing, but more testing has been done since then to ensure that this
addition doesn't cause side-effects (and so far, it hasn't).  I've
decided to leave the catching code in too as there's the possibility
we'll discover other ways to separate header from body (a possibility
might be \r\n\r).

Comments and debug statements are removed or tidied as applicable.

Regards,

jah

--- http.lua.orig       2008-09-20 21:55:48.484375000 +0100
+++ http.lua    2008-10-02 12:31:51.718750000 +0100
@@ -1,12 +1,14 @@
 --- The http module provides functions for dealing with the client side
 -- of the http protocol. The functions reside inside the http namespace.
 -- The return value of each function in this module is a table with the
--- following keys: status, header and body. status is a number representing
--- the HTTP status code returned in response to the HTTP request. In case
--- of an unhandled error, status is nil. The header value is a table
--- containing key-value pairs of HTTP headers received in response to the
--- request. The header names are in lower-case and are the keys to their
--- corresponding header values (e.g. header.location = "http://nmap.org/";).
+-- following keys: status, status-line, header and body. status is a number
+-- representing the HTTP status code returned in response to the HTTP
+-- request. In case of an unhandled error, status is nil. status-line is
+-- the entire status message which includes the HTTP version, status code
+-- and reason phrase. The header value is a table containing key-value
+-- pairs of HTTP headers received in response to the request. The header
+-- names are in lower-case and are the keys to their corresponding header
+-- values (e.g. header.location = "http://nmap.org/";).
 -- Multiple headers of the same name are concatenated and separated by
 -- commas. The body value is a string containing the body of the HTTP
 -- response.
@@ -14,8 +16,8 @@
 
 module(... or "http",package.seeall)
 
-require 'stdnse'
-require 'url'
+local url    = require 'url'
+local stdnse = require 'stdnse'
 
 --
 -- http.get( host, port, path, options )
@@ -26,7 +28,7 @@
 -- port may either be a number or a table
 --
 -- the format of the return value is a table with the following structure:
--- {status = 200, header = {}, body ="<html>...</html>"}
+-- {status = 200, status-line = "HTTP/1.1 200 OK", header = {}, body ="<html>...</html>"}
 -- the header table has an entry for each received header with the header name being the key
 -- the table also has an entry named "status" which contains the http status code of the request
 -- in case of an error status is nil
@@ -43,7 +45,7 @@
 -- @param host The host to query.
 -- @param port The port for the host.
 -- @param path The path of the resource.
--- @param options A table of optoins. See function description.
+-- @param options A table of options. See function description.
 -- @return table
 get = function( host, port, path, options )
   options = options or {}
@@ -113,7 +115,7 @@
   options = options or {}
 
   if type(host) == 'table' then
-    host = host.ip
+    host = host.targetname or host.ip
   end
 
   local protocol = 'tcp'
@@ -124,7 +126,7 @@
     port = port.number
   end
 
-  local result = {status=nil,header={},body=""}
+  local result = {status=nil,["status-line"]=nil,header={},body=""}
   local socket = nmap.new_socket()
   local default_timeout = {}
   if options.timeout then
@@ -133,36 +135,76 @@
     default_timeout = get_default_timeout( nmap.timing_level() )
     socket:set_timeout( default_timeout.connect )
   end
+
   if not socket:connect( host, port, protocol ) then
     return result
   end
+
   if not options.timeout then
     socket:set_timeout( default_timeout.request )
   end
+
   if not socket:send( data ) then
     return result
   end
 
-  local buffer = stdnse.make_buffer( socket, "\r\n" )
+  -- no buffer - we want everything now!
+  local response = {}
+  while true do
+    local status, part = socket:receive()
+    if not status then
+      break
+    else
+      response[#response+1] = part
+    end
+  end
 
-  local line, _
-  local header, body = {}, {}
+  socket:close()
 
-  -- header loop
-  while true do
-    line = buffer()
-    if (not line or line == "") then break end
-    table.insert(header,line)
+  response = table.concat( response )
+
+  -- try and separate the head from the body
+  local header, body, h1, h2, b1, b2
+  if response:match( "\r\n\r\n" ) and response:match( "\n\n" ) then
+    h1, b1 = response:match( "^(.-)\r\n\r\n(.*)$" )
+    h2, b2 = response:match( "^(.-)\n\n(.*)$" )
+    if h1 and h2 and h1:len() <= h2:len() then
+      header, body = h1, b1
+    else
+      header, body = h2, b2
+    end
+  elseif response:match( "\r\n\r\n" ) then
+    header, body = response:match( "^(.-)\r\n\r\n(.*)$" )
+  elseif response:match( "\n\r\n" ) then
+    header, body = response:match( "^(.-)\n\r\n(.*)$" )
+  elseif response:match( "\n\n" ) then
+    header, body = response:match( "^(.-)\n\n(.*)$" )
+  else
+    body = response
+  end
+
+  local head_delim, body_delim
+  if type( header ) == "string" then
+    head_delim = ( header:match( "\r\n" ) and "\r\n" )  or
+                 ( header:match( "\n" )   and "\n" ) or nil
+    header = ( head_delim and stdnse.strsplit( head_delim, header ) ) or { header }
   end
 
+  if type( body ) == "string" then
+    body_delim = ( body:match( "\r\n" ) and "\r\n" )  or
+                 ( body:match( "\n" )   and "\n" ) or nil
+  end
+
+  local line, _
+
   -- build nicer table for header
   local last_header, match
-  for number, line in ipairs( header ) do
+  for number, line in ipairs( header or {} ) do
     if number == 1 then
       local code
       _, _, code = string.find( line, "HTTP/%d\.%d (%d+)")
       result.status = tonumber(code)
-      if not result.status then table.insert(body,line) end
+      if code then result["status-line"] = line end
     else
       match, _, key, value = string.find( line, "(.+): (.*)" )
       if match and key and value then
@@ -177,39 +219,42 @@
         match, _, value = string.find( line, " +(.*)" )
         if match and value and last_header then
           result.header[last_header] = result.header[last_header] .. ',' .. value
-        elseif match and value then
-          table.insert(body,line)
         end
       end
     end
   end
 
-  -- handle body
-  if result.header['transfer-encoding'] == 'chunked' then
-    -- if the server used chunked encoding we have to 'dechunk' the answer
-    local counter, chunk_size
-    counter = 0; chunk_size = 0
-    while true do
-      if counter >= chunk_size then
-        counter = 0
-        chunk_size = tonumber( buffer(), 16 )
-        if chunk_size == 0 or not chunk_size then break end
+  -- handle chunked encoding
+  if type( result.header ) == "table" and result.header['transfer-encoding'] == 'chunked' and type( body_delim ) == 
"string" then
+    body = body_delim .. body
+    local b = {}
+    local start, ptr = 1, 1
+    local chunk_len
+    local pattern = ("%s([^%s]+)%s"):format( body_delim, body_delim, body_delim )
+    while ( ptr < ( type( body ) == "string" and body:len() ) or 1 ) do
+      local hex = body:match( pattern, ptr )
+      if not hex then break end
+      chunk_len = tonumber( hex or 0, 16 ) or nil
+      if chunk_len then
+        start = ptr + hex:len() + 2*body_delim:len()
+        ptr = start + chunk_len
+        b[#b+1] = body:sub( start, ptr-1 )
       end
-      line = buffer()
-      if not line then break end
-      counter = counter + #line + 2
-      table.insert(body,line)
     end
-  else
-    while true do
-      line = buffer()
-      if not line then break end
-      table.insert(body,line)
+    body = table.concat( b )
+  end
+
+  -- special case for conjoined header and body
+  if type( result.status ) ~= "number" and type( body ) == "string" then
+    local code, remainder = body:match( "HTTP/%d\.%d (%d+)(.*)") -- The Reason-Phrase will be prepended to the body :(
+    if code then
+      stdnse.print_debug( "Interesting variation on the HTTP standard.  Please submit a --script-trace output for this 
host (%s) to nmap-dev[at]insecure.org.", host )
+      result.status = tonumber(code)
+      body = remainder or body
     end
   end
 
-  socket:close()
-  result.body = table.concat( body, "\r\n" )
+  result.body = body
 
   return result
 

--- scripting.xml.orig  2008-10-02 14:13:39.687500000 +0100
+++ scripting.xml       2008-10-02 14:12:46.781250000 +0100
@@ -1587,16 +1587,19 @@
        The <literal>http</literal> module provides functions for dealing with the client side of the http protocol.
        The functions reside inside the <literal>http</literal> namespace.
        The return value of each function in this module is a table with the following keys:
-       <literal>status</literal>, <literal>header</literal> and <literal>body</literal>.
+       <literal>status</literal>, <literal>status-line</literal>, <literal>header</literal>
+       and <literal>body</literal>.
 
        <literal>status</literal> is a number representing the HTTP
        status code returned in response to the HTTP request. In case
        of an unhandled error, <literal>status</literal>
-       is <literal>nil</literal>. The <literal>header</literal> value
-       is a table containing key-value pairs of HTTP headers received
-       in response to the request. The header names are in lower-case
-       and are the keys to their corresponding header values
-       (e.g. <literal>header.location =
+       is <literal>nil</literal>. <literal>status-line</literal> is
+       the entire status message which includes the HTTP version,
+       status code and reason phrase. The <literal>header</literal>
+       value is a table containing key-value pairs of HTTP headers
+       received in response to the request. The header names are in
+       lower-case and are the keys to their corresponding header
+       values (e.g. <literal>header.location =
        "http://nmap.org/";</literal>).  Multiple headers of the same
        name are concatenated and separated by
        commas. The <literal>body</literal> value is a string


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org