Re: [RFC][NSE] Incomplete HTTP response body

[nnposter <nnposter () users sourceforge net>]

If it ultimately turns out there is a desire for this then there are a
few aspects that could be debated:

* The body fragment is intentionally returned in a unique response
member, "fragment", to have a clear separation from the (complete) body,
forcing a consuming code to acknowledge the intent to operate on less
than the complete body. That said, it would be trivial to store the
fragment in the "body" member.

* The body fragment does process chunked encoding. It concatenates all
processed chunks up to the point of failure. IMHO this is more useful
than to provide the raw body before the decoding took place. Again, this
could be changed if the consensus goes the other way.

* The patch does not try to preserve received headers or the status
line, only the body itself. This functionality could be added. (If yes,
then the follow-up topic would be how to store them in the response
object. My proposal would be to repurpose "fragment" to contain an
incomplete response object, i.e. the headers or the body would be in
resp.fragment.header and resp.fragment.body, respectively.)


Cheers,
nnposter



On 3/19/17 6:55 AM, Daniel Miller wrote:
> My initial thought is that this is very interesting. I can't remember
> specifically, but it does seem like I've encountered cases where it
> would have been useful. I'll look at the code itself shortly and give
> feedback.
>
> Dan
>
> On Fri, Mar 17, 2017 at 3:16 PM, nnposter
> <nnposter () users sourceforge net <mailto:nnposter () users sourceforge net>>
> wrote:
>
>     Nmappers,
>     Recent e-mail thread "Get value in IncompleteRead exception",
>     http://seclists.org/nmap-dev/2017/q1/191
>     <http://seclists.org/nmap-dev/2017/q1/191>, covered the fact that the
>     current NSE HTTP library does not provide means how to obtain partially
>     received response bodies. If the response parsing fails for whatever
>     reason the caller only gets a nil status, and the status line contains a
>     one-line error message. The caller specifically does not get any data
>     from the response itself.
>
>     I have put together a patch that adds a new member to the response
>     object. If an HTTP response fails while processing the body then this
>     member gets populated with a body fragment received up to that point.
>
>     The value proposition is that probing for HTTP vulnerabilities sometimes
>     results in incorrectly formed bodies. The content length might be off or
>     the chunks are corrupted. With this modification a vulnerability test
>     script might still be able to use the HTTP library, instead of
>     hand-rolling the request.
>
>     I am looking for opinions whether such a functionality is desirable
>     or not.
>
>     The attached patch applies cleanly against r36651 if you want to try it
>     out. I am not going to commit it by default, without a reasonable
>     consensus.
>
>
>     Thank you for your thoughts,
>     nnposter
>
>     _______________________________________________
>     Sent through the dev mailing list
>     https://nmap.org/mailman/listinfo/dev
>     <https://nmap.org/mailman/listinfo/dev>
>     Archived at http://seclists.org/nmap-dev/
>
>
>
>
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/