Re: Ncat and windows line endings

[David Fifield <david () bamsoftware com>]

On Fri, May 08, 2009 at 08:42:25PM -0700, Fyodor wrote:
> On Thu, May 07, 2009 at 10:48:46PM +0100, jah wrote:
> > I felt sure that something like this has been raised before, but haven't
> > found anything pertinent.  Is this a known issue and, more to the point,
> > can we afford to modify ncat to do something about it given that \r\n
> > might have been sent as part of a HTTP GET, for example?
>
> Hi Jah.  Excellent point.  I wish we there was a way we could
> automatically detect and do "the right thing".  But as you noted,
> passing \r\n straight through may be desirable in some cases
> (e.g. writing HTTP requests) while changing \r\n to \n is desirable in
> others (talking to a UNIX shell).  We should probably at least provide
> options for both approaches.  We currently offer:
>
> First of all, maybe -C should format all bare \n in the buffer to
> \r\n.  Is there a good reason (other than ease of implementation) for
> only handling bare \n which happens to appear at the end of the
> buffer?  Of course it needs to make sure that the \n chars in the
> buffer are bare (not preceeded by a \r already).  We don't want to end
> up with \r\r\n.
>
> Second, we should probably have an option which does the opposite and
> converts \r\n in input to plain \n.  Maybe it could be -N/--newline or
> -L/--lf or something.  It should check the whole buffer for \r\n too,
> IMHO.
>
> Does anyone have a different idea or should we just implement these
> two changes?  Anyone want to write a patch?

I did some research on this. It turns out that Ncat on Windows used to
*automatically* convert \r\n to \n on input, no matter the source,
because of the way Windows handles newline translation. There is a
global variable _fmode that controls whether newline translation is
done. It can be _O_TEXT (do translation) or _O_BINARY (no translation),
and its default value is _O_TEXT.

Text and Binary Mode File I/O
http://msdn.microsoft.com/en-us/library/ktss1a9b.aspx
_fmode
http://msdn.microsoft.com/en-us/library/ee2849wt.aspx

This automatic translation was unintentionally lost when I made the
change to read from stdin in a thread so that non-blocking reads are
possible. Background: http://seclists.org/nmap-dev/2009/q1/0612.html. In
copying file descriptors to make the thread work, we lose the default
_O_TEXT mode.

You can verify that this is the case by commenting out the call to
win_stdin_start_thread and making win_stdin_ready always true in
nbase/nbase_misc.c (patch attached). Without the patch, pressing Enter
in connect mode with -vvv gives
NSOCK (1.2500s) Callback READ SUCCESS for EID 26 (peer unspecified) (2 bytes)
With the patch, it is
NSOCK (0.2970s) Callback READ SUCCESS for EID 26 (peer unspecified) (1 bytes)

I found a way to retain the _O_TEXT setting after duplicating the file
descriptor. This is like a permanent -L setting, and is the way Ncat
used to work. There's no extra work on Ncat's part; as far as it can
tell, the bare \n it sees comes straight from the terminal. -C works to
convert the \n that Ncat sees into \r\n, which is nice because the
command line usage will be exactly the same on Windows and Unix; on
Windows we can simply pretend that the terminal sends \n. Windows users
don't have to remember to use -C to talk to a web server when they're on
Unix, and Unix users don't have to remember to use -L to talk to a shell
when they're on Windows. You always use -C when talking to protocols
that require it, and otherwise your newlines are \n on any platform.

A problem with this is that the newline translation applies to all forms
of input: pipes and files, not just the console. That will break binary
file transfers, for example. So this is what I propose: Set _O_TEXT on
Windows when stdin is a terminal, and set _O_BINARY otherwise. Set
_O_BINARY on stdout unconditionally.

With this, all terminals would appear the same, so there would be no -L
option. We would apply half of jah's patch, the part that fixes -C to
replace \n with \r\n anywhere in an input buffer. I have a patch for
_O_TEXT that I'll submit in a little bit.

David Fifield

Attachment: nbase-disable-crlf.diff
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org