CRLF Newlines and libc

[Gabriel Genellina]

En Mon, 27 Jul 2009 12:21:29 -0300, Eric <gmweezel at gmail.com> escribió:

> I am working on the subprocess.Popen module for Google Summer of Code.
> Right now, I am having difficulty trying to work out how to deal with
> my '\n' newlines being converted to '\r\n' newlines when reading from
> a pipe on windows; see this blog post (http://subdev.blogspot.com/
> 2009/07/stdout.html) and this chain on Python-Dev (http://
> mail.python.org/pipermail/python-dev/2009-July/090720.html).
>
> Right now, unless there is way to disable the newline conversions, I
> am thinking I will just have to document the caveat and call it a day.
> Is there a better way to handle this?

It isn't clear to me what exactly your problem is. What are you reading,  
using which functions, how did you open the file? On the writing side, how  
did you open the file, which functions are used to write?

The Windows API functions like CreateFile, ReadFile, WriteFile... don't  
perform any kind of '\n' conversion, neither reading nor writing. On the  
other hand, the C library does differentiate between files opened in text  
mode or binary mode (this comes from the ANSI C standard, it's not Windows  
specific). The default is "text" mode (unless you add "b" to the fopen  
call, or O_BINARY to the open call). In that mode, '\n' is translated to  
'\r\n' on writing, and '\r\n' is translated to '\n' on reading.
Python files are implemented on top of the C library (on 2.x) so they  
inherit that behaviour.
Console programs written in C and using printf/fwrite/write to stdout  
share that behaviour too (standard handles are opened in text mode by  
default).

Note that """'\n' newlines being converted to '\r\n' newlines when  
reading""" cannot happen -- if you see '\r\n' when reading, it's because
a) you're reading the file in binary mode, and
b) the file was written in text mode.

b) is true for process output redirected to a pipe, so you'll have to fix  
a). How to do that, depends on what exactly you're doing.

-- 
Gabriel Genellina