[Python-ideas] Replacing the standard IO streams (was Re: changing sys.stdout encoding)

[Nick Coghlan]

On Mon, Jun 11, 2012 at 12:34 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 10.06.12 00:22, Mark Lawrence wrote:
>>
>> On 09/06/2012 21:02, Serhiy Storchaka wrote:
>>>
>>> None of these methods are not guaranteed to work if the input or output
>>> have occurred before.
>>
>>
>> That's a double negative so I'm not sure what you meant to say. Can you
>> please rephrase it. I assume that English is not your native language,
>> so I'll let you off :)
>
>
> open(sys.stdin.fileno()) is not guaranteed to work if the input or output
> have occurred before. And io.TextIOWrapper(sys.stdin.detach()) is not
> guaranteed to work if the input or output have occurred before. sys.stdin
> internal buffer can contains read by not used characters. sys.stdin.buffer
> internal buffer can contains read by not used bytes. With multibyte encoding
> sys.stdin.decoder internal buffer can contains uncompleted multibyte
> character.

Right, but the point of this discussion is to document the cleanest
available way for an application to change these settings at
*application start* (e.g. to support an "--encoding" parameter). Yes,
there are potential issues if you use any of these mechanisms while
there is data in the buffers, but that's a much harder problem and not
one we're trying to solve here.

Regardless, the advantage of the "open + fileno" idiom is that it
works for *any* level of change. If you want to force your streams to
unbuffered binary IO rather than merely changing the encoding:

    sys.stdin = open(sys.stdin.fileno(), 'rb', buffering=0, closefd=False)
    sys.stdout = open(sys.stdout.fileno(), 'wb', buffering=0, closefd=False)
    sys.stderr = open(sys.stderr.fileno(), 'wb', buffering=0, closefd=False)

Keep them as text, but force them to permissive utf-8, no matter how
the interpreter originally created them?:

    sys.stdin = open(sys.stdin.fileno(), 'r', encoding="utf-8",
errors="surrogateescape", closefd=False)
    sys.stdout = open(sys.stdout.fileno(), 'w', encoding="utf-8",
errors="surrogateescape", closefd=False)
    sys.stderr = open(sys.stderr.fileno(), 'w', encoding="utf-8",
errors="surrogateescape", closefd=False)

This approach also has the advantage of leaving
sys.__std(in/out/err)__ in a somewhat usable state.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia