[Python-Dev] Implementing File Modes

[Glyph Lefkowitz]

On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> I like MRAB's idea of using a (non-standard) "e" flag to include
> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
>
> Anything more complicated probably should just use "raw" Popen
> objects. Don't overcomplicate the interface.
>

In my opinion, mangling stderr and stdout together is already an
overcomplication.  It shouldn't be implemented.

It *seems* like a good idea, until you realize that subtle changes to your
OS, environment, or buffering behavior may result in arbitrary, unparseable
output.

For example, let's say you've got a program whose output is a list of lines,
each one containing a number.  Sometimes it tries to import gtk, and fails
to open its display.

That's fine, and you can still deal with it, as long as the interleaved
output looks like this:

100
200
Gtk-WARNING **: cannot open display:
300
400

but of course the output *might* (although unlikely with such small chunks
of output) end up looking like this, instead:

100
2Gtk-WAR0NING0 **:
can30not 0open display:

400

this is the sort of thing which is much more likely to happen once you start
dealing with large volumes of data, where there are more page-boundaries for
your buffers to get confused around, and you are playing with buffering
options to improve performance.  In other words, it's something that fails
only at scale or under load, and is therefore extremely difficult to debug.

This option *might* be okay if it were allowed only on subprocesses opened
in a *text* mode, and if the buffering logic involved forced stderr and
stdout to be line-delimited, and interleave only lines, rather than
arbitrary chunks of bytes.  Of course then if you use this flag with a
program that outputs binary data with no newlines it will buffer forever and
crash your program with a MemoryError, but at least that's easy to debug
when it happens.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090727/34b5c7ff/attachment.htm>