Reassign or discard Popen().stdout from a server process

[Chris Rebert]

On Tue, Feb 1, 2011 at 12:30 AM, John O'Hagan <mail at johnohagan.com> wrote:
> I'm starting a server process as a subprocess. Startup is slow and
> unpredictable (around 3-10 sec), so I'm reading from its stdout until I get a
> line that tells me it's ready before proceeding, in simplified form:
>
> import subprocess
> proc = subprocess.Popen(['server', 'args'], stdout=subprocess.PIPE)
> while proc.stdout.readline() != "Ready.\n":
>    pass
>
> Now I can start communicating with the server, but I eventually realised that
> as I'm no longer reading stdout, the pipe buffer will fill up with output from
> the server and before long it blocks and the server stops working.
>
> I can't keep reading because that will block - there won't be any more output
> until I send some input, and I don't want it in any case.
>
> To try to fix this I added:
>
> proc.stdout = os.path.devnull
>
> which has the effect of stopping the server from failing, but I'm not convinced
> it's doing what I think it is. If I replace devnull in the above line with a
> real file, it stays empty although I know there is more output, which makes me
> think it hasn't really worked.

Indeed. proc.stdout is a file, whereas os.devnull is merely a path
string; the assignment is nonsensical type-wise.

> Simply closing stdout also seems to stop the crashes, but doesn't that mean
> it's still being written to, but the writes are just silently failing? In

Based on some quick experimentation, yes, more or less.

> either case I'm wary of more elusive bugs arising from misdirected stdout.
>
> Is it possible to re-assign the stdout of a subprocess after it has started?

I think that's impossible. (Most of Popen's attributes probably should
be read-only properties to clarify that such actions are don't have
the intended effect.)

> What's the right way to read stdout up to a given line, then
> discard the rest?

I would think calling Popen.communicate() after you've reached the
given line should do the trick.
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
Just ignore its return value. However, that does require sending the
input all at once in a single chunk, which it sounds like may not be
feasible in your case; if so, I have no idea how to do it cleanly.

Cheers,
Chris
--
http://blog.rebertia.com