Reassign or discard Popen().stdout from a server process

[John O'Hagan]

On Tue, 1 Feb 2011, Chris Rebert wrote:
> On Tue, Feb 1, 2011 at 12:30 AM, John O'Hagan <mail at johnohagan.com> wrote:
> > I'm starting a server process as a subprocess. Startup is slow and
> > unpredictable (around 3-10 sec), so I'm reading from its stdout until I
> > get a line that tells me it's ready before proceeding, in simplified
> > form:
> > 
> > import subprocess
> > proc = subprocess.Popen(['server', 'args'], stdout=subprocess.PIPE)
> > while proc.stdout.readline() != "Ready.\n":
> >    pass
> > 
> > Now I can start communicating with the server, but I eventually realised
> > that as I'm no longer reading stdout, the pipe buffer will fill up with
> > output from the server and before long it blocks and the server stops
> > working.
> > 
> > I can't keep reading because that will block - there won't be any more
> > output until I send some input, and I don't want it in any case.
> > 
> > To try to fix this I added:
> > 
> > proc.stdout = os.path.devnull
> > 
> > which has the effect of stopping the server from failing, but I'm not
> > convinced it's doing what I think it is. If I replace devnull in the
> > above line with a real file, it stays empty although I know there is
> > more output, which makes me think it hasn't really worked.
> 
> Indeed. proc.stdout is a file, whereas os.devnull is merely a path
> string; the assignment is nonsensical type-wise.

My mistake, of course I meant open(os.path.devnull).

> > Simply closing stdout also seems to stop the crashes, but doesn't that
> > mean it's still being written to, but the writes are just silently
> > failing? In
> 
> Based on some quick experimentation, yes, more or less.
> 
> > either case I'm wary of more elusive bugs arising from misdirected
> > stdout.
> > 
> > Is it possible to re-assign the stdout of a subprocess after it has
> > started?
> 
> I think that's impossible. (Most of Popen's attributes probably should
> be read-only properties to clarify that such actions are don't have
> the intended effect.)

I don't doubt what you say, but attempting to assign it does seem to do 
something, as it consistently stops the crashes which occur otherwise. What it 
does, I have no idea.

> > What's the right way to read stdout up to a given line, then
> > discard the rest?
> 
> I would think calling Popen.communicate() after you've reached the
> given line should do the trick.
> http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
> Just ignore its return value. However, that does require sending the
> input all at once in a single chunk, which it sounds like may not be
> feasible in your case; if so, I have no idea how to do it cleanly.

Yes, unfortunately I need to send a lot of precisely-timed short strings, and 
communicate() blocks after the first call. I tried calling stdout.readline() 
the right number of times after each input, but that seems fiddly and fragile - 
for example the number of lines of output is not guaranteed and may vary in 
the case of errors, and also the extra reads had a noticeable effect on 
latency, which is important in this case. 

So far my best bet seems to be closing stdin, which doesn't seem very clean, 
but it does what I want and seems to be just as fast as using 
stdin=open(os.devnull) in the Popen call in the first place.

Thanks,

John