subprocess pipe question

[Chris Kaynor]

On Tue, Feb 22, 2011 at 3:44 PM, Rita <rmorgan466 at gmail.com> wrote:

> I have a process like this,
>
> def run(cmd):
>   #cmd=a process which writes a lot of data. Binary/ASCII data
>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>
> I would like to get cmd's return code so I am doing this,
>
> def run(cmd):
>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>   rc=p.poll()
>   print rc # and I get 'None' regardless of what cmd gives me (0 thru 255
> are valid return codes)
>   return p.stdout
>

Here, you are accessing the return code, however the process has not exited
yet, and thus you get None.


>
>
> When using wait() it works a bit better but not consistent
> def run(cmd):
>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>   rc=p.wait()
>   print rc
>   return p.stdout
>
> When the output of cmd is a small ascii file it works perfectly fine, but
> when the file is large (more than 2MB) the process just waits for ever (I am
> guessing its blocking?).  When I use the communicate call it works perfectly
> but my process is consuming way too much memory.


Here, you wait for the process to exit, with wait returning the exit code.
As you have waited for the process, it has exited and thus you get the
return code.

The issue here is that the stdout pipe has a limited size, and the process
will block once the pipe becomes full. You are entering a dead-lock as the
pipe is full, and thus the sub process is blocking waiting for room to write
more output. As nothing is reading that output, that will never happen. It
works fine on smaller output as you don't hit the limit. See the warning on
http://docs.python.org/library/subprocess.html#popen-objects under wait.

.communicate works as it reads the output, however it eats up memory as all
of the data has to be buffered in memory.

A possible work around would be to redirect the output into a file rather
than to a pipe (see the comments about stdin, stdout and stderr at
http://docs.python.org/library/subprocess.html#using-the-subprocess-module).
In this way, the data will be buffered to a file rather than memory.


>
> Is there a better way to get my return code consistently efficiently and
> not take up so much memory?
>
>
>
>
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110222/4a7f7247/attachment-0001.html>