subprocess pipe question

[Rita]

Thanks everyone for your replies.

Chris,
Unfortunately, I can't redirect the output to a file because there are other
processes which use this processes output as stdin.

Rob,
I will give this a try.



On Tue, Feb 22, 2011 at 7:48 PM, Chris Kaynor <ckaynor at zindagigames.com>wrote:

>
> On Tue, Feb 22, 2011 at 3:44 PM, Rita <rmorgan466 at gmail.com> wrote:
>
>> I have a process like this,
>>
>> def run(cmd):
>>   #cmd=a process which writes a lot of data. Binary/ASCII data
>>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>>
>> I would like to get cmd's return code so I am doing this,
>>
>> def run(cmd):
>>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>>   rc=p.poll()
>>   print rc # and I get 'None' regardless of what cmd gives me (0 thru 255
>> are valid return codes)
>>   return p.stdout
>>
>
> Here, you are accessing the return code, however the process has not exited
> yet, and thus you get None.
>
>
>>
>>
>> When using wait() it works a bit better but not consistent
>> def run(cmd):
>>   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>>   rc=p.wait()
>>   print rc
>>   return p.stdout
>>
>> When the output of cmd is a small ascii file it works perfectly fine, but
>> when the file is large (more than 2MB) the process just waits for ever (I am
>> guessing its blocking?).  When I use the communicate call it works perfectly
>> but my process is consuming way too much memory.
>
>
> Here, you wait for the process to exit, with wait returning the exit code.
> As you have waited for the process, it has exited and thus you get the
> return code.
>
> The issue here is that the stdout pipe has a limited size, and the process
> will block once the pipe becomes full. You are entering a dead-lock as the
> pipe is full, and thus the sub process is blocking waiting for room to write
> more output. As nothing is reading that output, that will never happen. It
> works fine on smaller output as you don't hit the limit. See the warning on
> http://docs.python.org/library/subprocess.html#popen-objects under wait.
>
> .communicate works as it reads the output, however it eats up memory as all
> of the data has to be buffered in memory.
>
> A possible work around would be to redirect the output into a file rather
> than to a pipe (see the comments about stdin, stdout and stderr at
> http://docs.python.org/library/subprocess.html#using-the-subprocess-module).
> In this way, the data will be buffered to a file rather than memory.
>
>
>>
>> Is there a better way to get my return code consistently efficiently and
>> not take up so much memory?
>>
>>
>>
>>
>>
>>
>>
>> --
>> --- Get your facts first, then you can distort them as you please.--
>>
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>


-- 
--- Get your facts first, then you can distort them as you please.--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110222/cb83ed61/attachment-0001.html>