subprocess pipe question

[Rob Williscroft]

Rita wrote in
news:AANLkTi=88dCPM_kqRs2g620OBsNXz0maJUbfWPEmEtnL at mail.gmail.com in
gmane.comp.python.general: 

[Top post relocated]

> On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft <rtw at rtw.me.uk>
> wrote: 
> 
>> Rita wrote in
>> news:AANLkTi=w95gXoSc1TKt2BntGjqys1cbmDnoJHOKQ4YBA at mail.gmail.com in
>> gmane.comp.python.general:
>>
>> >
>> > When using wait() it works a bit better but not consistent
>> > def run(cmd):
>> >   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
>> >   rc=p.wait()
>> >   print rc
>> >   return p.stdout
>> >
>>
>> > When the output of cmd is a small ascii file it works perfectly
>> > fine, but when the file is large (more than 2MB) the process just
>> > waits for ever (I am guessing its blocking?).
>>
>> Your OS has supplied a pipe buffer of 2MB, its full and the prossess
>> is waiting until you read something from the pipe (i.e.
>> p.stdout.read()). 
>>
>> >            When I use the communicate call
>> > it works perfectly but my process is consuming way too much memory.
>> >
>> > Is there a better way to get my return code consistently
>> > efficiently and not take up so much memory?
>>
>> If you don't need the processes output then don't use the PIPE
>> argument. If you do you will need to read from p.stdout until the
>> process is complete, then get the return code:
>>
>> while True:
>>        buf = p.stdout.read( 1024 )
>>        # do somthing with buf
>>        if len( buf ) < 1024:
>>                break
>>
>> rc = p.wait()

> 
> For extra points is there a way to speed up the p.stdout.read(bufsize)
> ? 

Its unlikely that is the problem, most likely you are reading all of the 
2MB OS pipe buffer and then having to wait for the programme you are 
calling to produce some more output.

Before trying to speed things up be sure to test with real work being 
done to the output you recieve.

If its still slow and you have multiple cores/processors then you will 
need to put you read loop in a thread to speed it up.

-- 
Rob.