subprocess and non-blocking IO (again)

[Nick Craig-Wood]

Marc Carter <mcarter at uk.ibm.com> wrote:
>  import subprocess,select,sys
> 
>  speakers=[]
>  lProc=[]
> 
>  for machine in ['box1','box2','box3']:
>       p = subprocess.Popen( ('echo '+machine+';sleep 2;echo goodbye;sleep 
>  2;echo cruel;sleep 2;echo world'), stdout=subprocess.PIPE, 
>  stderr=subprocess.STDOUT, stdin=None, universal_newlines=True )
>       lProc.append( p )
>       speakers.append( p.stdout )
> 
>  while speakers:
>       speaking = select.select( speakers, [], [], 1000 )[0]
>       for speaker in speaking:
>           speech = speaker.readlines()
>           if speech:
>               for sentence in speech:
>                   print sentence.rstrip('\n')
>                   sys.stdout.flush() # sanity check
>           else: # EOF
>               speakers.remove( speaker )
>  - SNIP ---------
>  The problem with the above is that the subprocess buffers all its output 
>  when used like this and, hence, this automation is not informing me of 
>  much :)

The problem with the above is that you are calling speaker.readlines()
which waits for all the output.

If you replace that with speaker.readline() or speaker.read(1) you'll
see that subprocess hasn't given you a buffered pipe after all!

In fact you'll get partial reads of each line - you'll have to wait
for a newline before processing the result, eg

import subprocess,select,sys

speakers=[]
lProc=[]

for machine in ['box1','box2','box3']:
     p = subprocess.Popen( ('echo '+machine+';sleep 2;echo goodbye;sleep 2;echo cruel;sleep 2;echo world'), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=None, universal_newlines=True, shell=True)
     lProc.append( p )
     speakers.append( p.stdout )

while speakers:
     speaking = select.select( speakers, [], [], 1000 )[0]
     for speaker in speaking:
         speech = speaker.readline()
         if speech:
             for sentence in speech:
                 print sentence.rstrip('\n')
                 sys.stdout.flush() # sanity check
         else: # EOF
             speakers.remove( speaker )

gives

b
o
x
1

b
o
x
3

b
o
x
2

pause...

g
o
o
d
b
y
e

etc...

I'm not sure why readline only returns 1 character - the pipe returned
by subprocess really does seem to be only 1 character deep which seems
a little inefficient! Changing bufsize to the Popen call doesn't seem
to affect it.
-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick