Thread blocked by file.read/readline

[Neil Hodge]

All:

I have the following code (more or less):

########################################
#!/usr/bin/env python

import threading
import Queue
import os
import string
import sys

class Program:
    def start_thread(self):
        print 'creating queue object'
        self.q = Queue.Queue(2)
        print "creating thread"
        thread = threading.Thread(name="thread1", target=self.work)
        print "starting thread"
        thread.start()

    def work(self):
        print 'beginning work'
        command = './cli.py'
        print 'beginning os.popen'
        file = os.popen(command, 'r')
        print file
        for f in file.readlines():
            s = string.strip(f)
            print 'put in queue: %s' % s
            self.q.put(f, 1)

    def get_output(self):
        print 'beginning get_output'
        while 1:
            s = self.q.get()
            s = string.strip(s)
            print '               get from queue: %s' % s
            if float(s) == 1.0:
                return


if __name__ == "__main__":
    p = Program()
    p.start_thread()
    p.get_output()

########################################

Here is cli.py.

########################################

#!/usr/bin/env python

import time

i = 0.0
while i < 1.01:
    print i
    i = i + 0.01
    time.sleep(0.01*10)

########################################

cli.py emulates a CLI progress bar that I need to grab and display
interactively.  My output is as follows:

########################################
[nhodge at zeus py_thread_test]$ ./program.py
creating queue object
creating thread
starting thread
beginning get_output
beginning work
beginning os.popen
<open file './cli.py', mode 'r' at 0x8194f28>
put in queue: 0.0
put in queue: 0.01
put in queue: 0.02
               get from queue: 0.0
               get from queue: 0.01
put in queue: 0.03
put in queue: 0.04
               get from queue: 0.02
               get from queue: 0.03
put in queue: 0.05
put in queue: 0.06
               get from queue: 0.04
               get from queue: 0.05
put in queue: 0.07
put in queue: 0.08
               get from queue: 0.06
               get from queue: 0.07
put in queue: 0.09
put in queue: 0.1
               get from queue: 0.08
               get from queue: 0.09
########################################

etc.  My problem is here:

   def work(self):
        print 'beginning work'
        command = './cli.py'
        print 'beginning os.popen'
        file = os.popen(command, 'r')
        print file
        for f in file.readlines():
            s = string.strip(f)
            print 'put in queue: %s' % s
            self.q.put(f, 1)

Basically, the for loop here does not start processing until it has
all of the output from the command (i.e., the pipe is closed).  Is
there any way to read items from the pipe as they are being added (i.e.,
while the pipe is still open)?  I have some control over the output of the
CLI, if that helps.  Thanks.

Neil