Doing readline in a thread from a popen4('rsync ...') stream blo cks when the stream ends.

[Rasmusson, Lars]

Hi,

I have a problem doing I/O in a python thread.

I want a thread to execute a command using 
os.popen, read its input until the end, and
then print 'DONE' and terminate.

However, the thread hangs when it reaches
the end of the stream.  

The command that I want to do is 'rsync'
which is used to transfer files between hosts.
Rsync uses ssh to transfer the data securely.

Here is a small program that demonstrates
the problem:

------START-------
#!/usr/bin/python2.3
# -*- python -*-

# This program demonstrates a case when Python locks up.
# It runs the program 'rsync' and reads its output.
# When this is done inside a thread, the program locks up
# while doing the last i.readline()

# To try the program, make sure that you can ssh into localhost
# without password prompt, and that /etc/hosts exists
#

fileToGet = '/etc/hosts'
cmd='rsync -v -e ssh localhost:%s file.txt' % fileToGet

def do():
    import os
    (o,i)=os.popen4(cmd)
    readAll(i)

def readAll(i):
    x=i.readline()
    while x:
        print x
        x=i.readline()    # this is where the program locks up
    print 'DONE'

from threading import Thread

class RunInThread(Thread):
    def __init__(self,run,daemon=True):
        Thread.__init__(self)
        self.setDaemon(daemon)
        self.run = run
        self.start()

print '\nFirst we run the command outside of a thread. This works fine.\n'

do()

print '\nNow we will run the command inside a python thread. '
print 'The program will never print DONE, nor finish.\n'

RunInThread(do,daemon=False)

-------END-------


This problem occurs with rsync, so one guess is that it
is related to the interaction with python's threads and
and rsync or ssh.

One message I google'd up mentioned that non-standard 
sighandlers could be a problem when running rsync from perl,
but I have not had any luck there.

I have also tried pexpect and pty.fork as alternatives to popen,
but with no luck.

I have the same problem on both Windows and Linux platforms.
It occurs on many versions of python.


I'd be very grateful for any suggestions/hints/solutions
to this problem!  


Thanks!
Lars