popen2 with large input

Jeff Epler jepler at unpythonic.net
Thu Jan 29 09:14:49 EST 2004


The connection to the child process created by the popen family have
some inherent maximum size for data "in flight".  I'm not sure how to
find out what that value is, but it might be anywhere from a few bytes
to a few K.

So tr starts to write its output as it gets input, but you won't read
its output before you've written all your output.  If the size of tr's
output is bigger than the size of the buffer for tr's unread output,
you'll deadlock.

As an aside, the particular problem you pose can be solved with Python's
str.translate method.  If the actual goal is to "work like tr", then use
that instead and forget about popen.

Anyway, to solve the popen2 problem, you'll need to write something like this:
[untested, and as you can see there's lots of pseudocode]
    def getoutput( command, input ):
        r, w = popen2(command)
        rr = [r]; ww = [w]
        output = []
        set r and w nonblocking
        while 1:
            _r, _w, _ = select.select(rr, ww, [], 0)

            if _w:
                write some stuff from input to w
                if nothing left:
                    w.close(); ww = []
            if _r:
                read some stuff into output
                if nothing to read:
                    handle the fact that r was closed
                    if w was closed: break
                    else: probably an error condition
        return "".join(output)

You could also write 'input' into a temporary file and use
commands.getoutput() or os.popen(.., "r").

Jeff




More information about the Python-list mailing list