Thread blocked by file.read/readline

[Alex Martelli]

Neil Hodge wrote:
   ...
>         for f in file.readlines():
   ...
> Basically, the for loop here does not start processing until it has
> all of the output from the command (i.e., the pipe is closed).  Is

Definitely, since the semantic of readlines is to read ALL lines.
(Btw, don't use the name of built-in types, such as 'file', to name
your own variables -- it makes for no end of confusion).

> there any way to read items from the pipe as they are being added (i.e.,
> while the pipe is still open)?  I have some control over the output of the
> CLI, if that helps.  Thanks.

Changing the loop to "for f in file:" might help, as this will only
read a small-lish buffer before it starts looping.  That may not
suffice -- you may want to read line by line -- in which case, make
the loop start with
    while True:
        f = file.readline()
        if not f: break
        # continue as above from here
AND make sure the file is line-buffered (NOT fully buffered!) on
both ends -- in the "CLI output" you might alternatively ensure
the output is flushed after each line or small group of lines, but
you do need line buffering (or no buffering at all, but that might
be slowish -- still, do try it!) on the side reading the file.


Alex