read and readline hanging

[Thomas Bellman]

Olivier Lefevre <lefevrol at yahoo.com> wrote:

>> 1. The subprocess has stopped producing output.

> Indeed, if I do this interactively, I can tell after 3 lines that I've
> gotten all there is to get right now and the fourth readline() call
> hangs.

Can you really?  How do you know if the program has finished or
if it is just taking a very long time to produce the next line in
its response?  Unless there is some way to differentiate between
the last line all the other lines of a response, you can't really
be sure.

> But how can I find out *programmatically* that there is no more
> input?

It is possible to check if there is something more to read at the
moment, but you can't check if the subprocess will produce more
to read in the future.

To check if there is something to read at this very moment, you
can use any of the following methods:

  - select.select()
  - the FIONREAD ioctl (the ioctl() function lives in the fcntl
    module, and the FIONREAD constant is in the termios module)
  - set the underlying file descriptor in non-blocking mode:
        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
        fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NDELAY)
    After that, reads on the pipe will raise an IOError exception
    with the errorcode EWOULDBLOCK.
  - start a thread that does blocking reads from the pipe, and
    puts the chunks it reads on a queue for your main thread to
    grab.

For the last approach, you might be interrested in my asyncproc
module, which does exactly that.  You can download it from
<http://www.lysator.liu.se/~bellman/download/asyncproc.py>.

However, none of these approaches absolves you from the necessity
of knowing when one response ends.  You still need to solve that
problem.

The proper way is to define a protocol between your program and
the subprocess, in which you can clearly tell when you have
reached the end of a response.  Then you need to get the program
you are calling to adher to that protocol, of course...

The SMTP protocol is a good example of how this can look.  In
SMTP, each response to a command consists of a number of lines.
Each line has a three-digit response code, an "end of response"
flag, and a text message.  The "end of response" flag is a space
(" ") for the last line in the response, and a dash ("-") for all
the other lines.  The response to an EHLO command can look like
this:

    250-sellafield Hello localhost [127.0.0.1], pleased to meet you
    250-ENHANCEDSTATUSCODES
    250-PIPELINING
    250-EXPN
    250-VERB
    250-8BITMIME
    250-SIZE
    250-DSN
    250-ETRN
    250-DELIVERBY
    250 HELP

Since there is a space instead of a dash after the "250" code in
the last line above, the SMTP client knows that there won't be
any more lines in response to its command.

If you can't get the program you are calling to follow some
protocol like this, then you can only make guesses.  Sometimes
you can make fairly good guesses, and sometimes it will be more
or less impossible...


-- 
Thomas Bellman,   Lysator Computer Club,   Linköping University,  Sweden
"Life IS pain, highness.  Anyone who tells   !  bellman @ lysator.liu.se
 differently is selling something."          !  Make Love -- Nicht Wahr!