Non blocking IO

[Donn Cave]

Quoth Graeme Winter <g.winter at dl.ac.uk>:
...
| I have rewritten the example entirely in Python - and I still get the 
| same effect:
|
| import popen2, select, sys, os, fcntl
|
| def blocking_test(timeout = 5):
|      pipe = popen2.Popen3("./test_blocking_executable", 0, 0)
|      stdin = pipe.tochild
|      stdout = pipe.fromchild
|
|      stdin.write("10\n")
|      stdin.close()
|
|      fcntl.fcntl(stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
|      # read the output
|      while 1:
|          channel = select.select([], [stdout], [], timeout)
|          if channel[1] == []:
|              # we have timed out
|              raise RuntimeError, "Timed out"
|          else:
|              line = stdout.readline()
|              if not line:
|                  break
|              sys.stdout.write(">" + line)
|              sys.stdout.flush()
|
|      stdout.close()
|
| if __name__ == "__main__":
|      blocking_test(5)
|
| (get the same error message when I am finished)
|
| So am I right to be thinking that readline() can't cope with non 
| blocking IO? If so, is it worth writing something to handle this?

I don't ever use non-blocking I/O, so I don't know right off hand
how it interacts with readline() but it doesn't sound like something
that would end well.  But I think you've misinterpreted the documentation
for select, and gotten input and output mixed up in the parameters -
when you're waiting for data, write select([file], [], [])  It may
be another process' output, but to select() it's input.

I sometimes use the file objects returned from popen2 et al., but
only when I know for certain that I can afford to block until the
other process has written all its output and exited.  When that
isn't the case, the file object's buffering (per C stdio) is more
of a problem than a solution, and it's better to read directly from
the pipe file descriptor with posix.read() (a.k.a. os.read.)  As in,

    p0 = proc.fromchild.fileno()
    rem = ''
    inl.append(p0)
    while inl:
        inf, ouf, erf = select.select(inl, [], [])
        if p0 in inf:
            data = os.read(p0, 16384)
            if not data:
                rmfd(p0, inl)
            ldata = data.split('\n')
            ldata[0] = rem + ldata[0]
            rem = ldata[-1]
            del ldata[-1]
            processlines(ldata)

(Untested.)  You get no readline function this way, and must split
out your own lines, but it avoids confusion between data ready on
the pipe device, which select can see, and data ready in the file
object buffer, which it cannot.

	Donn Cave, donn at drizzle.com