Unbuffer stdout?

[Donn Cave]

Quoth "Francis Avila" <franga at shentel.net>:
| I'm trying to run the stdout of an os.popen*() to sys.stdout.  Problem is
| that the output seems to get trapped in a buffer until the command exits.
|
| Code:
|
| >>> import os, sys
| >>> sys.stdout.write( os.popen("yes", 'r', 0).readline() )
| yes: standard output: Broken pipe
| y
| >>>
|
| I assume the broken pipe is from having the pipe closed before 'yes' exits.
| Strangely enough, the popen[2-4]() variants don't have this problem:
|
| >>> sys.stdout.write( os.popen2("yes", 'r', 0)[1].readline() )
| y
| >>>

It would, if the pipe ever closed, but the above doesn't make that
happen, I imagine because some internal references keep the file
object from being deleted.  By the way, I don't think your parameters
to popen2() are right.

| Anyway, this:
|
| >>> sys.stdout.write( os.popen("yes", 'r', 0).read() )
|
| hangs forever (with some disk activity after a while) until you interrupt
| the interpreter.  I strongly suspect this is because the output is getting
| trapped in a buffer until the command exits (which is what I want to avoid)
| because the following works fine:
|
| >>> sys.stdout.write( os.popen("ls", 'r', 0).read() )
| directoryOne
| directoryTwo
| someRandomFile
| >>>
|
| This:
|
| >>> sys.stderr.write( os.popen("yes", 'r', 0).read() )
|
| does the same, which seems odd to me because stderr is never buffered, at
| least not by the OS.  In any case, even _this_:
|
| #!/bin/sh
| yes | more
|
| does give you a screen of y's as soon as 'yes' fills the buffer, but before
| 'yes' exits.  So python is somewhere doing extra buffering of the whole
| output of popen().

Sort of.  What do you think read() is supposed to do?  Check it out:

  >>> print sys.stdin.read.__doc__
  read([size]) -> read at most size bytes, returned as a string.

  If the size argument is negative or omitted, read until EOF is reached.

| What I'm looking for is to pipe the stdout of a child process directly to
| the stdout of the parent process (i.e., the python script), without any
| buffering along the pipeline.  As I see it, this is either not possible in
| python, or I'm going about this the wrong way.

Well, it really is not possible to control this where it counts, which
is on the child process end.  It will work only when the child process
takes it upon itself to flush its output buffer;  some applications,
notable the shell, already do this, but most don't.

If you don't want to read a line at a time but just whatever input is
available, then I think it makes more sense (as usual) to forget the
file object and use the pipe file descriptor directly --
  fp = os.popen('yes')
  fd = fp.fileno()
  while 1:
      data = os.read(fd, 16000)
      if data:
          os.write(1, data)
      else:
          break
  fp.close()

| BTW, I was trying to look at the code of popen(), but I couldn't find it
| anywhere in the python library, much less in os.py. What's more:
|
| >>> os.popen
| <built-in function popen>
|
| But if it's a built-in, why do I have to import os?

Because popen is in the os module?  Built-in basically means that the
os module was written in C.  popen is a C library function, so you
might find some documentation for it with "man 3 popen".

	Donn Cave, donn at drizzle.com