popen2 bug?

[Donn Cave]

Quoth Roman Suzi <rnd at onego.ru>:
| I came across unidentified difficulty when doing the following:
|
| >>> d = (("a"*120) + "\n") * 300     # prepare data
| >>> import popen2
| >>> a, b = popen2.popen2("grep a")   # pipe to/from grep setup
| >>> b.write(d)                       # trying to write... takes forever!
| Traceback (innermost last):
|   File "/usr/lib/python1.5/exceptions.py", line 102, in __init__
|     def __init__(self, *args):
| KeyboardInterrupt
|
| - I used C-C to break.
|
| I think this behavior is wrong! But I am not sure how to avoid it. It
| happens when lines, passed to grep, are incomplete and when the incomplete
| line matches.

Grep has to write what it finds to stdout, before it can read more stdin.
You want to write everything to stdin, before reading anything from stdout.
When stdout fills up, grep stops, stdin fills up and you stop.  That's my
theory.  This behavior is not what you want, but it's what you can expect.

| My idea was to pipe logs thru grep and then read separate records as
| lines.

You'll have to break the I/O up into smaller pieces, use the UNIX file
descriptors (as in b.fileno()) and posix.write/read and select to know
when you can read and when you can write, and even then I wouldn't be
to positive you won't occasionally get into trouble.  When you have it
working, do get some wall clock times to see if it's actually faster.

	Donn Cave, donn at u.washington.edu