pipe problem on FreeBSD....

[Steve Spicklemire]

Hi Folks,

   Hmmm.. what I thought started out as a "Zope Bug" isn't. Let me 
forgo all the history here and cut to the current analysis. There
seems to be a problem on FreeBSD with python sending large amounts of 
data through a pipe that blocks. I saw it first in Zope when the
DC folks changed their File object to break large files into chunks
and started sending them one chunk at a time rather than as one
big chunk. Here is the reference to that problem report:

http://lists.zope.org/pipermail/zope/1999-December/015874.html

I was oddly dissapointed when I found that in fact this was only 
a problem on FreeBSD (2.2.8, and 3.2 both exhibit the problem.)
I think I've now simplified it to it's most basic form:

This python script:

tryit.py:
----------------------------------------------------------------------
#!/usr/local/bin/python

import sys

f = open('blah')

while(1):
    s = f.read(1024)
    if not s:
        break

    sys.stdout.write(s)

f.close()
----------------------------------------------------------------------

fails on FreeBSD when invoked thusly:

pluto.spvi.com> python tryit.py | head -3
(83d0"@eKBdP5!!!!-m4F"e&*2!GP8I3!!!!!!!!!!!GD,D#@R`!!!!!!!!!!!!!
!!!!!!!!!!!!+Mj!!!!!!!2rrrrp"8&"-8(PdB5%!Y(r$aE4r`mF!#5KQ!"-ic`!
#f3d!"lAp(RrP2!!!!!!!!,iF"X"9[CqGcE0IZ3fdcPNc0pZq!aYXRD4GeXSrbYF
Traceback (innermost last):
  File "tryit.py", line 12, in ?
    sys.stdout.write(s)
IOError: [Errno 32] Broken pipe

where this c program does not:

tryit.c
----------------------------------------------------------------------

#include <stdio.h>

FILE *f;
size_t charCount;
int moreStuff;
char buffer[1024];

int main() {
  f = fopen("blah","r");
  if (f) {

    moreStuff = 1;

    while(moreStuff) {
      charCount = fread(buffer, 1, 1024, f);
      write(fileno(stdout), buffer, charCount);
      if (charCount < 1024) {
        moreStuff = 0;
      }
    }
    fclose(f);
  }
  else
    {
      printf("Can't open file... blah\n");
    }
}

----------------------------------------------------------------------

pluto.spvi.com> ./tryit | head -3
(83d0"@eKBdP5!!!!-m4F"e&*2!GP8I3!!!!!!!!!!!GD,D#@R`!!!!!!!!!!!!!
!!!!!!!!!!!!+Mj!!!!!!!2rrrrp"8&"-8(PdB5%!Y(r$aE4r`mF!#5KQ!"-ic`!
#f3d!"lAp(RrP2!!!!!!!!,iF"X"9[CqGcE0IZ3fdcPNc0pZq!aYXRD4GeXSrbYF
pluto.spvi.com>


----------------------------------------------------------------------

My thinking is that they should behave in the same way. (This also
happens with 'more' and other such blocking programs.) Where in python
should I look for the difference between tryit.c and tryit.py? Of
course on linux both programs *do* behave the same way... I just
created the 'c' version to see if it was deep in FreeBSD or somewhere
in the python port. The python and c version work correctly in linux.
<dang!>

thanks,
-steve