[Python-Dev] Draft PEP to make file objects support non-blocking mode.

[Donovan Baarda]

On Mon, 2005-03-21 at 17:32 +1200, Greg Ewing wrote:
> > On 18 March 2005, Donovan Baarda said:
> 
> >>Many Python library methods and classes like select.select(), os.popen2(),
> >>and subprocess.Popen() return and/or operate on builtin file objects.
> >>However even simple applications of these methods and classes require the
> >>files to be in non-blocking mode.
> 
> I don't agree with that. There's no need to use non-blocking
> I/O when using select(), and in fact things are less confusing
> if you don't.

You would think that... and the fact that select, popen2 etc all use
file objects encourage you to think that. However, this is a trap that
can catch you out badly. Check the attached python scripts that
demonstrate the problem.

Because staller.py outputs and flushes a fragment of data smaller than
selector.py uses for its reads, the select statement is triggered, but
the corresponding read blocks.

A similar thing can happen with writes... if the child process consumes
a fragment smaller than the write buffer of the selector process, then
the select can trigger and the corresponding write can block because
there is not enough space in the file buffer.

The only ways to ensure that a select process does not block like this,
without using non-blocking mode, are;

1) use a buffer size of 1 in the select process.

2) understand the child process's read/write behaviour and adjust the
selector process accordingly... ie by making the buffer sizes just right
for the child process,

Note that it all interacts with the file objects buffer sizes too...
making for some extremely hard to debug intermittent behaviour.

> >>The read method's current behaviour needs to be documented, so its actual
> >>behaviour can be used to differentiate between an empty non-blocking read,
> >>and EOF.  This means recording that IOError(EAGAIN) is raised for an empty
> >>non-blocking read.
> 
> Isn't that unix-specific? The file object is supposed to
> provide a more or less platform-independent interface, I
> thought.

I think the fread/fwrite and read/write behaviour is posix standard and
possibly C standard stuff... so it _should_ be the same on other
platforms.

-- 
Donovan Baarda <abo at minkirri.apana.org.au>
http://minkirri.apana.org.au/~abo/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: selector.py
Type: application/x-python
Size: 756 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20050321/b70dd852/selector.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: staller.py
Type: application/x-python
Size: 126 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20050321/b70dd852/staller.bin