[Python-Dev] Draft PEP to make file objects support non-blocking mode.

[Donovan Baarda]

On Fri, 2005-03-18 at 20:41 -0500, James Y Knight wrote:
> On Mar 18, 2005, at 8:19 PM, Greg Ward wrote:
> > Is having to use fcntl and os really so awful?  At least it requires
> > the programmer to prove he knows what he's doing putting this file
> > into non-blocking mode, and that he really wants to do it.  ;-)

Consider the following. This is pretty much the only way you can use
popen2 reliably without knowing specific behaviours of the executed
command;

import os,fnctl,select

def process_data(cmd,data):
  child_in, child_out = os.popen2(cmd)
  child_in = child_in.fileno()				      # /
  flags = fcntl.fcntl(child_in, fcntl.F_GETFL)		      # |1)
  fcntl.fcntl(child_in, fcntl.F_SETFL, flags | os.O_NONBLOCK) # \
  child_out = child_out.fileno()			      # /
  flags = fcntl.fcntl(child_out, fcntl.F_GETFL)		      # |2)
  fcntl.fcntl(child_out, fcntl.F_SETFL, flags | os.O_NONBLOCK)# \
  ans = ""
  li = [child_out]
  lo = [child_in]
  while li or lo:
    i,o,e = select.select(li,lo,[])		# 3
    if i:
      buf = os.read(child_out,2048)		# 4
      if buf:
        ans += buf
      else:
        li=[]
    if o:
      if data:
        count=os.write(child_in,data[:2048])   # 4
        data = data[count:]
      else:
        lo=[]
 return ans

For 1) and 2), note that popen2 returns file objects, but as they cannot
be reliably used as file objects, we ignore them and grab their
fileno(). Why does popen2 return file objects if they cannot reliably be
used? The flags get/set using fnctl is arcane stuff for what is pretty
much essential operations after a popen2. These could be replaced by;

 child_in.setblocking(False)
 child_out.setblocking(False)

For 3), select() can operate on file objects directly. However, since
you cannot reliably read/write file objects in non-blocking mode, we use
the fileno's. Why can select operate with file objects if file objects
cannot be reliably read/written?

For 4), we are using the os.read/write methods to operate on the
fileno's. Under the proposed PEP we could use the file objects
read/write methods instead.

I guess the thing that annoys me the most is the asymmetry of popen2 and
select using file objects, but needing to use the os.read/write and
fileno()'s for reading and writing.

> I'd tend to agree. :) Moreover, I don't think fread/fwrite are 
> guaranteed to work as you would expect with non-blocking file 
> descriptors. So, providing a setblocking() call to files would require 
> calling read/write instead of fread/fwrite in all the file methods, at 
> least when in non-blocking mode. I don't think that's a good idea.

Hmm.. I assumed file.read() and file.write() were implemented using
read/write from their observed behaviour. The documentation of
fread/fwrite doesn't mention the behaviour in non-blocking mode at all.
The observed behaviour suggests that fread/fwrite are implemented using
read/write and hence get the same behaviour. The documentation implies
that the behaviour in non-blocking mode will reflect the behaviour of
read/write, with EAGAIN errors reported via ferror() indicating empty
non-blocking reads/writes.

If the behaviour of fread/fwrite is indeed indeterminate under
non-blocking mode, then yes, file objects in non-blocking mode would
have to use read/write instead of fread/fwrite. However, I don't think
this is required.

I know this PEP is kinda insignificant and minor. It doesn't save much,
but it doesn't change much, and makes things a bit cleaner.

-- 
Donovan Baarda <abo at minkirri.apana.org.au>
http://minkirri.apana.org.au/~abo/