[Web-SIG] Returned application object and fileno.

[Phillip J. Eby]

At 03:50 PM 9/1/04 +0100, Alan Kennedy wrote:

>But there is no special method name or attribute on file-like objects that 
>I can look for: file methods such as read() are the only options. Jython 
>file objects have an identical interface to cpython file objects, except 
>that they don't have fileno() methods.

"File-like" is a complete red herring: the spec has never supported them 
(and IMO never will).

What the spec calls for is an *iterable*: an object that can be used in a 
"for" loop.  In Python 2.2 and up, file objects are iterable.  In older 
versions of Python, they are not.

Thus, an application that returns a file object implicitly requires Python 
2.2 or up.  (This issue is mentioned in the spec, where it warns that if 
you are using an older version of Python, you may *not* return a file object.)

WSGI does not support returning files or file-like objects: it is simply an 
artifact of Python 2.2 and up that returning a file works at all!

Appealing to the 'fileno()' case as supporting file-like objects is also a 
red herring: the object must *still* be an iterable, because not every 
server or gateway will support 'fileno()'.  Thus, code that relies on an 
object that has a 'fileno()' but isn't iterable, is in violation of the 
spec and is inherently non-portable.

But, because file objects are iterable in 2.2 or up, if the application 
doesn't care about older versions, it is free to return file objects.

The fact that you would like such code to run in a Jython 2.1 server 
doesn't mean that the spec should expand its scope to cover even *file 
objects*, let alone "file-like" objects.  It simply means that you'll have 
to deal with the special cases that entails, until Jython 2.2 is ready for 
prime-time.


>1. I see nothing 2.2 specific in my above code sample: it works on all 
>pythons. I don't see what differs between 2.1 vs. 2.2 in this case.

2.1 doesn't allow iteration over file objects.


>2. The spec, as is, explicitly permits authors of cpython applications to 
>return file-like objects,

Only if they are *iterable*, which is only true of the 'file' object in 2.2 
and up.


>due to the cpython-specific special case "your application object may have 
>a fileno()".

You misunderstand: in CPython 2.1 returning a file is *not* acceptable 
under the spec.  It is purely coincidental that it will happen to work if 
the server checks for 'fileno()' and supports doing something with it.  But 
it's not portable behavior for 2.1 and the spec has said that as soon as 
the "Supporting Older Versions" section was added.


>Of course, most application authors won't know that the reason why their 
>file return is succeeding is because the file object has a fileno() 
>method, and then wonder why their app doesn't work on jython.

You're effectively arguing for removing the 'fileno()' special case 
altogether, or else adding language to require the server to *first* check 
for iterability and raise an error if the return isn't iterable, so that 
running a 2.2 app in a 2.1 server won't "accidentally" succeed when the 2.1 
server supports 'fileno()'.

However, this is such an obscure use case as to be ludicrous to worry 
about.  So far, yours is the only server that has suggested that supporting 
2.2 apps under Python 2.1 is anything even approaching a good idea.  I find 
it hard to imagine any reason to do that, other than the lack of 
availability of a Python 2.2 implementation.  Other than Jython, I'm not 
aware of any other platforms where this is the case.

I applaud your bravery in trying to make it work for Jython, but changing 
the spec to allow other kinds of objects isn't going to decrease the amount 
of work you have to do, only increase it for other people who *aren't* 
trying to support 2.2 apps in a server running under Python 2.1.


> > Don't let the spec stop you from supporting your use case.  The problem
> > is simply that your use case is outside the spec's scope, and I don't
> > want to expand the spec's scope to make *everybody else* have to
> > implement the extras you're implementing.  I don't want to force
> > everybody else to try to support 2.2 features in a 2.1 Python.
>
>Sorry, Phillip, I'm confused. I don't see that this has anything to do 
>with 2.1 vs. 2.2: it's got to do with how to recognise the case where the 
>application returns a file-like object, which can then be treated 
>specially, e.g. for high-performance reasons.

It's not about "file-like" objects, only *actual* file objects.  Returning 
a "file-like" object offers no meaningful performance boost, and it is 
*not* supported -- and never was.


>I think we should explicitly allow return of a file-like object, and thus 
>freedom to use the read() method, etc.

I disagree.  In 2.2, you can return a file-like object thus:

     return iter(lambda: filelike.read(bufsize), "")

In 2.1 and prior, you can do this:

     class Reader:
         def __init__(self,filelike,bufsize=4096):
             self.stream = filelike
             self.bufsize = bufsize
             if hasattr(filelike,'fileno'):
                 self.fileno = filelike.fileno

         def __getitem__(self,ind):
             data = self.stream.read(self.bufsize)
             if data:
                 return data
             raise IndexError

     return Reader(filelike)

or even:

    return xreadlines.xreadlines(filelike)

Any of these approaches results in a spec-compliant iterable for the 
applicable or higher version of Python.

You are trying to let 2.2 code run in a 2.1 Python.  But you don't need to 
support "file-like" objects to do that.  You need only special case for an 
*actual* file object, because such an object *would* be iterable under 
2.2.  The issue there isn't high-performance, it's merely that file objects 
are unacceptable return values in Python 2.1, but code written for 2.2. 
will expect that returning a file object is valid.

There are other objects that technically would need to be special-cased for 
this.  For example, a dictionary object is iterable in 2.2 but not in 
2.1.  In practice, it would be silly to bother since nobody in their right 
mind is going to use a dictionary as a WSGI return value...  unless of 
course it had only one key.

The point is that trying to run 2.2 code in a 2.1 Python is necessarily a 
collection of special case hacks.  The spec calls for *iterability*, and 
2.2 code may return objects of built-in types that are iterable in 2.2, but 
not in 2.1.

That is why this is a Python versioning issue, and specific to your attempt 
to run 2.2 code in a 2.1 Python.  It has absolutely nothing to do with 
accepting "file-like" objects in the spec, which never accepted them, nor 
is it intended to ever do so.

Is this getting any clearer?