[Web-SIG] Reviewing WSGI open issues, again...

[Alan Kennedy]

[Phillip J. Eby]
 > Java doesn't have file descriptors, so
 > you're not going to be able to use 'sendfile()' in Java.  So, what's
 > the point of having 'fd_wrapper' available there?

and

 > An object that works with 'fd' isn't going to work with 'nio', or vice
 > versa is it?  Or am I missing something about how nio works?
 >
 > I suppose the alternative is to specify 'wsgi.file_wrapper' such that
 > it's required to always return *something* usable, even if it can't
 > figure out any way to optimize it.  Objects passed to 'file_wrapper'
 > would have to have a 'read', optionally a 'close', and optionally
 > 'fileno'.  (A Jython WSGI server would ignore fileno, of course.)

Ah, I think I see where the confusion lies. Perhaps I should have taken 
more time to explain a certain issue earlier than this.

Jython files *may* have the local analogue of a file descriptor, i.e. a 
channel, but only when the jython code is running on a jvm that supports 
java.nio, which means 1.4 or greater.

I could define the fileno method of jython files like this

class file:

     def fileno(self):
         if hasattr(self.java_file, 'getChannel'):
              # java >= 1.4 behaviour
              return self.java_file.getChannel()
         else:
              # java < 1.4 behaviour
              raise UnimplementedException()

The current jython 2.1 library only raises the exception, because 
java.nio didn't exist when it was written.

Now, I could suggest a patch to the jython runtime to redefine fileno as 
above, but that's not a safe thing to do: existing python code that is 
expecting a cpython file descriptor will almost certainly break if it 
gets passed a java.nio.channels.FileChannel instead.

Unless the entire I/O subsystem has been rewritten, as I am doing for 
jynio sockets, which *do* have a useful fileno() method which each of 
the new modules knows how to use properly. The returned object confers 
identical semantics to cpython file descriptors, when passed to jynio 
socket modules. For example, when jynio is finished, this code will run 
identically on cpython and jython

s = socket.socket(AF_INET, SOCK_STREAM)
fd = s.fileno()
po = select.poll()
po.register(fd)

http://www.xhaus.com/alan/python/jynio/socket.html#socketvschannel

A very similar set of operations can be carried out on both file 
descriptors and channels: i.e. selectability/event notification, bulk 
transfers, etc.

http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SelectableChannel.html
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/InterruptibleChannel.html
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/WritableByteChannel.html
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/GatheringByteChannel.html

So, when running on java 1.4+, I *can* get the local equivalent of a 
file descriptor, and do meaningful things with it, in terms of bulk 
transfer, etc.

But that doesn't work on older JVMs where only java.io is available: 
There is no other way (without examining private file object data, i.e. 
the java.io.FileInputStream encapsulated in the jython file) that I can 
determine if something is file-like, other than to do this

if type(app_object) is types.FileType:
     do_high_performance_file_stuff(app_object)

That's why I pushed for permitting return of file-likes from 
applications: because it's the only "safe" way to recognise files on 
pre-1.4 jvms. And it's also portable to all other python platforms.

It would still definitely be useful to recognise the optimisation on 
older VMs, because I could still have a fast native-java 
loop-while-sending-blocks implementation of "sendfile", which would be 
substantially faster than a jython one, because it would avoid the 
unnecessary transformation of the file data into jython data structures 
(i.e. binary strings) and then back again.

But your solution of a single server-provided file_wrapper class solves 
the problem nicely. Because the application has hinted that the 
application object is a file, I now have a simple way of checking, that 
works across all jvms. So I can now very simply provide the bulk 
transfer optimisation, and implement it differently, depending on the 
availability of the java.nio classes, e.g.

try:
     import java.nio

     class file_wrapper:

         def send_file(self, dest)
             use_nio_transfer_to(dest)

except ImportError:
     import java.io

     class file_wrapper:

         def send_file(self, dest)
             use_looping_sendfile(dest)

Also, the 'file_wrapper' solution alleviates the need for me look at 
private data inside jython file objects, to see if the underlying 
java.io.FileInputStream has a getChannel method. So it's definitely the 
cleanest solution.

As for the 'file_wrapper' class name across platforms, as you can see 
from the above, having different class names for each platform would not 
change the above considerations one bit: it would just make the 
application authors life more difficult.

I hope that makes the situation clearer!

Regards,

Alan.