[Python-Dev] Towards native fileevents in Python (Was Re: Python multiplexing is too hard)

[Guido van Rossum]

> Yup. One easy answer is 'just copy from Tcl'...

Tcl seems to be your only frame of reference.  I think it's too early
to say that borrowing Tcl's design is right for Python.  Don't forget
that part of Tcl's design was guided by the desire for backwards
compatibility with Tcl's strong (stronger than Python I find!) Unix
background.

> Seriously, I'm really too new to Python to suggest the details or even
> the *style* of this 'level 2 API to multiplexing'. However, I can sketch
> the implementation since select() (from C or Tcl) is the one primitive I
> most depend on !
> 
> Basically, as shortly mentioned before, the key problem is the
> heterogeneity of seemingly-selectable things in Windoze. On unix, not
> only does select() work with
> all descriptor types on which it makes sense, but also the fd used by
> Xlib is accessible; hence clean multiplexing even with a GUI package is
> trivial. Now to the real (rotten) meat, that is M$'s. Facts:

Note that on Windows, select() is part of SOCKLIB, which explains why
it only understands sockets.  Native Windows code uses the
wait-for-event primitives that you are describing, and these are
powerful enough to wait on named pipes, sockets, and GUI events.
Complaining about the select interface on Windows isn't quite fair.

> 	1. 'Handle' types are not equal. Unnames pipes are (surprise!) not
> selectable. Why ? Ask a relative in Redmond...

Can we cut the name-calling?

> 	2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true
> handles. They are selectable, but for example you can't use'em for
> redirections. Okay in our case we don't care. I only mention it cause
> its scary and could pop back into your face some time later.

Handles are a much more low-level concept than file descriptors.  get
used to it.

> 	3. The GUI API doesn't expose a descriptor (handle), but fortunately
> (though disgustingly) there is a special syscall to wait on both "the
> message queue" and selectable handles: MsgWaitForMultipleObjects. So its
> doable, if not beautiful.
> 
> The Tcl solution to (1.), which is the only real issue,

Why is (1) the only issue?  Maybe in Tcl-land...

> is to have a
> separate thread blockingly read 1 byte from the pipe, and then post a
> message back to the main thread to awaken it (yes, ugly code to handle
> that extra byte and integrate it with the buffering scheme).

Or the exposed API could deal with this in a different way.

> In summary, why not peruse Tcl's hard-won experience on
> selecting-on-windoze-pipes ?

Because it's designed for Tcl.

> Then, for the API exposed to the Python programmer, the Tclly exposed
> one is a starter:
> 
> 	fileevent $channel readable|writable callback
> 	...
> 	vwait breaker_variable
> 
> Explanation for non-Tclers: fileevent hooks the callback, vwait does a
> loop of select(). The callback(s) is(are) called without breaking the
> loop, unless $breaker_variable is set, at which time vwait returns.

Sorry, you've lost me here.  Fortunately there's more info at
http://dev.scriptics.com/man/tcl8.3/TclCmd/fileevent.htm.  It looks
very complicated, and I'm not sure why you rejected my earlier
suggestion to use threads outright as "too complicated".  After
reading that man page, threads seem easy compared to the caution one
has to exert when using non-blocking I/O.

> One note about 'breaker_variable': I'm not sure I like it. I'd prefer
> something based on exceptions. I don't quite understand why it's not
> already this way in Tcl (which has (kindof) first-class exceptions), but
> let's not repeat the mistake: let's suggest that (the equivalent of)
> vwait loops forever, only to be broken out by an exception from within
> one of the callbacks.

Vwait seems to be part of the Tcl event model.  Maybe we would need to
think about an event model for Python?  On the other hand, Python is
at the mercy of the event model of whatever GUI package it is using --
which could be Tk, or wxWindows, or Gtk, or native Windows, or native
MacOS, or any of a number of other event models.

Perhaps this is an issue that each GUI package available to Python
will have to deal with separately...

--Guido van Rossum (home page: http://www.python.org/~guido/)