[Python-Dev] Towards native fileevents in Python (Was Re: Python multiplexing is too hard)

[Alexandre Ferrieux]

Guido van Rossum wrote:
> 
> > From: Alexandre Ferrieux <alexandre.ferrieux@cnet.francetelecom.fr>
> >
> > I'm an oldtime Tcler, firmly decided to switch to Python, 'cause it is
> > just so beautiful inside. But while Tcl is weaker in the algorithms, it
> > is stronger in the os-wrapping library, and taught me to love high-level
> > abstractions. [fileevent] shines in this respect, and I'll miss it in
> > Python.
> 
> Alex, it's disappointing to me too!  There just isn't anything
> currently in the library to do this, and I haven't written apps that
> needs this often enough to have a good feel for what kind of
> abstraction is needed.

Thanks for the empathy. Apologies for my slight overreaction.

> However perhaps we can come up with a design for something better?  Do
> you have a suggestion here?

Yup. One easy answer is 'just copy from Tcl'...

Seriously, I'm really too new to Python to suggest the details or even
the *style* of this 'level 2 API to multiplexing'. However, I can sketch
the implementation since select() (from C or Tcl) is the one primitive I
most depend on !

Basically, as shortly mentioned before, the key problem is the
heterogeneity of seemingly-selectable things in Windoze. On unix, not
only does select() work with
all descriptor types on which it makes sense, but also the fd used by
Xlib is accessible; hence clean multiplexing even with a GUI package is
trivial. Now to the real (rotten) meat, that is M$'s. Facts:

	1. 'Handle' types are not equal. Unnames pipes are (surprise!) not
selectable. Why ? Ask a relative in Redmond...

	2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true
handles. They are selectable, but for example you can't use'em for
redirections. Okay in our case we don't care. I only mention it cause
its scary and could pop back into your face some time later.

	3. The GUI API doesn't expose a descriptor (handle), but fortunately
(though disgustingly) there is a special syscall to wait on both "the
message queue" and selectable handles: MsgWaitForMultipleObjects. So its
doable, if not beautiful.

The Tcl solution to (1.), which is the only real issue, is to have a
separate thread blockingly read 1 byte from the pipe, and then post a
message back to the main thread to awaken it (yes, ugly code to handle
that extra byte and integrate it with the buffering scheme).

In summary, why not peruse Tcl's hard-won experience on
selecting-on-windoze-pipes ?

Then, for the API exposed to the Python programmer, the Tclly exposed
one is a starter:

	fileevent $channel readable|writable callback
	...
	vwait breaker_variable

Explanation for non-Tclers: fileevent hooks the callback, vwait does a
loop of select(). The callback(s) is(are) called without breaking the
loop, unless $breaker_variable is set, at which time vwait returns.

One note about 'breaker_variable': I'm not sure I like it. I'd prefer
something based on exceptions. I don't quite understand why it's not
already this way in Tcl (which has (kindof) first-class exceptions), but
let's not repeat the mistake: let's suggest that (the equivalent of)
vwait loops forever, only to be broken out by an exception from within
one of the callbacks.

HTH,

-Alex