[Python-Dev] Trial balloon: microthreads library in stdlib

[Andrew Dalke]

On 2/14/07, glyph at divmod.com <glyph at divmod.com> wrote
in response to richard.m.tew at gmail.com:
> As far as I can tell, you still haven't even clearly expressed what your
> needs are, let alone whether or not Twisted is suitable.  In the reply
> you're citing, you said that "this" sounded like something "low level" that
> "twisted would be written on top of" - but the "this" you were talking
> about, based on your previous messages, sounded like monkeypatching the
> socket and asyncore modules to provide asynchronous file I/O based on the
> platform-specific IOCP API for Windows.

I don't know Richard's needs nor requirements.  I do know the mechanism
of the monkeypatch he's talking about.  I describe it a bit in my draft
for my Stackless talk at PyCon
  http://www.stackless.com/pipermail/stackless/2007-February/002212.html

It uses asyncore for the I/O and a scheduler which can figure out if
there are other running tasklets and how long is needed until the next
tasklet needs to wake up.

Yes, this is another reactor.  Yes, it's not widely cross-platform.
Yes, it doesn't work with gtk and other GUI frameworks.  Yes,
as written it doesn't handle threads.

But also yes, it's a different style of writing reactors because of
how Stackless changes the control flow.

I and others would like to see something like the "stacklesssocket"
implemented on top of Twisted. Andrew Francis is looking in to it
but I don't know to what degree of success he's had.  Err, looking
through the email archive, he's has had some difficulties in doing
a Twisted/Stackless integration.  I don't think Twisted people
understand Stackless well enough (nor obviously he Twisted) or
what he's trying to do.

> >It is a large dependency and it is a secondary framework.
>
> Has it occurred to you that it is a "large" dependency not because we like
> making bloated and redundant code, but because it is doing something that is
> actually complex and involved?

Things like Twisted's support for NNTP, POP3, etc. aren't needed
with the monkeypatch approach because the standard Python
libraries will work, with Stackless and the underlying asyc library
collaborating under the covers.  So those parts of Twisted aren't
needed or relevant.

Twisted is, after all, many things.


> I thought that I provided several reasons before as well, but let me state
> them as clearly as I can here.  Twisted is a large and mature framework with
> several years of development and an active user community.  The pluggable
> event loop it exposes is solving a hard problem, and its implementation
> encodes a lot of knowledge about how such a thing might be done.  It's also
> tested on a lot of different platforms.
>
> Writing all this from scratch - even a small subset of it - is a lot of
> work, and is unlikely to result in something robust enough for use by all
> Python programs without quite a lot of effort.  It would be a lot easier to
> get the Twisted team involved in standardizing event-driven communications
> in Python.  Every other effort I'm aware of is either much smaller, entirely
> proprietary, or both.  Again, I would love to be corrected here, and shown
> some other large event-driven framework in Python that I was heretofore
> unaware of.

Sorry for the long quote; wasn't sure how to trim it.

I made this point elsewhere and above but feel it's important
enough to emphasize once more.

Stackless lets people write code which looks like blocking code
but which is not.  The blocking functions are forwarded to the
reactor, which does whatever it's supposed to do, and the results
returned.

Because most Python networking code will work unchanged
(assuming changes to the underlying APIs to work with
Stackless, as we hack now through monkeypatches), a
microthreads solution almost automatically and trivially gains
a large working code base, documentation, and active developer
base.

There does not need to be "some other large event-driven
framework in Python" because all that's needed is the 13kLOC
of reactor code from twisted.internet and not 140+kLOC in
all of Twisted.

>  Standardization is much easier to achieve when you have
> multiple interested parties with different interests to accommodate.  As
> Yitzhak Rabin used to say, "you don't engage in API standardization with
> your friends, you engage in API standardization with your enemies" - or...
> something like that.

I thought you made contracts even with -- or especially with -- your
friends regarding important matters so that both sides know what
they are getting into and so you don't become enemies in the future.

> You say that you weren't proposing an alternate implementation of an event
> loop core, so I may be reacting to something you didn't say at all.
>  However, again, at least Tristan thought the same thing, so I'm not the
> only one.

For demonstration (and for my case pedagogical reasons) we have
an event core loop.  I would rather pull out the parts I need from
Twisted.  I don't know how.  I don't need to know how right now.

> I think that *that* could be described as bullying.  I -- and many others --
> have a considerable investment in Twisted, and attempts to "standardize" on
> another, new main-loop within the standard library is an attempt to subvert
> that investment and pre-empt honest competition between different

I've lost the thread of this as it's too late in the night for me.  I do
want to point out that whatever Richard proposed cannot be done
without Stackless or an equivalent microthread library.  As that's
unlikely to happen within the next couple of years there's no
chance of bullying here.

Please note that yes I am aware that this means my argument,
now stressed twice in this email, is at present for intellectual
amusement and contemplation only.  Were Stackless, or Dustin's
microthread thoughts, or some solution with libtask (http://swtch.com/libtask/)
or other library added, and if there was more than about 3 days of
experience with this monkeyhack, then things would be different.

But at present this is only a glimmer in about 5 pairs of eyes.


> I've tried to restrict my comments here to the possibility of a new
> event-driven core in the Python standard library.

I do not advocate one.

I do advocate my going to sleep though.  :)

        Andrew
        dalke at dalkescientific.com