[Python-Dev] Trial balloon: microthreads library in stdlib

[Richard Tew]

On 2/11/07, "Martin v. Löwis" <martin at v.loewis.de> wrote:
> Richard Tew schrieb:
> > If these generator coroutine microthreads went ahead and part
> > of it was improving the support for asynchronous calls in the
> > runtime and standard library, this would also be something
> > which also benefited Stackless Python.  Even if they didn't go
> > ahead I would be interested in hearing about whether these
> > sort of changes would be of interest or not for whatever reason.
>
> For me to respond to such a question, I first need to understand
> what changes precisely you propose. "wrapping calls out of the Python
> runtime to things like network and file IO, so that microthreads
> can naturally take advantage of them." is a bit vague: how precisely
> would you wrap them?

I was meaning wrapping on the Stackless side of things and it
was not meant as a reference to how better support for asynchronous
calls might be added.  You can see how wrapping is done with
Stackless channels in the examples I linked to at the end of this
mail on the off chance you were curious about that instead and I
am happy to give any further detail needed.

> I assume you would like to see operating system mechanism be used
> that currently aren't used (similar to the way asynchore uses
> select/poll).

Yes.

> If so, what mechanism would you like to use, and on what systems?

It depends what you mean by mechanism.  I can't go into low level details
because I do not have the necessary experience or knowledge to know
which approach would be best.  But I do have some general thoughts
and would be willing to do any work involved, if there was a feasible
solution and the changes required were thought worthwhile.

Let me describe the situation I currently face in order to give context,
please excuse the Windows specifics and the lack of consideration
for other platforms.

I currently use Windows.  By using asyncore and monkeypatching in a
replacement socket module based on it [1] I can give users of Stackless
socket operations which invisibly block at the microthread level allowing
the scheduler to continue to run.  However there is no support for
asynchronous file IO.  Now I can monkeypatch in a replacement file
object which uses IO completion ports [2], but at that stage I need to
poll two different resources for events.  In order to avoid this it makes
sense to stop using asyncore for sockets and to write a new
replacement socket object based on IO completion ports.

Then there are other calls which come into the mix.  Like in the
subprocess module.  As I understand it I would need to monkeypatch
the Popen.wait, and os.waitpid calls so that they were polled internally
to the monkeypatching and Stackless could handle the wake up of the
caller there as well.  And there might be other blocking calls in the
runtime which it would be worth handling also, which might not have
methods any way of operating asynchronously (like sockets, files and
subprocesses).

The ideal mechanism at the high level would be expanding asyncore into
a "one-stop shop".  Where all these things can be passed into it and
it can do the work to notify of events on the objects in a standard way.
However even if this were judged feasible, one immediate hurdle is that
files passed to it would need to have been opened in overlapped mode
for IO completion ports could be used with it.  I presume generally
opening all files in overlapped mode on Windows would address this
thought.  And I guess all the other "stuff" which does not relate to IO
completion ports like subprocesses would be bundled into
a WaitForMultipleObject call.

Perhaps there is a better way.  And I of course have no concept of
how this might be done on other platforms.

Thanks,
Richard.

[1] http://svn.python.org/view/stackless/sandbox/examples/stacklesssocket.py
[2] http://svn.python.org/view/stackless/sandbox/libraries/slpmonkeypatch/resources/iocp.py