[Python-ideas] [Python-Dev] Python needs a standard asynchronous return object

[Guido van Rossum]

On Mon, Sep 20, 2010 at 2:41 PM, James Yonan <james at openvpn.net> wrote:
> I think that Glyph hit the nail on the head when he said that "you can go
> from any arbitrary Future to a full-featured Deferred, but not the other way
> around."

Where by "go from X to Y" you mean "take a program written using X and
change it to use Y", right?

> This is exactly my concern, and the reason why I think it's important for
> Python to standardize on an async result type that is sufficiently general
> that it can accommodate the different kinds of async semantics in common use
> in the Python world today.

I think I get your gist.

Unfortunately there's only a small number of people who know enough
about async semantics in order to write the PEP that is needed.

> If you don't think this is a problem, just Google for "twisted vs. tornado".
>  While the debate is sometimes passionate and rude,

Is it ever distanced and polite? :-)

> it points to the
> fragmentation that has occured in the Python async space due to the lack of
> direction from the standard library.  And there's a real cost to this
> fragmentation -- it's not easy to build an application that uses different
> async frameworks when there's no standardized result object or reactor
> model.

But, circularly, the lack of direction from the standard library is
that nobody has contributed an async framework to the standard library
since asyncore was added in, oh, 1999.

> My concern is that PEP 3148 was really designed for the purpose of thread
> and process pooling, and that the Future object is designed with the minimum
> functionality required to achieve this end.  The problem is that the Future
> object starts to look like a stripped-down version of a Twisted Deferred.
>  And that begs the question of why are we standardizing on the special case
> and not the general case?

Because we could reach agreement fairly quickly on PEP 3148. There are
some core contributors who know threads and processes inside out, and
after several rounds of comments (a lot, really) they were satisfied.

At this point it is probably best to forget about PEP 3148 if you want
to improve the async situation in the stdlib, and start thinking about
that async PEP instead.

> Wouldn't it be better to break this into two problems:
>
> * Develop a full-featured standard async result type and reactor model to
> facilitate interoperability of different async libraries.  This would
> consist of a standard async result type and an abstract base class for a
> reactor model.

Unless you want to propose to include Twisted into the stdlib, this is
not going to be ready for inclusion into Python 3.2.

> * Let PEP 3148 focus on the problem of thread and process pooling and
> leverage on the above async result type.

But PEP 3148 *is* ready for inclusion in Python 3.2. So you've got the
ordering wrong. It doesn't make sense to hold up PEP 3148, waiting for
the perfect solution to appear. In fact, the changes that were made to
PEP 3148 at Glyph's suggestion are probably all you are going to get
regarding PEP 3148.

> The semantics that a general async type should support include:
>
> 1. Semantics that allow you to define a callback channel for results and and
> optionally a separate channel for exceptions as well.
>
> 2. Semantics that offer the flexibility of working with async results at the
> callback level or at the generator level (having a separate channel for
> exceptions makes it easy for the generator decorator implementation (that
> facilitates "yield function_returning_async_object()") to dispatch
> exceptions into the caller).
>
> 3. Semantics that can easily be used to pass results and exceptions back
> from thread or process pools.
>
> 4. Semantics that allow for aggregate processing of parallel asynchronous
> results, such as "fire async result when all of the async results in an
> async set have fired" or "fire async result when the first result from an
> async set has fired."
>
> Deferreds presently support all of the above.  My point here is not so much
> that Deferreds should be the standard, but that whatever standard is chosen,
> that the semantics be general enough that different async Python
> libraries/platforms can interoperate.

Do you want to champion a PEP? I hope you do -- it will be a long
march but rewarding, especially if you get the Tornado folks to
participate and contribute.

-- 
--Guido van Rossum (python.org/~guido)