[Python-ideas] Thoughts on the future of concurrency in Python: PEP 3148

[Steven Hazel]

Hi,

I'm one of the authors of monocle (https://github.com/saucelabs/monocle),
an async Python programming framework with a blocking look-alike syntax
based on generators. We've been using monocle at my startup Sauce Labs, and
I used its predecessor on a small team at BitTorrent. I think it's met with
some amazing success vs. similar projects using threads. In about five
years of working with teams with some members who haven't always understood
monocle very well, we've never written a deadlock, for example.

I've been keeping up with PEPs and skimming python-ideas discussions about
concurrency-related features in Python, and feeling unsure about how best
to jump into the conversation. Today it occurred to me that sooner is
better and I should just get all my various opinions out there in case
they're helpful.

First of all, I want to reiterate: while I have mainly criticism to offer,
it is only intended to be helpful. I don't have a lot of time to devote to
this myself, so I'm trying to do what I can to be useful to those who do.

I think I'll take this one PEP at a time, and start with the easiest one.

PEP 3148 - Futures

First, I like the idea of this PEP. I think it'll improve interoperability
between async frameworks in Python.

I believe the name "future" is a poor choice. After many years of
explaining this concept to programmers, I've found that its various
abstract-conceptual names do a lot to confuse people. Twisted's "deferred"
is probably the worst offender here, but both "future" and "promise" also
make this very simple construct sound complicated, and it's a crime.

There are two ways I've found to explain this that seem to keep people from
getting confused:

The first is, call it a "callback registry". Many programmers are familiar
with the idea of passing callbacks into a function. You just explain that
here, instead of taking the callbacks as parameters, we return a callback
registry object and you can add your callbacks there. (If the operation is
already done they'll just get called when you add them.)

The second is a variant on that: say that this function, instead of taking
a callback as a parameter, returns its callback. That sounds useless, but
fortunately the callback it returns is a callable object, and it also has
an "add" method that lets you add functions that it will call when called
back. So it's a callback that you can manipulate to do what you want. (This
is what we do in monocle:
https://github.com/saucelabs/monocle/blob/master/monocle/callback.py)

These approaches, by focusing on the simple and familiar idea of callbacks,
do a lot to eliminate confusing ideas about communicating with the future
of the process.

Finally, I'm not sure I get the point of the Executor class in PEP 3148. It
seems to me that any real implementation of that class needs a call_later
method for scheduling operations, and a way of connecting up to event-based
IO APIs. I don't really understand what Executor would be good for.

Thanks for reading.

-- 
Steven Hazel
Sauce Labs Cofounder and VP Product
http://saucelabs.com/
work: sah at saucelabs.com
other: sah at awesame.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130317/02459f2b/attachment.html>