[Python-ideas] Yielding through context managers
Nick Coghlan
ncoghlan at gmail.com
Sun Jan 6 10:06:31 CET 2013
On Sun, Jan 6, 2013 at 5:23 AM, Guido van Rossum <guido at python.org> wrote:
> Possibly (though it will have to be a separate PEP -- PEP 3156 needs
> to be able to run on unchanged Python 3.3). Does anyone on this thread
> have enough understanding of the implementation of context managers
> and generators to be able to figure out how this could be specified
> and implemented (or to explain why it is a bad idea, or impossible)?
There aren't any syntax changes needed to implement asynchronous
locks, since they're unlikely to experience high latency in __exit__.
For that and similar cases, it's enough to use an asynchronous
operation to retrieve the CM in the first place (i.e. acquire in
__iter__ rather than __enter__) or else have __enter__ produce a
Future that acquires the lock in __iter__ (see
http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/async_programming.html#asynchronous-context-managers)
The real challenge is in handling something like an asynchronous
database transaction, which will need to yield on __exit__ as it
commits or rolls back the database transaction. At the moment, the
only solutions for that are to switch to a synchronous-to-asynchronous
adapter like gevent or else write out the try/except block and avoid
using the with statement.
It's not an impossible problem, just a tricky one to solve in a
readable fashion. Some possible constraints on the problem space:
- any syntactic solution should work for at least "for" statements and
"with" statements
- also working for comprehensions is highly desirable
- syntactic ambiguity with currently legal constructs should be
avoided. Even if the compiler can figure it out, large behavioural
changes due to a subtle difference in syntax should be avoided because
they're hard for *humans* to read
For example:
# Synchronous
for x in y: # Invokes _iter = iter(y) and _iter.__next__()
print(x)
#Asynchronous:
for x in yielding y: # Invokes _iter = yield from iter(y) and
yield from _iter.__next__()
print(x)
# Synchronous
with x as y: # Invokes _cm = x, y = _cm.__enter__() and
_cm.__exit__(*args)
print(y)
#Asynchronous:
with yielding x as y: # Invokes _cm = x, y = yield from
_cm.__enter__() and yield from _cm.__exit__(*args)
print(y)
A new keyword like "yielding" would make it explicit that what is
going on differs from a (yield x) or (yield from x) in the
corresponding expression slot.
Approaches with function level granularity may also be of interest -
PEP 3152 is largely an exploration of that idea (but would need
adjustments in light of PEP 3156)
Somewhat related, there's also a case to be made that "yield from x"
should fall back to being equivalent to "x()" if x implements __call__
but not __iter__. That way, async ready code can be written using
"yield from", but passing in a pre-canned result via lambda or
functools.partial would no longer require a separate operation that
just adapts the asynchronous call API (i.e. __iter__) to the
synchronous call one (i.e. __call__):
def async_call(f):
@functools.wraps(f)
def _sync(*args, **kwds):
return f(*args, **kwds)
yield # Force this to be a generator
return _iterable_call
The argument against, of course, is the ease with which this can lead
to a "wrong answer" problem where the exception gets thrown a long way
from the erroneous code which left out the parens for the function
call.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
More information about the Python-ideas
mailing list