[Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore())

[Guido van Rossum]

On Thu, Oct 17, 2013 at 3:51 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com>wrote:

> On 17 October 2013 20:01, Guido van Rossum <guido at python.org> wrote:
> > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin
> > <oscar.j.benjamin at gmail.com> wrote:
> >>
> >> On 17 October 2013 19:40, Xavier Morel <python-dev at masklinn.net> wrote:
> >> > I think there's already a significant split between context managers
> >> > which handle the lifecycle of a local resource (file, transaction) and
> >> > those which purport to locally alter global-ish state (cwd,
> >> > decimal.localcontext, logging.captureWarnings, redirect_stdout).
> >> >
> >> > And the latter worries me (much more than the very localized behavior
> of
> >> > suppress) because I don't see any way to implement them safely and
> >> > correctly when mixing it with coroutines in today's Python (some of
> them
> >> > aren't even thread-safe), all of that while I expect coroutines will
> see
> >> > significantly more use in the very near future with yield from and
> >> > tulip's promotion of coroutine-style async.
> >>
> >> I maybe misunderstanding how the  coroutine-style async works but I
> >> would have thought that it would be as simple as: don't use
> >> global-state-restoring-context-managers around statements that yield
> >> control (it would be simpler if there was a good term for describing
> >> that kind of CM). That's simpler to implement and computationally
> >> cheaper than e.g. the thread-local state used by the decimal module.
> >
> > Context managers that actually save and restore *global* state are
> already
> > not thread-safe, so concluding they are also not coroutine-safe (or
> > task-safe?) seems a small step.
> >
> > I'd be more worried about context manager that use thread-local state --
> > there is no similar concept in Tulip.
>
> It's unnecessary in Tulip. The need for thread-local state in e.g.
> decimal contexts is driven by the fact that multi-threaded execution
> switches in an uncontrollable way. Tulip specifically makes it
> possible to control the points at which a switch occurs making this
> safe (even if localcontext() wasn't thread-safe):
>
> with decimal.localcontext() as ctx:
>     ctx.prec = 100
>     c = a + b
>     # more synchronous decimal calculations
>
> # State is restored before allowing other code to execute
> yield from save_in_database(c)
>
> So it's fine to use global/thread-local state modifying/restoring
> context managers in Tulip as long as you don't yield control to other
> code within the with block. (unless I misunderstand - I lost track of
> Tulip some time ago).
>

You've got it exactly right.


> The issue with decimal.localcontext() and yield arises when using
> generators as much as coroutines e.g.:
>
> def exact_sum(nums):
>     start = Decimal(0)
>     with decimal.localcontext() as ctx:
>         ctx.traps[decimal.Inexact] = True
>         for num in nums:
>             try:
>                 total += Decimal(num)
>             except decimal.Inexact:
>                 ctx.prec *= 2
>     return total
>
> The above is fine for computing the sum of a list of
> Decimals/ints/floats. However it fails if you pass in a generator that
> carelessly modifies the arithmetic context around yield calls:
>
> def compute():
>     with decimal.localcontext() as ctx:
>         ctx.prec = 15
>         ctx.traps[decimal.Inexact] = False
>         yield a + b
>         yield b - c
>         # etc.
>
> exact_sum(compute())
>
> There needs to be a convention that either functions like exact_sum()
> mustn't assume continuity of the context between iterations or a
> function like compute() must restore before yielding. IMO the only
> sane approach for async coroutines is to say that if you yield or
> yield from then it is your responsibility to restore any temporarily
> altered global/thread-local state first.
>

Right again. The simplest rule to remember seems to be "don't use yield or
yield-from inside a with-statement". You can relax it by limiting it to
context managers that manage any kind of shared resource, but that is
probably already too subtle: e.g. yielding inside "with open(file) as f"
seems fine, but yielding inside "with lock" is problematic, since the other
side might try to acquire the same lock, and deadlock.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20131017/015918d2/attachment.html>