[Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore())

[Nick Coghlan]

On 18 Oct 2013 09:05, "Guido van Rossum" <guido at python.org> wrote:
>
> On Thu, Oct 17, 2013 at 3:51 PM, Oscar Benjamin <
oscar.j.benjamin at gmail.com> wrote:
>>
>> On 17 October 2013 20:01, Guido van Rossum <guido at python.org> wrote:
>> > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin
>> > <oscar.j.benjamin at gmail.com> wrote:
>> >>
>> >> On 17 October 2013 19:40, Xavier Morel <python-dev at masklinn.net>
wrote:
>> >> > I think there's already a significant split between context managers
>> >> > which handle the lifecycle of a local resource (file, transaction)
and
>> >> > those which purport to locally alter global-ish state (cwd,
>> >> > decimal.localcontext, logging.captureWarnings, redirect_stdout).
>> >> >
>> >> > And the latter worries me (much more than the very localized
behavior of
>> >> > suppress) because I don't see any way to implement them safely and
>> >> > correctly when mixing it with coroutines in today's Python (some of
them
>> >> > aren't even thread-safe), all of that while I expect coroutines
will see
>> >> > significantly more use in the very near future with yield from and
>> >> > tulip's promotion of coroutine-style async.
>> >>
>> >> I maybe misunderstanding how the  coroutine-style async works but I
>> >> would have thought that it would be as simple as: don't use
>> >> global-state-restoring-context-managers around statements that yield
>> >> control (it would be simpler if there was a good term for describing
>> >> that kind of CM). That's simpler to implement and computationally
>> >> cheaper than e.g. the thread-local state used by the decimal module.
>> >
>> > Context managers that actually save and restore *global* state are
already
>> > not thread-safe, so concluding they are also not coroutine-safe (or
>> > task-safe?) seems a small step.
>> >
>> > I'd be more worried about context manager that use thread-local state
--
>> > there is no similar concept in Tulip.
>>
>> It's unnecessary in Tulip. The need for thread-local state in e.g.
>> decimal contexts is driven by the fact that multi-threaded execution
>> switches in an uncontrollable way. Tulip specifically makes it
>> possible to control the points at which a switch occurs making this
>> safe (even if localcontext() wasn't thread-safe):
>>
>> with decimal.localcontext() as ctx:
>>     ctx.prec = 100
>>     c = a + b
>>     # more synchronous decimal calculations
>>
>> # State is restored before allowing other code to execute
>> yield from save_in_database(c)
>>
>> So it's fine to use global/thread-local state modifying/restoring
>> context managers in Tulip as long as you don't yield control to other
>> code within the with block. (unless I misunderstand - I lost track of
>> Tulip some time ago).
>
>
> You've got it exactly right.
>
>>
>> The issue with decimal.localcontext() and yield arises when using
>> generators as much as coroutines e.g.:
>>
>> def exact_sum(nums):
>>     start = Decimal(0)
>>     with decimal.localcontext() as ctx:
>>         ctx.traps[decimal.Inexact] = True
>>         for num in nums:
>>             try:
>>                 total += Decimal(num)
>>             except decimal.Inexact:
>>                 ctx.prec *= 2
>>     return total
>>
>> The above is fine for computing the sum of a list of
>> Decimals/ints/floats. However it fails if you pass in a generator that
>> carelessly modifies the arithmetic context around yield calls:
>>
>> def compute():
>>     with decimal.localcontext() as ctx:
>>         ctx.prec = 15
>>         ctx.traps[decimal.Inexact] = False
>>         yield a + b
>>         yield b - c
>>         # etc.
>>
>> exact_sum(compute())
>>
>> There needs to be a convention that either functions like exact_sum()
>> mustn't assume continuity of the context between iterations or a
>> function like compute() must restore before yielding. IMO the only
>> sane approach for async coroutines is to say that if you yield or
>> yield from then it is your responsibility to restore any temporarily
>> altered global/thread-local state first.
>
>
> Right again. The simplest rule to remember seems to be "don't use yield
or yield-from inside a with-statement". You can relax it by limiting it to
context managers that manage any kind of shared resource, but that is
probably already too subtle: e.g. yielding inside "with open(file) as f"
seems fine, but yielding inside "with lock" is problematic, since the other
side might try to acquire the same lock, and deadlock.

Even the file case can be tricky - if you actually yield *the file* (rather
than nothing or its contents), then the recipient may be surprised when it
is later closed automatically.

It wouldn't surprise me if we see idioms to temporarily revert the effects
of context managers start to grow in popularity once asyncio is released as
part of 3.4.

Cheers,
Nick.

>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20131018/c2a9f295/attachment.html>