[Python-ideas] Enhanced context managers with ContextManagerExit and None

[Nick Coghlan]

On 13 Aug 2013 08:27, "Kristján Valur Jónsson" <kristjan at ccpgames.com> wrote:
> Perhaps my example was not clear enough.  The body will be skipped entirely if the __enter__() method raises an exception.  An outer context manager can then suppress this exception.
>
> However, you cannot create a single context manager that does both.  This, to  me, was the most serious problem with the old nested() context manager: Nesting of two context managers could not be correctly done for arbitrary context managers

nested() was deprecated and removed because it didn't handle files (or
any other CM that does resource acquisition in __init__) correctly.
The fact you can't factor out arbitrary context managers had nothing
to do with it.

> I am not advocating that people do create such context manager, but pointing out that this omission means that you cannot always combine two context managers into one and preserve the semantics because it is possible to do something with two nested context managers that you cannot achieve with a single one.  And this is my completeness argument.

At the moment, a CM cannot prevent execution of the body - it must be
paired with an if statement or an inner call that may raise an
exception, keeping the flow control at least somewhat visible at the
point of execution.

The following is also an illegal context manager:

    @contextmanager
    def bad_cm(broken=False):
        if not broken:
            yield

It's illegal for exactly the same reason this is illegal (this is the
expanded form of your nested CM example):

    @contextmanager
    def bad_cm2(broken=False):
        class Skip(Exception): pass
        try:
            if broken:
                raise Skip
            yield
        except Skip:
            pass

> Imagine if there were code that could only be written using two nested functions, and that the nested function could not be folded into the outer one without sacrificing semantic correctness?  I.e., you could do:
>
>     foo = bar(spam(value))
>
> but this:
>
>    def ham(value):
>         return bar(spam(value))
>
>    foo  = ham(value)
>
> would not be correct for all possible “bar” and “spam”?

But that's not what you're asking about. You're asking for the ability
to collapse two independent try statements into one.

There are already things you can't factor out as functions - that's
why we have generators and context managers. It's also a fact that
there are things you can't factor out as single context managers. This
is
why we have nested context managers and also still have explicit
try/except/else/finally statements.

> As an example, here is what you can currently do in python (again, not recommending it but providing as an example)
>
> class ContextManagerExit(): pass
>
> @contextmanager
> def if_a():
>     try:
>         yield
>     except ContextManagerExit:
>         pass
>
> @contextmanager
> def if_b(condition):
>     if not condition:
>         raise ContextManagerExit
>     yield
>
> with if_a(), if_b(condition):
>     execute_code()  #this line is executed only if “condition” is True

Expand it out to the underlying constructs and you will see this code
is outright buggy, because the exception handler is too broad:

    try:
         if not condition:
            raise ContextManagerExit
         execute_code() # ContextManagerExit will be eaten here
    except ContextManagerExit:
         pass

> In current python, it is impossible to create a combined context manager that does this:
>
> if_c = nested(if_a(), if_b(condition))
>
> with if_c:
>     execute_code()  #A single context manager cannot both raise an exception from __enter__() _and_ have it supressed.

This is a feature, not a bug: the with statement body will *always*
execute, unless __enter__ raises an exception. Don't be misled by the
ability to avoid repeating the with keyword when specifying multiple
context managers in the same statement: semantically, that's
equivalent to multiple nested with statements, so the outer one always
executes, and the inner ones can only skip the body by raising an
exception from __enter__.

> With my patch, “if_b()” is all that is needed, because ContextManagerExit is a special exception that is caught and suppressed by the interpreter.  And so, it becomes possible to nest arbitrary context managers without sacrificing semantic correctness.

I'd be open to adding the following context manager to contextlib:

    @contextmanager
    def skip(keep=False):
        class Skip(Exception): pass
        Skip.caught = None
        try:
            yield Skip
        except Skip as exc:
            if keep:
                Skip.caught = exc

This would allow certain currently awkward constructs to be expressed
more easily without needing to drop back to a try/except block (note
that 3.4 already adds contextlib.ignored to easily suppress selected
exceptions in a block of code). Creating a custom exception type each
time helps avoid various problems with overly broad exception
handlers.

> But I agree that None is problematic for the reason you demonstrate, and did consider that.  I’m suggesting it here as a demonstration of the concept, and also to reduce the need for yet another built-in.  Perhaps the ellipsis could be used, that’s everyone’s favourite singleton :)

An empty contextlib.ExitStack() instance is already a perfectly
serviceable "do nothing" context manager, we don't need (and won't
get) another one.

Cheers,
Nick.