[Python-ideas] ExitStack: Allow exiting individual context managers

[Nick Coghlan]

On 7 December 2015 at 10:32, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sun, Dec 06, 2015 at 11:48:41PM +0200, Ram Rachum wrote:
>> Hi guys,
>>
>> I'm using `contextlib.ExitStack` today, and pushing context managers into
>> it. I find myself wanting to exit specific context managers that I've
>> pushed into it, while still inside the `with` suite of the `ExitStack`. In
>> other words, I want to exit one of the context managers but still keep the
>> `ExitStack`, and all other context managers, acquired. This isn't currently
>> possible, right? What do you think about implementing this?
>
> I'm not entirely sure what you mean. Can you give an example?

It's a concept I considered implementing, but decided to hold off on
it because there are a few different design options here and I didn't
have any use cases to guide the API design, nor the capacity to do
usability testing to see if the additional API complexity actually
made ExitStack easier to use overall.

The first design option is the status quo: using multiple with
statement blocks, potentially in conjunction with multiple ExitStack
instances. The virtue of this approach is that it means that once a
context manager is added to an ExitStack instance, that's it - its
lifecycle is now coupled to that of the other context managers in the
stack. You can postpone cleaning up all of them with "pop_all()"
(transferring responsibility for the cleanup to a fresh ExitStack
instance), but you can't selectively clean them up from the end. The
usage guidelines are thus relatively simple: if you don't want to
couple the lifecycles of two context managers together, then don't add
them to the same ExitStack instance.

However, there is also that "Stack" in the name, so it's natural for
users to expect to be able to both push() *and* pop() individual
context managers on the stack.

The (on the surface) simplest design option to *add* to the API would
be a single "pop()" operation that returned a new ExitStack instance
(for return type consistency with pop_all()) that contained the last
context manager pushed onto the stack. However, this API is
problematic, as you've now decoupled the nesting of the context
manager stack - the popped context manager may now survive beyond the
closure of the original ExitStack. Since this kind a pop_all()
inspired selective clean-up API would result in two different
ExitStack instances anyway, the status quo seems cleaner to me than
this option, as it's obvious from the start that there are seperate
ExitStack instances with potentially distinct lifecycles.

The next option would then be to offer a separate
"exit_last_context()" method, that exited the last context manager
pushed onto the stack. This would be a single-stepping counterpart to
the existing close() method, that allowed you to dynamically descend
and ascend the context management stack during normal operation, while
still preserving the property that the entire stack will be cleaned up
when encountering an exception.

Assuming we went with that simpler in-place API, there would still be
a number of further design questions to be answered:

* Do we need to try to manage the reported exception context the way
ExitStack.__exit__ does?
* Does "exit_last_context()" need to accept exception details like
__exit__ does?
* Does "exit_last_context()" need to support the ability to suppress exceptions?
* What, if anything, should the return value be?
* What happens if there are no contexts on the stack to pop?
* Should it become possible to query the number of registered callbacks?

Taking them in order, as a matter of implementation feasibility, the
answer to the first question likely needs to be "No". For consistency
with calling __exit__ methods directly, the answers to the next three
questions likely need to be "support the same thing __exit__
supports".

For the second last question, while it's reasonable to call close(),
pop_all() or __exit__() on an empty stack and have it silently do
nothing, if someone has taken it on themselves to manage the stack
depth manually, then it's likely more helpful to complain that the
stack is empty than it is to silently do nothing. Since
exit_last_context() may behave differently depending on whether or not
there are items on the stack, and the number of items on the stack
would be useful for diagnostic purposes, then it likely also makes
sense to implement a __len__ method that delegated to
"len(self._exit_callbacks)".

That all suggests a possible implementation along the lines of the following:

    def exit_last_context(self, *exc_details):
        if not self._exit_callbacks:
            raise RuntimeError("Attempted to exit last context on
empty ExitStack instance")
        cb = self._exit_callbacks.pop()
        return cb(*exc_details)

    def __len__(self):
        return len(self._exit_callbacks)

What I don't know is whether or not that's actually a useful enough
improvement over the status quo to justify the additional cognitive
burden when learning the ExitStack API - the current API was designed
around the ExitStack recipes in the documentation, which were all
fairly common code patterns, but most cases where I might consider
using an "exit_last_context()" method, I'd be more inclined to follow
Steven's advice and use a separate context manager for the resource
with an independent lifecycle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia