[Python-Dev] context manager - generator interaction?

[Phillip J. Eby]

At 03:54 PM 4/5/2007 -0400, Bob Sidebotham wrote:
>The interaction shown below feels like a bug, or at least very much a
>trap for the unwary. It's some sort of interaction between a context
>manager and a generator, both of which can raise StopIteration. The
>code is excised from a real application, so it's a bit artificial
>looking. Nevertheless, it represents an entirely natural evolution of
>some code (whereby the call to a context manager was added deep within
>the code). Since it seems to work with open as the context manager,
>but not with my own null context manager defined with
>context_manager(), it feels like this is probably a bug in
>contextmanager().

It actually appears to be a quirk in the "with" machinery itself, that only 
affects generator context managers because of their need to handle 
StopIteration specially.  If you rewrite your context manager as a class 
with __enter__ and __exit__, the problem will go away.

Checking what was happening via pdb, I found that the contextmanager's 
__exit__ is being called with a "value" of None, rather than with an 
exception *instance*.  This fouls up GeneratorContextManager.__exit__'s 
special handling for StopIteration -- which was apparently never tested by 
any unit tests.

I can see a few different ways to work around the problem in 
GeneratorContextManager itself, the simplest of which is to add these two 
lines right before the self.gen.throw() call:

                 if type is StopIteration and value is None:
                     value = type()

It seems to me that this is a bit of a kludge, but trying to make it so 
that __exit__ always gets called with an instance is much more of a pain 
since it appears that the None a performance enhancement for simple errors 
like StopIteration, that avoids creating an instance.