"Exception ... in <generator object ...> ignored" Messages

[Peter Otten]

Ami Tavory wrote:

> From: Peter Otten <__peter__ at web.de>
> To: python-list at python.org
> Cc:
> Date: Thu, 14 Feb 2013 09:00:58 +0100
> Subject: Re: "Exception ... in <generator object ...> ignored" Messages

How did the subject get into the message body?


> Ami Tavory wrote:
> 
>>   Hi,
>>
>>   Running the unit tests for some generator code, prints, as a side
>>   effect,
>> numerous messages of the form:
>>
>> ...
>> Exception NameError: "global name 'l' is not defined" in <generator
>> object _dagpype_internal_fn_act at 0x9d4c500> ignored
>> Exception AttributeError: "'NoneType' object has no attribute 'close'" in
>> <generator object split at 0x7601640> ignored
>> Exception AttributeError: "'NoneType' object has no attribute 'close'" in
>> <generator object split at 0x7601690> ignored
>> ...
>>
>> The tests otherwise run fine.
>>
>>   Is there any way to catch the point where such a message originates,
>>   and
>> print a traceback? I find it difficult to debug otherwise. I've tried
>> running Python with -W error, catching warnings with context managers,
>> and so forth, but without any success.
> 
>>>> def g():
> ...     try:
> ...             yield 42
> ...     finally:
> ...             1/0
> ...
>>>> for item in g():
> ...     break
> ...
> Exception ZeroDivisionError: 'integer division or modulo by zero' in
> <generator object g at 0x7f990243b0f0> ignored
> 
> Can you exhaust the generator?
> 
>>>> for item in g():
> ...     pass
> ...
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 5, in g
> ZeroDivisionError: integer division or modulo by zero
> 
> Explicitly closing the generator seems to work, too:
> 
>>>> x = g()
>>>> next(x)
> 42
>>>> x.close()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 5, in g
> ZeroDivisionError: integer division or modulo by zero


>   So basically the thing is that, as you've shown, if you exhaust a
> generator or force-close it, an exception is indeed thrown. However, in
> this were the case then unittest would identify it as a problem, and I
> could relatively easily get a traceback and see from where it's
> originating. However, the unittests are passing fine (except for the pesky
> messages), so it's something else.
>   I'm starting to think that what's happening is something like this: I
> have a cascade of generator objects (some hold on to others), and the
> unittests check both the exhaustion case and the non-exhaustion case (both
> cases are valid use cases). *Later on*, after the tests finish, the
> garbage collector starts tearing down the generator cascade, but in a
> different order. Some generators, while closing down, are attempting to
> close down other generators that no longer exist.
>   So, notwithstanding your correct answer - I'm still stumped about
>   finding
> out how to get rid of these annoying messages.

Posting a minimal example that exposes the behaviour and reflects the 
situation in your actual code would definitely a good idea ;)

But short of closing your generators in a controlled manner the only thing 
that might help is avoiding global names in the finally clause. Globals are 
all set to None when the module object is destroyed. Code that runs in that 
phase may fail for that reason:

$ cat tmp.py
class X:
    def __init__(self, name):
        self.name = name

x = X("whatever")

def g():
    try:
        yield 42
    finally:
        print x.name

y = g()
next(y)
$ python tmp.py
Exception AttributeError: "'NoneType' object has no attribute 'name'" in 
<generator object g at 0x7f3dbf3580f0> ignored
$ cat tmp2.py
class X:
    def __init__(self, name):
        self.name = name

x = X("whatever")

def g(x=x):
    try:
        yield 42
    finally:
        print x.name

y = g()
next(y)
$ python tmp2.py
whatever

I'd still recommend fixing the cause rather than the symptoms...