[py-dev] execnet: shutdown issues in a heavily threaded environment

[holger krekel]

Hi Gordon,

On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
> > One issue that can pop up is tearing down the whole
> > machinery cleanly - that can sometimes be more
> > involved and i believe execnet still might need
> > some more work there.
> 
> On that note can you shed any light on this message:
> 
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
> open>> ignored
>
> Currently I'm only seeing them on my test machine, but they are
> occurring very frequently, they drop out at the end of the py.test
> invocation, I will paste the tail end of the py.test output below.

> I'm not sure exactly what is causing this, naively it seems to
> indicate a circular chain of references involving one or more
> channels. However immediately prior to the conclusion of each test
> exit gc.garbage is empty and there are none of my objects in
> gc.get_objects.

Yes, some cycle is probably there.  Your data point hints that
there is some cycle in the execnet code itself.  

> Also this only started when I reworked our remote invocation system
> recently, the most notable change in the new system is that it makes
> more use of threads and particularly it has daemon threads waiting on
> channel receive calls.

Hum, that might be related - GC-finalization issues in
MT-environments are not easy to debug :( 
Might also be python-version dependent.  What are 
the differences between your test and production system? 

I am a bit at a loss on how to best proceed at the moment. 
I think that adding more debugging information
to execnet and systematically implementing and
checking scenarios is due - but quite a bit of effort.  
Could you maybe try coming up with an self-contained 
example test-script leading to these messages? 

On a sidenote, probably around 60% of the execnet core
programming effort revolve around teardown/finalization issues
- i wonder if it would be better to avoid usage of __del__
alltogether, only have a process-wide atexit handler and
otherwise recommend explicit calling of
gateway.exit()/channel.close methods for proper resource
handling.  

best,

holger
 
> lab at okum:~$ uname -a
> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
> 2009 i686 GNU/Linux
> lab at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import py
> >>> py.version
> '1.0.0b1'
> 
> 
> ============================================== 35 passed in 231.71
> seconds ===============================================
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
> method Thread.__bootstrap of <Thread(receiver, stopped daemon
> 1177635728)>>
> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
> 38, in apport_excepthook
>     from apport.packaging_impl import impl as packaging
> ImportError: No module named apport.packaging_impl
> 
> Original exception was:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>     self.__bootstrap_inner()
>   File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>     (self.name, _format_exc()))
> TypeError: 'NoneType' object is not callable
>  ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu