[ python-Bugs-1055820 ] weakref callback vs gc vs threads

[SourceForge.net]

Bugs item #1055820, was opened at 2004-10-27 21:58
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 9
Submitted By: Tim Peters (tim_one)
Assigned to: Neil Schemenauer (nascheme)
Summary: weakref callback vs gc vs threads

Initial Comment:
Oh oh.  It's that time of year again.  I'm sure the 
attached (temp2a.py) can be simplified, perhaps down 
to two objects and one thread.  As is, I *think* it 
demonstrates that invoking a weakref callback can do 
fatal damage, not necessarily because of what the 
callback does, but because simply calling it while gc is 
running can allow other threads to run during gc too, 
and resurrect a piece of cyclic trash T that's already 
been tp_clear()'ed, via invoking a still-living weakref to 
T.

If so, this isn't new in 2.4.  It's a real problem since 
temp2a.py is what's left of ZODB 3.4a1 <wink>.

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2004-10-29 05:11

Message:
Logged In: YES 
user_id=31435

I strongly suspect that abusing the weakref struct's `hash` 
member is responsible for the gazillions of KeyErrors.  Dang.  I 
need one lousy bit ...

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-29 02:54

Message:
Logged In: YES 
user_id=31435

Neil, can you make some time to eyeball this (patch.txt)?  If 
not, please assign to Fred.  The Python -uall suite passes in 
debug and release builds.

I just ran the -all Zope3 test suite in a debug buld, and that 
passes as well as it does with Python 2.3.4 on Windows, but 
I see many thousands of lines like:

Exception exceptions.KeyError: <weakref at 104EA620; dead> 
in <function remove at 0x01595560> ignored

So there's something still wrong here.  I'm not sure who's 
writing those msgs; I expect it's in the guts of gc, when 
invoking delayed weakref callbacks, and triggered by the 
weak dictionary implementations (which haven't changed).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-28 16:55

Message:
Logged In: YES 
user_id=31435

A fix is in progress, as sketched on Python-Dev.  I expect it 
to land Friday (or tonight yet, if I'm lucky).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-28 04:56

Message:
Logged In: YES 
user_id=31435

temp2d.py shows that we don't need any weakrefs with 
callbacks to get in trouble -- a __del__ method can hose us 
too.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-28 04:06

Message:
Logged In: YES 
user_id=31435

temp2c.py may be as bad as it gets.  It shows that the 
problem can occur on a gc collection that doesn't see *any* 
object that has a weakref with a callback.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-27 22:37

Message:
Logged In: YES 
user_id=31435

Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3.  
That's all the Pythons I have handy right now.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-10-27 22:28

Message:
Logged In: YES 
user_id=31435

Yup, temp2b.py probably shows the same problem, with 2 
objects and one thread.  This one is definitely more strained, 
though, since the weakref callback does the damage 
directly.  In temp2a.py, nothing is trying to fool anything, and 
the only damage done by the wr callbacks there is simply in 
releasing the GIL (thus allowing other threads to do perfectly 
ordinary things with weakrefs).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470