[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

[Larry Hastings]

On 10/10/2016 09:36 PM, Chris Angelico wrote:
> Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
> deallocation lock". While it is held (by any thread - it's a guard,
> more than a lock), Py_DECREF will not actually deallocate memory -
> objects can fall to zero references without being wiped. Once the
> lock/guard is freed/cleared, anything that had fallen to zero is now
> deallocated. This probably would mean stuffing them onto a list of
> "doomed objects", and upon release of the guard, any doomed objects
> that still have no refs would get deallocated.

If this worked, this would actually be really easy with my current 
"buffered reference counting" approach.  I literally already have a list 
of doomed objects, which a separate thread queues up for each thread to 
process later.

(This was a necessary part of "buffered reference counting", in which 
reference count changes are stored in a transaction log and later 
committed by a single thread.  Since there's only one thread making 
reference count changes, there's no contention, so it doesn't have to 
use atomic incr/decr, which is a big performance win.)

But I don't think this fixes the problem.  Consider:

 1. Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed
    reference.  Thread A then gets suspended, before it has a chance to
    Py_INCREF(Q).
 2. Thread B does L.clear(), the reference count of Q goes to 0, Q is
    added to the doomed objects list.
 3. Thread A gets to run again.  It does Py_INCREF(Q); Q's refcount is
    now back up to 1, as if Q had been resurrected.
 4. At some point in the future the "memory deallocation lock" is
    released and Q is deleted.

If you say "well, just look at the reference count, and if it's 1 throw 
it off the doomed objects list", keep in mind that I no longer have 
accurate real-time reference counts.  These hacks where we play games 
with the reference count are mostly removed in my branch.

Also, I don't know when it would ever be safe to release the "memory 
deallocation lock".  Just because it's safe for your thread doesn't mean 
it's safe for another thread.  And if you do it on a thread-by-thread 
basis, in the above example it might be safe from thread B's perspective 
to release its "memory deallocation lock", but as illustrated that can 
have an effect on thread A.

I appreciate the brainstorming but I'm not currently sanguine about this 
idea,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/3b343b29/attachment.html>