[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

[Chris Angelico]

On Tue, Oct 11, 2016 at 3:49 AM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2016-10-10 10:45, Chris Angelico wrote:
>>
>> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings <larry at hastings.org>
>> wrote:
>>>
>>> Huh?  In all other circumstances, a "borrowed" reference is exactly that:
>>> X
>>> has a reference, and you are relying on X's reference to keep the object
>>> alive.  Borrowing from a borrowed reference is simply a chain of these; Z
>>> borrows from Y, Y borrows from X, and X is the original person who did
>>> the
>>> incref.  But you're borrowing from something specific, somebody who the
>>> API
>>> guarantees has a legitimate reference on the object and won't drop it
>>> while
>>> you're using it.  I bet for every other spot in the API I can tell you
>>> from
>>> whom you're borrowing the reference.
>>
>>
>> Okay. Here's a test:
>>
>> PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
>> Return value: Borrowed reference.
>>
>> Presumably you own a reference to the list itself before you call
>> this, and the list has a reference to its items. But suppose another
>> thread clear()s the list immediately after you call this; whose
>> reference are you borrowing now? The list's is gone.
>>
>> Or is this another API that will have to change post gilectomy?
>>
> Couldn't the reference to the list also be borrowed?
>
> If you lookup something in a dict, it'll be a borrowed reference.
>
> If the dict is globals() and there's no GIL, another thread could delete the
> item while your code had the borrowed reference.
>
> It looks like there might be a lot that will need to changed post gilectomy!

If you're trying to manipulate a list, you should probably own a
reference to it. Otherwise, you're taking great risks. The trouble is
that there's time enough for a context switch between PyList_GetItem
and even an immediately-following incref, so you could have a junk
pointer despite your best efforts.

Contrast:

PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i)
Return value: New reference.

If you use sequence protocol, you are guaranteed to have a valid
reference to the object. To be sure, the item might no longer be in
the sequence by the time you look at it... but you know you're not
looking at junk memory. With PyList_GetItem, it would be possible for
you to have a dud pointer.

Also contrast:

PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
Return value: Borrowed reference.

As long as you own a reference to the tuple itself, you can be
confident that the borrowed reference will still be valid. There's no
way that the object's refcount can validly hit zero so long as the
tuple exists. (I'm ignoring PyTuple_SetItem here, which basically is
for initialization.) It's only with lists that you have this risk.

PyObject* PyDict_GetItem(PyObject *p, PyObject *key)
Return value: Borrowed reference.

... okay, it's only with lists and dictionaries. Here come the other
lot called 'Python', I guess.

ChrisA