[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

[Nathaniel Smith]

On Mon, Oct 10, 2016 at 3:27 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nathaniel Smith wrote:
>>
>> IIRC to handle
>> this gilectomy adds per-object mutexes that you have to hold whenever
>> you're mucking around with that object's internals.
>
>
> What counts as "mucking around with the object's internals",
> though?
>
> If I do the C equivalent of:
>
>    x = somedict[5]
>    x.dosomething()
>
> am I mucking around with the internals of somedict?
> Intuitively, I would not think so. But the way
> PyDict_GetItem currently works, this would be
> dangerous.

I guess you already have to be aware of this to some extent -- e.g.,
the C version of this code is dangerous, even though in Python it
would be fine:

  x = somedict[5]
  del somedict
  x.dosomething()

But Larry pointed out to me separately that while the per-object locks
exist and can be used, the they don't help as much as you'd hope
because the goal is to *not* require C API users to change their code.
So it's safe to have two simultaneous calls to PyDict_Clear, or to any
other single API call, because each PyDict_Clear call wil *internally*
takes the lock, so they won't stomp on each other. But any code
sequence that makes multiple C API calls and assumes that the world
won't shift under its feet becomes problematic. E.g. pre-GILectomy you
can do

Py_ssize_t count = PyList_Size(list);
for (int i = 0; i < count; ++i) {
    PyObject *obj = PyList_Get_Item(list, i);
    /* code which blithely proceeds without checking for obj == NULL */
    ...
}

but post-GILectomy, even putting aside the borrowing issues, this code
might segfault if someone shrinks the list when you aren't looking. I
can't see any way to avoid breaking the API here -- it seems like
callers will just have to take a lock or otherwise update their code.

I guess you can think of borrowed references as a particularly common
and insidious form of this problem -- so the tactical question is
whether it's better to look for a general solution like teaching
everyone to take locks, or try to attack borrowed references
specifically.

(Full API compatibility is never going to be preserved with the
GILectomy anyway -- if nothing else C extensions currently use the GIL
to protect internal globals, and that will have to be replaced by some
kind of more specific locking. So the open question is more like, can
the porting burden be kept low enough to make it viable.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org