Writing a C extension - borrowed references

[Chris Angelico]

On Wed, Mar 21, 2018 at 3:22 AM, Tom Evans via Python-list
<python-list at python.org> wrote:
> Hi all
>
> I'm writing my first C extension for Python here, and all is going
> well. However, I was reading [1], and the author there is advocating
> Py_INCREF 'ing *every* borrowed reference.
>
> Now, I get that if I do something to mutate and perhaps invalidate the
> PyObject that was borrowed I can get unpredictable results - eg, by
> getting a borrowed reference to a value from a dictionary, clearing
> the dictionary and then expecting to use the borrowed reference.
>
> However, if my code does not do things like that, is there a chance of
> a borrowed reference being invalidated that is not related to my use
> of the thing I borrowed it from? Is this cargo cult advice (sometimes
> the gods need this structure, and so to please the gods it must be
> everywhere), sensible belt and braces or needless overkill?

Yep, that's cargo cult advice. If it were good advice, the Python API
would have been built to never return borrowed references, so you
always have to decref everything.

You should incref those sorts of borrowed references if you're going
to do something that could call arbitrary Python code and then
continue using the reference; but most of the time, that borrowed ref
is going to be used immediately and then finished with, and in those
very common situations, it's pointless to incref and decref. It's also
perfectly safe to use borrowed refs with immutable objects that you
have refs to; for instance, you can inspect the elements of a tuple
that way. The tuple MUST be holding references to all its members, and
the tuple itself can't be disposed of while you have a reference to
it, so you have a guarantee that those borrowed refs are valid.

"""The analogy in C is having two pointers to the same memory
location: so who is responsible for freeing the memory and what
happens if the other pointer tries to access that free’d memory?"""
That ignores the entire point of reference counting. By using a
borrowed reference, you're clearly declaring that the OTHER guy is
responsible for freeing it.

>From footnote 3:
"""Of course we never remove items in a list we merely decrement their
reference count (and if that hits zero then they are deleted)."""
This is conflating two separate concepts. You most assuredly CAN
remove items from a list, and the code in the footnote does exactly
that. What it won't necessarily do is destroy those objects.

"""An important takeaway here is that incrementing and decrementing
reference counts is a cheap operation but the consequences of getting
it wrong can be expensive. A precautionary approach in your code might
be to always increment borrowed references when they are instantiated
and then always decrement them before they go out of scope. That way
you incur two cheap operations but eliminate a vastly more expensive
one."""
Actually, managing ref counts isn't always cheap. So instead of just
always incrementing and always decrementing, how about this:

If you are going to call arbitrary code that could do anything at all,
guard around THAT CODE and nothing else, by protecting any borrowed
references.

People talk a lot about calling functions that could do literally
anything, but honestly, how often does that even happen? Not often
enough to justify cargo cult advice.

ChrisA