Writing a C extension - borrowed references

[Paul Moore]

On 20 March 2018 at 16:22, Tom Evans via Python-list
<python-list at python.org> wrote:
> Hi all
>
> I'm writing my first C extension for Python here, and all is going
> well. However, I was reading [1], and the author there is advocating
> Py_INCREF 'ing *every* borrowed reference.
>
> Now, I get that if I do something to mutate and perhaps invalidate the
> PyObject that was borrowed I can get unpredictable results - eg, by
> getting a borrowed reference to a value from a dictionary, clearing
> the dictionary and then expecting to use the borrowed reference.
>
> However, if my code does not do things like that, is there a chance of
> a borrowed reference being invalidated that is not related to my use
> of the thing I borrowed it from? Is this cargo cult advice (sometimes
> the gods need this structure, and so to please the gods it must be
> everywhere), sensible belt and braces or needless overkill?

You *can* safely use a borrowed reference, but knowing when it's safe
to do so is definitely not easy. For example, in the presence of
threads, your borrowed reference could be invalidated on another
thread, so you have to be careful how the GIL is managed. It's often
very difficult to be sure that code doesn't call back into arbitrary
Python code, and you need to take care of that if you want to use the
borrowed reference. Conversely, the benefit of not doing an INCREF is
generally a pretty tiny performance improvement, which is only
worthwhile if you're in an extremely tight loop. So while it's OK to
use a borrowed reference, the reality is that if you don't fully
understand why you want to, and all the trade-offs, it's probably not
worth the risk.

Or, to put it another way, "if you need to ask, you can't afford to".

Paul