Puzzling behaviour of Py_IncRef

[Chris Angelico]

On Thu, Jan 20, 2022 at 1:22 AM Tony Flury <tony.flury at btinternet.com> wrote:
>
>
> On 19/01/2022 11:09, Chris Angelico wrote:
> > On Wed, Jan 19, 2022 at 10:00 PM Tony Flury via Python-list
> > <python-list at python.org> wrote:
> >> Extension function :
> >>
> >>      static PyObject *_Node_test_ref_count(PyObject *self)
> >>      {
> >>           printf("\nIncrementing ref count for self - just for the hell
> >>      of it\n");
> >>           printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
> >>           Py_INCREF(self);
> >>           printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
> >>           fflush(stdout);
> > At this point, the refcount has indeed been increased.
> >
> >>           return self;
> >>      }
> > And then you say "my return value is this object".
> >
> > The normal thing to do is to add a reference to whatever you're
> > returning. For instance, Py_RETURN_NONE will incref None and then
> > return it.
> >
> > So you're incrementing the refcount, then returning it without
> > incrementing the refcount. Your code is actually equivalent to "return
> > self".
> >
> > In order to actually leak a reference, you'd need to incref it twice.
> >
> > ChrisA
>
>
> Chris - I am still puzzled - does  doing 'return self' automatically
> decrement the ref count of the object ?, and why is that the desired
> behaviour ? Effectively it results in a decrement of two, since at the
> exit of the function the ref count is only 1 (as witnessed by the
> subsequent call to assertEqual).

Imagine you have a function which creates a brand new object. What
should its reference count be just before you return it? What should
the refcount be after it's been given to the caller? There has to be a
reference at all times, but only one, so that it can be properly
garbage collected; the act of returning it from a function has to
steal away a reference.

> (I am not suggesting that it should be changed - I understand that would
> be a breaking change !).
>
> You say I am returning it without incrementing, but I am explicitly
> incrementing it before the return.
>

Yes, you are incrementing it - but only once. The act of returning
something that you already have a reference to has to increment the
reference count and then return the thing. Picture this Python code:

_THING = object()
def get_thing():
    return _THING

When you call get_thing(), it needs to give you back a new reference
to that object. Written in C, that would have to increment the
refcount, then return the thing. (If you disassemble that code in
CPython, what you'll see is that it does a LOAD_GLOBAL, which does the
reference count incrementing, and then it returns what's on the top of
the stack. In C, you simply return the pointer, so you manually
increment the refcount first.)

In order to increment the reference count and keep it higher, you need
to (1) increment the reference count, then (2) return the object,
which consists of incrementing the reference count and returning the
pointer.

ChrisA