Puzzling behaviour of Py_IncRef

[Barry Scott]

> On 19 Jan 2022, at 10:57, Tony Flury via Python-list <python-list at python.org> wrote:
> 
> I am writing a C extension module for an AVL tree, and I am trying to ensure reference counting is done correctly. I was having a problem with the reference counting so I worked up this little POC of the problem, and I hope someone can explain this.

The ref counting in the C API not easy to get right. Sometimes you must inc some times you must not inc.
There are a lot of libraries that take that problem away from you.

For example I maintain PyCXX that is a C++ library that allows your drive the C API without needed to worry about ref counting.
There are lots of other libraries as well that aim to do the same thing.

http://cxx.sourceforge.net/ <http://cxx.sourceforge.net/>

Barry




> 
> Extension function :
> 
>   static PyObject *_Node_test_ref_count(PyObject *self)
>   {
>        printf("\nIncrementing ref count for self - just for the hell
>   of it\n");
>        printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
>        Py_INCREF(self);
>        printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
>        fflush(stdout);
>        return self;
>   }
> 
> As you can see this function purely increments the reference count of the instance.
> 
> /Note: I understand normally this would be the wrong this to do, but this is a POC of the issue, not live code. In the live code I am attaching a 2nd nodes to each other, and the live code therefore increments the ref-count for both objects - so even if the Python code deletes it's reference the reference count for the instance should still be 1 in order to ensure it doesn't get garbage collected./
> 
> This function is exposed as the test_ref method.
> 
> This is the test case :
> 
>     def test_000_009_test_ref_count(self):
>         node = _Node("Hello")
>         self.assertEqual(sys.getrefcount(node), 2)
>         node.test_ref()
>         self.assertEqual(sys.getrefcount(node), 3)
> 
> The output of this test case is :
> 
> test_000_009_test_ref_count (__main__.TestNode) ...
> Incrementing ref count for self - just for the hell of it
> 
>  before self has a ref count of 2
> 
>  after self has a ref count of 3
> FAIL
> 
> ======================================================================
> FAIL: test_000_009_test_ref_count (__main__.TestNode)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/tony/Development/python/orderedtree/tests/test_orderedtree.py", line 62, in test_000_009_test_ref_count
>     self.assertEqual(sys.getrefcount(node), 3)
> AssertionError: 2 != 3
> 
> So I understand why the first assert will be true - when the sys.getrefcount() function is called the ref count is incremented temporarily (as a borrowed reference), so there are now two references - the 'node' variable, and the borrowed reference in the function call.
> 
> We then call the 'test_ref' method, and again that call causes a borrowed reference (hence the ref count being 2 initially within the method). The 'test_ref' method increments the reference of the instance - as you can see from the output we now have a ref count of 3 - (that count is the 'node' variable in the test case, the borrowed reference due to the method call, and the artificial increment from the 'ref_test' method).
> 
> When the 'ref_test' method exits I would expect the ref count of the instance to now be 2 (one for the 'node' variable, and one as a result of the artificial increment increment').
> 
> I would therefore expect the 2nd assertEqual in the test case to succeed. - in this case the borrowed reference within sys.getfrefcount() should cause the count to be 3.
> 
> As you see though that 2nd assertEqual fails - suggesting that the refcount of 'node' is actually only 1 when the 'test_ref' method exits.
> 
> Can someone explain why the 'test_ref' method fails to change the refcount of the 'node' instance.
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list