PyObject_CallFunctionObjArgs segfaults

[MRAB]

On 2022-09-30 17:02, Jen Kris wrote:
>
> Thanks very much for your detailed reply.  I have a few followup 
> questions.
>
> You said, “Some functions return an object that has already been 
> incref'ed ("new reference"). This occurs when it has either created a 
> new object (the refcount will be 1) or has returned a pointer to an 
> existing object (the refcount will be > 1 because it has been 
> incref'ed).  Other functions return an object that hasn't been 
> incref'ed. This occurs when you're looking up something, for example, 
> looking at a member of a list or the value of an attribute.”
>
> In the official docs some functions show “Return value: New reference” 
> and others do not.  Is there any reason why I should not just INCREF 
> on every new object, regardless of whether it’s a new reference or 
> not, and DECREF when I am finished with it?  The answer at 
> https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to 
> says “With out-of-order execution, the INCREF/DECREF are basically 
> free operations, so performance is no reason to leave them out.”  
> Doing so means I don’t have to check each object to see if it needs to 
> be INCREF’d or not, and that is a big help.
>
It's OK to INCREF them, provided that you DECREF them when you no longer 
need them, and remember that if it's a "new reference" you'd need to 
DECREF it twice.
> Also:
>
> What is a borrowed reference, and how does it effect reference 
> counting?  According to 
> https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use 
> Py_INCREF on a borrowed PyObject pointer you already have. This 
> increments the reference count on the object, and obligates you to 
> dispose of it properly.”  So I guess it’s yes, but I’m confused by 
> “pointer you already have.”

A borrowed reference is when it hasn't been INCREFed.

You can think of INCREFing as a way of indicating ownership, which is 
often shared ownership (refcount > 1). When you're borrowing a 
reference, you're using it temporarily, but not claiming ownership. When 
the last owner releases its ownership (DECREF reduces the refcount to 
0), the object can be garbage collected.

When, say, you lookup an attribute, or get an object from a list with 
PyList_GetItem, it won't have been INCREFed. You're using it 
temporarily, just borrowing a reference.
>
> What does it mean to steal a reference?  If a function steals a 
> reference does it have to decref it without incref (because it’s stolen)?
When function steals a reference, it's claiming ownership but not 
INCREFing it.
>
> Finally, you said:
>
> if (pMod_random == 0x0){
>     PyErr_Print();
> Leaks here because of the refcount
>
> Assuming pMod_random is not null, why would this leak?
>
It's pName_random that's the leak.

PyUnicode_FromString("random") will either create and return a new 
object for the string "random" (refcount == 1) or return a reference to 
an existing object (refcount > 1). You need to DECREF it before 
returning from the function.

Suppose it created a new object. You call the function, it creates an 
object, you use it, then return from the function. The object still 
exists, but there's no reference to it. Now call the function again. It 
creates another object, you use it, then return from the function. You 
now have 2 objects with no reference to them.

> Thanks again for your input on this question.
>
> Jen
>
>
>
> Sep 29, 2022, 17:33 by python at mrabarnett.plus.com:
>
>     On 2022-09-30 01:02, MRAB wrote:
>
>         On 2022-09-29 23:41, Jen Kris wrote:
>
>
>             I just solved this C API problem, and I’m posting the
>             answer to help anyone else who might need it.
>
>     [snip]
>
>     What I like to do is write comments that state which variables
>     hold a reference, followed by '+' if it's a new reference
>     (incref'ed) and '?' if it could be null. '+?' means that it's
>     probably a new reference but could be null. Once I know that it's
>     not null, I can remove the '?', and once I've decref'ed it (if
>     required) and no longer need it, I remobe it from the comment.
>
>     Clearing up references, as soon as they're not needed, helps to
>     keep the number of current references more manageable.
>
>
>     int64_t Get_LibModules(int64_t * return_array) {
>     PyObject * pName_random = PyUnicode_FromString("random");
>     //> pName_random+?
>     if (!pName_random) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> pName_random+
>     PyObject * pMod_random = PyImport_Import(pName_random);
>     //> pName_random+ pMod_random+?
>     Py_DECREF(pName_random);
>     //> pMod_random+?
>     if (!pMod_random) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> pMod_random+
>     PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>     //> pMod_random+ pAttr_seed?
>     if (!pAttr_seed) {
>     Py_DECREF(pMod_random);
>     PyErr_Print();
>     return 1;
>     }
>
>     //> pMod_random+ pAttr_seed
>     PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>     "randrange");
>     //> pMod_random+ pAttr_seed pAttr_randrange?
>     Py_DECREF(pMod_random);
>     //> pAttr_seed pAttr_randrange?
>     if (!pAttr_randrange) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> pAttr_seed pAttr_randrange
>     return_array[0] = (int64_t)pAttr_seed;
>     return_array[1] = (int64_t)pAttr_randrange;
>
>     return 0;
>     }
>
>     int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
>     PyObject * value_ptr = PyLong_FromLong(value_1);
>     //> value_ptr+?
>     if (!!value_ptr) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> value_ptr+
>     PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
>     value_ptr, NULL);
>     //> value_ptr+ p_seed_calc+?
>     Py_DECREF(value_ptr);
>     //> p_seed_calc+?
>     if (!p_seed_calc) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> p_seed_calc+
>     Py_DECREF(p_seed_calc);
>     return 0;
>     }
>
>     int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
>     PyObject * value_ptr = PyLong_FromLong(value_1);
>     //> value_ptr+?
>     if (!value_ptr) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //> value_ptr+
>     PyObject * p_randrange_calc =
>     PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
>     //> value_ptr+ p_randrange_calc+?
>     Py_DECREF(value_ptr);
>     //> p_randrange_calc+?
>     if (!p_randrange_calc) {
>     PyErr_Print();
>     return 1;
>     }
>
>     //Prepare return values
>     //> p_randrange_calc+
>     return_val = PyLong_AsLong(p_randrange_calc);
>     Py_DECREF(p_randrange_calc);
>
>     return return_val;
>     }
>
>     -- 
>     https://mail.python.org/mailman/listinfo/python-list
>
>