PyObject_CallFunctionObjArgs segfaults

Fri Sep 30 16:34:07 EDT 2022

That's great.  It clarifies things a lot for me, particularly re ref count for new references.  I would have had trouble if I didn't decref it twice.  

Thanks very much once again.  


Sep 30, 2022, 12:18 by python at mrabarnett.plus.com:

> On 2022-09-30 17:02, Jen Kris wrote:
>
>>
>> Thanks very much for your detailed reply.  I have a few followup questions.
>>
>> You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because it has been incref'ed).  Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.”
>>
>> In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished with it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is no reason to leave them out.”  Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help.
>>
> It's OK to INCREF them, provided that you DECREF them when you no longer need them, and remember that if it's a "new reference" you'd need to DECREF it twice.
>
>> Also:
>>
>> What is a borrowed reference, and how does it effect reference counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count on the object, and obligates you to dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.”
>>
>
> A borrowed reference is when it hasn't been INCREFed.
>
> You can think of INCREFing as a way of indicating ownership, which is often shared ownership (refcount > 1). When you're borrowing a reference, you're using it temporarily, but not claiming ownership. When the last owner releases its ownership (DECREF reduces the refcount to 0), the object can be garbage collected.
>
> When, say, you lookup an attribute, or get an object from a list with PyList_GetItem, it won't have been INCREFed. You're using it temporarily, just borrowing a reference.
>
>>
>> What does it mean to steal a reference?  If a function steals a reference does it have to decref it without incref (because it’s stolen)?
>>
> When function steals a reference, it's claiming ownership but not INCREFing it.
>
>>
>> Finally, you said:
>>
>> if (pMod_random == 0x0){
>>     PyErr_Print();
>> Leaks here because of the refcount
>>
>> Assuming pMod_random is not null, why would this leak?
>>
> It's pName_random that's the leak.
>
> PyUnicode_FromString("random") will either create and return a new object for the string "random" (refcount == 1) or return a reference to an existing object (refcount > 1). You need to DECREF it before returning from the function.
>
> Suppose it created a new object. You call the function, it creates an object, you use it, then return from the function. The object still exists, but there's no reference to it. Now call the function again. It creates another object, you use it, then return from the function. You now have 2 objects with no reference to them.
>
>> Thanks again for your input on this question.
>>
>> Jen
>>
>>
>>
>> Sep 29, 2022, 17:33 by python at mrabarnett.plus.com:
>>
>>  On 2022-09-30 01:02, MRAB wrote:
>>
>>  On 2022-09-29 23:41, Jen Kris wrote:
>>
>>
>>  I just solved this C API problem, and I’m posting the
>>  answer to help anyone else who might need it.
>>
>>  [snip]
>>
>>  What I like to do is write comments that state which variables
>>  hold a reference, followed by '+' if it's a new reference
>>  (incref'ed) and '?' if it could be null. '+?' means that it's
>>  probably a new reference but could be null. Once I know that it's
>>  not null, I can remove the '?', and once I've decref'ed it (if
>>  required) and no longer need it, I remobe it from the comment.
>>
>>  Clearing up references, as soon as they're not needed, helps to
>>  keep the number of current references more manageable.
>>
>>
>>  int64_t Get_LibModules(int64_t * return_array) {
>>  PyObject * pName_random = PyUnicode_FromString("random");
>>  //> pName_random+?
>>  if (!pName_random) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> pName_random+
>>  PyObject * pMod_random = PyImport_Import(pName_random);
>>  //> pName_random+ pMod_random+?
>>  Py_DECREF(pName_random);
>>  //> pMod_random+?
>>  if (!pMod_random) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> pMod_random+
>>  PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>>  //> pMod_random+ pAttr_seed?
>>  if (!pAttr_seed) {
>>  Py_DECREF(pMod_random);
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> pMod_random+ pAttr_seed
>>  PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>>  "randrange");
>>  //> pMod_random+ pAttr_seed pAttr_randrange?
>>  Py_DECREF(pMod_random);
>>  //> pAttr_seed pAttr_randrange?
>>  if (!pAttr_randrange) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> pAttr_seed pAttr_randrange
>>  return_array[0] = (int64_t)pAttr_seed;
>>  return_array[1] = (int64_t)pAttr_randrange;
>>
>>  return 0;
>>  }
>>
>>  int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
>>  PyObject * value_ptr = PyLong_FromLong(value_1);
>>  //> value_ptr+?
>>  if (!!value_ptr) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> value_ptr+
>>  PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
>>  value_ptr, NULL);
>>  //> value_ptr+ p_seed_calc+?
>>  Py_DECREF(value_ptr);
>>  //> p_seed_calc+?
>>  if (!p_seed_calc) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> p_seed_calc+
>>  Py_DECREF(p_seed_calc);
>>  return 0;
>>  }
>>
>>  int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
>>  PyObject * value_ptr = PyLong_FromLong(value_1);
>>  //> value_ptr+?
>>  if (!value_ptr) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //> value_ptr+
>>  PyObject * p_randrange_calc =
>>  PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
>>  //> value_ptr+ p_randrange_calc+?
>>  Py_DECREF(value_ptr);
>>  //> p_randrange_calc+?
>>  if (!p_randrange_calc) {
>>  PyErr_Print();
>>  return 1;
>>  }
>>
>>  //Prepare return values
>>  //> p_randrange_calc+
>>  return_val = PyLong_AsLong(p_randrange_calc);
>>  Py_DECREF(p_randrange_calc);
>>
>>  return return_val;
>>  }
>>
>>  --     https://mail.python.org/mailman/listinfo/python-list
>>