PyObject_CallFunctionObjArgs segfaults
MRAB
python at mrabarnett.plus.com
Thu Sep 29 20:02:23 EDT 2022
On 2022-09-29 23:41, Jen Kris wrote:
>
> I just solved this C API problem, and I’m posting the answer to help
> anyone else who might need it.
>
> The errors were:
>
> (1) we must call Py_INCREF on each object when it’s created.
>
Some functions return an object that has already been incref'ed ("new
reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the
refcount will be > 1 because it has been incref'ed).
Other functions return an object that hasn't been incref'ed. This occurs
when you're looking up something, for example, looking at a member of a
list or the value of an attribute.
> (2) in C_API_2 (see below) we don’t cast value_1 as I did before with
> PyObject * value_ptr = (PyObject * )value_1. Instead we use PyObject
> * value_ptr = PyLong_FromLong(value_1);
>
> (3) The command string to PyObject_CallFunctionObjArgs must be null
> terminated.
>
Always read the docs carefully!
> Here’s the revised code:
>
> First we load the modules, and increment the reference to each object:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
Possible null pointers here.
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
Py_INCREF will fail on a null pointer, which you haven't check for yet.
If they aren't null, then they're new references (incref'ed) that you've
incref'ed.
> if (pMod_random == 0x0){
> PyErr_Print();
Leaks here because of the refcount.
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
>
Possible null pointers here too.
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
Same issue as above.
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> Next we call a program to initialize the random number generator with
> random.seed(), and increment the reference to its return value
> p_seed_calc:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = PyLong_FromLong(value_1);
New reference.
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
>
> // _________
>
> if (p_seed_calc == 0x0){
> PyErr_Print();
Leak.
> return 1;}
>
> Py_INCREF(p_seed_calc);
>
p_seed_calc will be a new reference to Py_None, incref'ed again, so leak.
> return 0;
> }
>
> Now we call another program to get a random number:
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
> {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> PyObject * p_randrange_calc =
> PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
>
> if (p_randrange_calc == 0x0){
Leak.
> PyErr_Print();
> return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_randrange_calc);
>
Leak.
> return return_val;
> }
>
> That returns 28, which is what I get from the Python command line.
>
> Thanks again to MRAB for helpful comments.
>
> Jen
>
Getting refcounting right can be difficult!
>
> Sep 29, 2022, 15:31 by python at mrabarnett.plus.com:
>
> On 2022-09-29 21:47, Jen Kris wrote:
>
> To update my previous email, I found the problem, but I have a
> new problem.
>
> Previously I cast PyObject * value_ptr = (PyObject * )value_1
> but that's not correct. Instead I used PyObject * value_ptr =
> PyLong_FromLong(value_1) and that works. HOWEVER, while
> PyObject_CallFunctionObjArgs does work now, it returns -1,
> which is not the right answer for random.seed. I use "long
> return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
>
> random.seed returns None, so when you call
> PyObject_CallFunctionObjArgs it returns a new reference to Py_None.
>
> If you then pass to PyLong_AsLong a reference to something that's
> not a PyLong, it'll set an error and return -1.
>
> So my question is why do I get -1 as return value? When I
> query p_seed calc : get:
>
> (gdb) p p_seed_calc
> $2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>
>
> Exactly. It's Py_None, not a PyLong.
>
> Thanks again.
>
> Jen
>
>
>
>
> Sep 29, 2022, 13:02 by python-list at python.org:
>
> Thanks very much to @MRAB for taking time to answer. I changed my
> code to conform to your answer (as best I understand your comments
> on references), but I still get the same error. My comments
> continue below the new code immediately below.
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
> if (pMod_random == 0x0){
> PyErr_Print();
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random,
> "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
>
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = (PyObject * )value_1;
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
>
> if (p_seed_calc == 0x0){
> PyErr_Print();
> return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> return return_val;
> }
>
> So I incremented the reference to all objects in Get_LibModules,
> but I still get the same segfault at
> PyObject_CallFunctionObjArgs. Unfortunately, reference counting
> is not well documented so I’m not clear what’s wrong.
>
>
>
>
> Sep 29, 2022, 10:06 by python at mrabarnett.plus.com:
>
> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>
> Recently I completed a project where I used
> PyObject_CallFunctionObjArgs extensively with the NLTK
> library from a program written in NASM, with no problems.
> Now I am on a new project where I call the Python random
> library. I use the same setup as before, but I am getting
> a segfault with random.seed.
>
> At the start of the NASM program I call a C API program
> that gets PyObject pointers to “seed” and “randrange” in
> the same way as I did before:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Both PyUnicode_FromString and PyImport_Import return new
> references or null pointers.
>
> if (pMod_random == 0x0){
> PyErr_Print();
>
>
> You're leaking a reference here (pName_random).
>
> return 1;}
>
> PyObject * pAttr_seed =
> PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange =
> PyObject_GetAttrString(pMod_random, "randrange");
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
>
> You're leaking 2 references here (pName_random and pMod_random).
>
> return 0;
> }
>
> Later in the same program I call a C API program to call
> random.seed:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * p_seed_calc =
> PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>
>
> It's expecting all of the arguments to be PyObject*, but
> value_1 is Py_ssize_t instead of PyObject* (a pointer to a
> _Python_ int).
>
> The argument list must end with a null pointer.
>
> It returns a new reference or a null pointer.
>
>
> if (p_seed_calc == 0x0){
> PyErr_Print();
> return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> You're leaking a reference here (p_seed_calc).
>
> return return_val;
> }
>
> The first program correctly imports “random” and gets
> pointers to “seed” and “randrange.” I verified that the
> same pointer is correctly passed into C_API_2, and the
> seed value (1234) is passed as Py_ssize_t value_1. But I
> get this segfault:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
> ../Include/object.h:459
> 459 ../Include/object.h: No such file or directory.
>
> So I tried Py_INCREF in the first program:
>
> Py_INCREF(pMod_random);
> Py_INCREF(pAttr_seed);
>
> Then I moved Py_INCREF(pAttr_seed) to the second program.
> Same segfault.
>
> Finally, I initialized “random” and “seed” in the second
> program, where they are used. Same segfault.
>
> The segfault refers to Py_INCREF, so this seems to do with
> reference counting, but Py_INCREF didn’t solve it.
>
> I’m using Python 3.8 on Ubuntu.
>
> Thanks for any ideas on how to solve this.
>
> Jen
>
>
> -- https://mail.python.org/mailman/listinfo/python-list
>
>
> -- https://mail.python.org/mailman/listinfo/python-list
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
More information about the Python-list
mailing list