memory handling in C extension

[David Rushby]

Brad Hards <bhards at bigpond.net.au> wrote in message news:<mailman.1040784011.12455.python-list at python.org>...
> I am working on an wrapper for Service Location Protocol
> (RFC2608 and others), and am not certain I am handling
> memory correctly.

  I hope that a resident bot will correct me if I give you any bad
advice.

> Question 1. I assume that I am borrowing a reference to the arguments,
> if any, and that because the function call is synchronous, I am
> protected. True?

  True.  The function execution is synchronous with the code that
called it; that code necessarily holds references to the function
arguments.  See Python/C API Guide 1.2.1 Reference Counts:

"""
It is not necessary to increment an object's reference count for every
local variable that contains a pointer to an object. In theory, the
object's reference count goes up by one when the variable is made to
point to it and it goes down by one when the variable goes out of
scope. However, these two cancel each other out, so at the end the
reference count hasn't changed. The only real reason to use the
reference count is to prevent the object from being deallocated as
long as our variable is pointing to it. If we know that there is at
least one other reference to the object that lives at least as long as
our variable, there is no need to increment the reference count
temporarily. ***An important situation where this arises is in objects
that are passed as arguments to C functions in an extension module
that are called from Python; the call mechanism guarantees to hold a
reference to every argument for the duration of the call.***
"""


> Question 2. slp_getproperty() returns a string pointer, which will be free()'d
> by the python interpreter when no longer referenced. If I do
> something like:
> <sample>
> bradh at squirt python $ python
> Python 2.2.1 (#1, Oct  4 2002, 09:50:49)
> [GCC 2.95.3 20010315 (release)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import slp
> >>> slp.getproperty("net.slp.securityEnabled")
> 'true'
> </sample>
> then the string will be free()'d by the time I get the next prompt.
> That is bad, because the API says "do not free()". So I need to
> duplicate the string, and return the duplicate, which can be
> free()'d by the interpreter without violating the API. Is this true?
> Is there a better way?

  Be careful when you refer to the Python interpreter "free()ing"
memory; it doesn't necessarily use the standard free().  In Python
2.3, a custom memory allocator will be in place by default.  In C, use
PyMem_Malloc, PyMem_Free, and PyMem_Realloc to interface with the
custom Python allocator (if any).

  You mustn't allow memory allocated by an external library to be
collected by the Python interpreter in any case (even if premature
collection were not an issue, as it is here), because the external
library (SLP in this case) and Python might use different malloc/free
schemes.

> ... the API says "do not free()". So I need to
> duplicate the string, and return the duplicate, which can be
> free()'d by the interpreter without violating the API. Is this true?

  Yes, but your "Py_BuildValue("s", propvalue);" call is already
duplicating the memory.  "Py_BuildValue("s", propvalue)" is equivalent
to "PyString_FromString(propvalue)", which performs a memcpy on the
contents of the char* that you pass it.  The Python interpreter will
conceptually free this duplicated string if its reference count falls
low enough, though the interpreter actually caches some strings (in
such cases, the freeing would be purely conceptual).

> Is there a better way?

  If the source char* were guaranteed to be allocated using the same
memory manager as the Python core, you could do something like this:
------
PyObject *prop;
int n = strlen(propvalue);
/* Create an empty string that we can point to existing memory.
** (See caveats in documentation of PyString_FromStringAndSize.) */
prop = PyString_FromStringAndSize(NULL, n);
if (prop == NULL)
  return PyErr_NoMemory();

/* Make the internal buffer pointer of prop point to propvalue. */
PyString_AS_STRING(prop) = propvalue;

/* Artificially INCREF the string to prevent it from being 
** collected (PyString_FromStringAndSize already INCREFed it 
** once). */
Py_INCREF(prop);
------
  Then, at some later point when the string is finally ready to be
collected:
------
/* DECREF our artificially INCREFed string to cause it to be
** collected. */
Py_DECREF(prop);
------


> Question 3. slp_findscopes() would normally use SLPFree() on
> the return string. This is not equivalent to free() in the library I am
> using (it is a wrapper, but it also tracks memory usage). Is there
> any way to get python to use an alternative call to free memory
> (ie get SLPFree() rather than free())?

  I don't know.

> Or am I going to have to malloc() myself a new string, duplicate 
> the old string, and then SLPFree() the old string?

  See above; this is already happening with Py_BuildValue (except for
the SLPFree step, which should go after the Py_BuildValue call,
obviously).


> Question 4. The current slp_findscopes() returns a tuple consisting
> of a single string. That is OK for now, but I'd really like to split the
> string (it is a comma delimited set of scope names) into a tuple
> consisting of a variable number of strings - one per scope. I can't
> see how to do this with PyBuildValue().
> http://www.python.org/doc/current/ext/buildValue.html seems to
> imply I need to know how many elements are to be included. Any
> tips? Another Build function?

Perhaps you could use the string method 'split'?
------
/* scopes_unsplit = "a,b,c" */
PyObject *scopes_unsplit = PyString_FromString(scopes);
if (scopes_unsplit == NULL) {
  SLPFree(scopes);
  return PyErr_NoMemory();
}

/* scopes_split = scopes_unsplit.split(",") */
PyObject *scopes_split = PyEval_CallMethod(scopes_unsplit, "split",
"(Oc)", self, ',');

SLPFree(scopes);
Py_DECREF(scopes_unsplit);

return scopes_split;
------