Python C API String Memory Consumption

[Floris Bruynooghe]

On Apr 7, 2:10 pm, John Machin <sjmac... at lexicon.net> wrote:
> On Apr 7, 9:19 pm, MRAB <goo... at mrabarnett.plus.com> wrote:
>
>
>
> > k3xji wrote:
> > > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> > > the problem resolved. However, I really cannot understand why the
> > > first version does not work. Here is the latest code that has no
> > > problems at all:
>
> > > static PyObject *
> > > penc(PyObject *self, PyObject *args)
> > > {
> > >    PyObject * result = NULL;
> > >    unsigned char *s= NULL;
> > >    unsigned char *buf = NULL;
> > >    unsigned int v,len,i = 0;
>
> > >    if (!PyArg_ParseTuple(args, "s#", &s, &len))
> > >         return NULL;
>
> > >    buf = (unsigned char *) PyMem_Malloc(len);
> > >    if (buf == NULL) {
> > >            PyErr_NoMemory();
> > >            return NULL;
> > >    }
>
> > >         /* string manipulation. */
>
> > >    result = PyString_FromStringAndSize((char *)buf, len);
> > >    PyMem_Free(buf);
> > >    return result;
> > > }

I assume you're doing a memcpy() somewhere in there...  This is also
safer then your first version since the python string can contain an
embeded \0 and the strdup() of the first version would not copy that.
But maybe you're sure your input doesn't have NULLs in them so it
might be fine.

>
> > In general I'd say don't mix your memory allocators. I don't know
> > whether CPython implements PyMem_Malloc using malloc,
>
> The fantastic manual (http://docs.python.org/c-api/
> memory.html#overview) says: """the C allocator and the Python memory
> manager ... implement different algorithms and operate on different
> heaps""".
>
> > but it's better to
> > stick with CPython's memory allocators when writing for CPython.
>
> for the reasons given in the last paragraph of the above reference.

That document explictly says you're allowed to use malloc() and free()
in extensions.  There is nothing wrong with allocating things on
different heaps, I've done and seen it many times and never had
trouble.

Why the original problem ocurred I don't understand either tough.

Regards
Floris