PyCObject & malloc creating memory leak

[Tom Conneely]

Thanks for your reply, you've given me plenty to think about

On Sep 29, 11:51 pm, Antoine Pitrou <solip... at pitrou.net> wrote:
>
> > My original plan was to have the data processing and data acquisition
> > functions running in separate processes, with a multiprocessing.Queue
> > for passing the raw data packets. The raw data is read in as a char*,
> > with a non constant length, hence I have allocated memory using
> > PyMem_Malloc and I am returning from the acquisition function a
> > PyCObject containing a pointer to this char* buffer, along with a
> > destructor.
>
> That sounds overkill, and I also wonder how you plan to pass that
> object in a multiprocessing Queue (which relies on objects being
> pickleable). Why don't you simply create a PyString object instead?

Could you elaborate on why you feel this is overkill? Also, your right
about
passing the PyCObjects through a Queue, something which I hadn't
really
considered, so I've switched to using python strings as you
suggested,
an overhead I hoped to avoid but you can't win them all I suppose.

> > So if I call these functions in a loop, e.g. The following will
> > generate ~10GB of data
>
> >     x = MyClass()
> >     for i in xrange(0, 10 * 2**20):
> >         c = x.malloc_buffer()
> >         x.retrieve_buffer(c)
>
> > All my memory disapears, until python crashes with a MemoryError. By
> > placing a print in the destructor function I know it's being called,
> > however it's not actually freeing the memory. So in short, what am I
> > doing wrong?
>
> Python returns memory to the OS by calling free(). Not all OSes
> actually relinquish memory when free() is called; some will simply set
> it aside for the next allocation.
> Another possible (and related) issue is memory fragmentation. Again, it
> depends on the memory allocator.

Yes, I know that's the case but the "freed" memory should be used for
the
next allocation, or atleast at some point before python runs out of
memory.
Anyway, this is besides the point as I've switched to using strings.

Again thanks for taking the time to help me out,
Tom Conneely