embedding Python: how to avoid memory leaks?

[Andrew Trevorrow]

martin at v.loewis.de wrote:

> I could reproduce a memory leak with the code
> 
> #include <Python.h>
> int main()
> {
>   while(1){
>     Py_Initialize();
>     PyRun_SimpleString("execfile('foo.py')");
>     Py_Finalize();
>   }
> }
> 
> However, I could not reproduce a memory leak with the code
> 
> #include <Python.h>
> int main()
> {
>   Py_Initialize();
>   while(1){
>     PyRun_SimpleString("execfile('foo.py')");
>   }
>   Py_Finalize();
> }
> 
> So I recommend you do Py_Initialize only once. It is well-known
> that initializing the Python interpreter allocates memory that
> can never be freed, e.g. global variables in extension modules
> (there just isn't any API to tell all the modules to release their
> memory). So a cycle of Py_Initialize/Py_Finalize will certainly
> leak.

Surely that's a bug that should be fixed.  There should be some way
to tell Python "release all the memory you've ever allocated and
start again with a clean slate".

> OTOH, PyRun_SimpleString shouldn't leak, and didn't when I
> tried it.

Ok, after reading http://evanjones.ca/python-memory.html I think I
understand what's happening.  Apparently the Python memory allocator
never releases memory back to the OS!  So if a complicated script
happens to consume 100MB of interpreter memory then that amount is
no longer available to the app in which Python is embededded.
Even worse, if a script has a (Python) memory leak then there's
nothing the app can do about it.  It would be great if wrapping
each script inside Py_Initialize/Py_Finalize could avoid all that.

I've been told that the next version of Python will release memory,
so that's good news.  You can get it now if you're willing to build
Python from the latest source code.

Andrew