Hooking into Python's memory management

[Terry Reedy]

On 5/4/2011 12:51 PM, Daniel Neilson wrote:
>
> Hello,
> I'm hoping that there will be someone here with sufficient expertise to
> answer a question on Python 3 for me.
>
> I work in the Computer Science department at a large Canadian
> University. We are currently doing a feasibility analysis for switching
> to using Python in our first year major-stream courses.

If you did, I believe you would be following in the footsteps of MIT.

> Part of our first year curriculum requires that students be exposed to
> explicit dynamic memory allocation in the form of C++'s new/delete, C's
> malloc/free, etc. I realize that Python is garbage collected, and that
> there is never a need to explicitly allocate & deallocate objects.
> However, I am trying to determine whether or not it is possible to
> simulate that behaviour within Python via a module for the purposes of
> instruction.

The Python ctypes module allows one to invoke compiled C (or C++, I 
presume) functions in shared libraries (.dll on Windows, .so on *Nix).

> For these purposes, I would like to know whether it is possible within
> Python 3 to write a Python-only module that, essentially, hooks into the
> "constructor" and "destructor" of many of the Python built-in types

Python is compiled as a small startup executable (<30 KB). The builtins 
are all in shared libraries that you can access with ctypes. The 
functions in those libraries are documented in the Python/C API manual.

> (specifically list, dictionary, set, tuple, and string) so that the
> module can:
> 1) Maintain a list of object id()'s for objects that have been created.
> Ideally, this list would also contain the file & line number where the
> object was created.
> 2) Provide a "deallocate" function that will remove a given object's
> id() from the list from (1).
> 3) Print out an error message if the python script terminates with a
> non-empty list from (1). Preferably with a list of objects that are
> still "allocated."

I presume you can do all of this easily. For point 3, a script can 
register an 'atexit' function. As a sidenote, using ctypes 'allows' one 
to crash (segfault, bluescreen) a program just like when using C ;-). It 
thus voids the usual guarantee that one cannot do that.

> Baring a Python-only module, would this behaviour be possible to add via
> a C-language module?

I do not think you will *need* to do this, though you might eventually 
decide to make a custom library with just the functions you want, with 
the names you want.

> A module that hooked in to all memory allocation, and inspected the type
> of the object being allocated to conditionally add the object's id() to
> the list would also suffice.
>
> In either case, if such a module is possible, any pointers you could
> provide regarding how to implement such a module would be appreciated.

I hope the above helps. I think Python is a great learning language.

-- 
Terry Jan Reedy