[Python-Dev] PEP 454: Add a new tracemalloc module (final version)

[Victor Stinner]

2013/10/3 Victor Stinner <victor.stinner at gmail.com>:
> I worked on the implementation of the tracemalloc module and its PEP
> 454. I consider that this third version of the PEP is ready for a
> final review.
>
> What should be done to finish the PEP?

I splitted the module into two parts. Do you prefer the new API?

Module:
http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloc.html

Tool:
http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloctext.html


Full story:

Charles-François Natali made me some suggestions:

* why using a Task class instead of a thread?
* the module should be splitted into two parts: module (low-level
functions getting data), tool (display/compare data with colors)

I modified the implementation in a new "split" branch:

* I moved DisplayTop, DisplayTopTask, TakeSnapshotTask and Task
classes and the command line interface to a new "tracemalloctext"
module
* I rewrote Task into Python using a thread

The tool would not be integrated in Python, but hosted on PyPI. I just
kept it to get its documentation and show clearly how the split was
made.

tracemalloc API is simpler:
http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloc.html

API of the tool (using a temporary name "tracemalloctext" which may
change later):
http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloctext.html


I chose to keep Snapshot.top_by(), GroupedStats and StatsDiff in the
(tracemalloc) module. I prefer to "hide" internal structures of
tracemalloc (tracemalloc.get_stats(), tracemalloc.get_traces()), and
suggest to use higher level structures (GroupedStats, StatsDiff).
Internal structures are not hidden, they are even documented. But they
might change later, whereas GroupedStats structure should not change.

Or do you think that computing "top N allocations" is something
specific to a tool? Or each tool may implement this function
differently?


Using a thread has a drawback. Checking the memory threshold is
implemented using a short sleep (100 ms). If the memory peak is
shorter than the sleep, it will not be noticed. A workaround is to use
a threshold on the maximum traced memory. The advantage of the thread
is to have a better resolution than 1 second for the timer, and to not
have to get the current time at each memory allocation when a task is
scheduled.

The current implement (not splitted) uses Py_AddPendingCall(),
function already used by the C signal handler (signalmodule.c) to call
later the Python signal handler from the bytecode evaluation loop
(ceval.c).

Victor