[Python-Dev] PEP 454: add a new tracemalloc module (second round)
Victor Stinner
victor.stinner at gmail.com
Tue Sep 17 01:47:24 CEST 2013
Hi,
Thanks to the early remarks on the PEP 454, I redesigned and enhanced
the API of the new tracemalloc module. Changes:
* it is now possibility to record more than 1 frame per memory allocation
* add filters on filename and line number
* new GroupedStats and StatsDiff class to generate and compare statistics
* cumulative statistics
* display the traceback of a memory block
* almost all the code has unit tests
* better documentation
HTML version of the documentation:
http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html
The documentation contains output examples and a short tutorial
("Usage"), but also the documentation of the command line, which are
not included in the PEP.
HTML version of the PEP:
http://www.python.org/dev/peps/pep-0454/
Issue tracking the implementation:
http://bugs.python.org/issue18874
PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4
Abstract
========
Add a new ``tracemalloc`` module to trace memory blocks allocated by Python.
Rationale
=========
Common debug tools tracing memory allocations read the C filename and
number. Using such tool to analyze Python memory allocations does not
help because most memory block are allocated in the same C function,
in ``PyMem_Malloc()`` for example.
There are debug tools dedicated to the Python language like ``Heapy``
and ``PySizer``. These projects analyze objects type and/or content.
These tools are useful when most memory leaks are instances of the
same type and this type is only instancied in a few functions. The
problem is when the object type is very common like ``str`` or
``tuple``, and it is hard to identify where these objects are
instancied.
Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools cannot
be used on large applications with thousands of objects because the
diagram is too huge to be analyzed manually.
Proposal
========
Using the PEP 445, it becomes easy to setup an hook on Python memory
allocators. The hook can inspect the current Python frame to get the
Python filename and line number.
This PEP proposes to add a new ``tracemalloc`` module. It is a debug
tool to trace memory allocations made by Python. The module provides the
following information:
* Compute the differences between two snapshots to detect memory leaks
* Statistics on allocated memory blocks per filename and per line number:
total size, number and average size of allocated memory blocks
* For each allocated memory block: its size and the traceback where the block
was allocated
The API of the tracemalloc module is similar to the API of the
faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()``
functions, an environment variable (``PYTHONFAULTHANDLER`` and
``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X
faulthandler`` and ``-X tracemalloc``). See the
`documentation of the faulthandler module
<http://docs.python.org/dev/library/faulthandler.html>`_.
The tracemalloc module has been written for CPython. Other
implementations of Python may not provide it.
API
===
To trace most memory blocks allocated by Python, the module should be
enabled as early as possible by calling ``tracemalloc.enable()``
function, by setting the ``PYTHONTRACEMALLOC`` environment variable to
``1``, or by using ``-X tracemalloc`` command line option.
By default, the ``Trace.traceback`` attribute only stores one ``Frame``
instance per allocated memory block. Use ``set_traceback_limit()`` to
store more frames.
Functions
---------
``add_filter(filter)`` function:
Add a new filter on Python memory allocations, *filter* is a
``Filter`` instance.
All inclusive filters are applied at once, a memory allocation is
only ignored if no inclusive filter match its trace. A memory
allocation is ignored if at least one exclusive filter matchs its
trace.
The new filter is not applied on already collected traces. Use
``clear_traces()`` to ensure that all traces match the new filter.
``add_include_filter(filename: str, lineno: int=None, traceback:
bool=False)`` function:
Add an inclusive filter: helper for ``add_filter()`` creating a
``Filter`` instance with ``include`` attribute set to ``True``.
Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)``
only includes memory blocks allocated by the ``tracemalloc`` module.
``add_exclude_filter(filename: str, lineno: int=None, traceback:
bool=False)`` function:
Add an exclusive filter: helper for ``add_filter()`` creating a
``Filter`` instance with ``include`` attribute set to ``False``.
Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)``
ignores memory blocks allocated by the ``tracemalloc`` module.
``clear_filters()`` function:
Reset the filter list.
``clear_traces()`` function:
Clear all traces and statistics on Python memory allocations, and
reset the ``get_traced_memory()`` counter.
``disable()`` function:
Stop tracing Python memory allocations and stop the timer started by
``start_timer()``.
See also ``enable()`` and ``is_enabled()`` functions.
``enable()`` function:
Start tracing Python memory allocations.
See also ``disable()`` and ``is_enabled()`` functions.
``get_filters()`` function:
Get the filters on Python memory allocations as list of ``Filter``
instances.
``get_traceback_limit()`` function:
Get the maximum number of ``Frame`` instances stored in the
``traceback`` attribute of a ``Trace`` instance.
Use ``set_traceback_limit()`` to change the limit.
``get_object_address(obj)`` function:
Get the address of the memory block of the specified Python object.
``get_object_trace(obj)`` function:
Get the trace of a Python object *obj* as a ``Trace`` instance.
The function only returns the trace of the memory block directly
holding to object. The ``size`` attribute of the trace is smaller
than the total size of the object if the object is composed of more
than one memory block.
Return ``None`` if the ``tracemalloc`` module did not trace the
allocation of the object.
See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.
``get_process_memory()`` function:
Get the memory usage of the current process as a meminfo namedtuple
with two attributes:
* ``rss``: Resident Set Size in bytes
* ``vms``: size of the virtual memory in bytes
Return ``None`` if the platform is not supported.
``get_stats()`` function:
Get statistics on traced Python memory blocks as a dictionary
``{filename (str): {line_number (int): stats}}`` where *stats* in a
``TraceStats`` instance, *filename* and *line_number* can be
``None``.
Return an empty dictionary if the ``tracemalloc`` module is
disabled.
``get_traced_memory()`` function:
Get the total size of all traced memory blocks allocated by Python.
``get_tracemalloc_size()`` function:
Get the memory usage in bytes of the ``tracemalloc`` module.
``get_traces(obj)`` function:
Get all traces of Python memory allocations as a dictionary
``{address (int): trace}`` where *trace* is a ``Trace`` instance.
Return an empty dictionary if the ``tracemalloc`` module is
disabled.
``is_enabled()`` function:
``True`` if the ``tracemalloc`` module is tracing Python memory
allocations, ``False`` otherwise.
See also ``enable()`` and ``disable()`` functions.
``start_timer(delay: int, func: callable, args: tuple=(), kwargs:
dict={})`` function:
Start a timer calling ``func(*args, **kwargs)`` every *delay*
seconds. Enable the ``tracemalloc`` module if it is disabled. The
timer is based on the Python memory allocator, it is not real time.
*func* is called after at least *delay* seconds, it is not called
exactly after *delay* seconds if no Python memory allocation
occurred. The timer has a resolution of 1 second.
If the ``start_timer()`` function is called twice, previous
parameters are replaced. Call the ``stop_timer()`` function to stop
the timer.
The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods
use the ``start_timer()`` function to run regulary a task.
``set_traceback_limit(limit: int)`` function:
Set the maximum number of ``Frame`` instances stored in the
``traceback`` attribute of a ``Trace`` instance. Clear all traces
and statistics on Python memory allocations if the ``tracemalloc``
module is enabled,
Storing the traceback of each memory allocation has an important
overhead on the memory usage. Example with the Python test suite:
tracing all memory allocations increases the memory usage by
``+50%`` when storing only 1 frame and ``+150%`` when storing 10
frames. Use ``get_tracemalloc_size()`` to measure the overhead and
``add_filter()`` to select which memory allocations are traced.
Use ``get_traceback_limit()`` to get the current limit.
``stop_timer()`` function:
Stop the timer started by ``start_timer()``.
DisplayTop class
----------------
``DisplayTop()`` class:
Display the top of allocated memory blocks.
``display_snapshot(snapshot, count=10, group_by="filename_lineno",
cumulative=False, file=None)`` method:
Display a snapshot of memory blocks allocated by Python, *snapshot*
is a ``Snapshot`` instance.
``display_top_diff(top_diff, count=10, file=None)`` method:
Display differences between two ``GroupedStats`` instances,
*top_diff* is a ``StatsDiff`` instance.
``display_top_stats(top_stats, count=10, file=None)`` method:
Display the top of allocated memory blocks grouped by the
``group_by`` attribute of *top_stats*, *top_stats* is a
``GroupedStats`` instance.
``color`` attribute:
If ``True``, always use colors. If ``False``, never use colors. The
default value is ``None``: use colors if the *file* parameter is a
TTY device.
``compare_with_previous`` attribute:
If ``True`` (default value), compare with the previous snapshot. If
``False``, compare with the first snapshot.
``filename_parts`` attribute:
Number of displayed filename parts (int, default: ``3``). Extra
parts are replaced with ``'...'``.
``show_average`` attribute:
If ``True`` (default value), display the average size of memory blocks.
``show_count`` attribute:
If ``True`` (default value), display the number of allocated memory
blocks.
``show_size`` attribute:
If ``True`` (default value), display the size of memory blocks.
DisplayTopTask class
--------------------
``DisplayTopTask(count=10, group_by="filename_lineno",
cumulative=False, file=sys.stdout, user_data_callback=None)`` class:
Task taking temporary snapshots and displaying the top *count*
memory allocations grouped by *group_by*.
Call the ``start()`` method to start the task.
``display()`` method:
Take a snapshot and display the top *count* biggest allocated memory
blocks grouped by *group_by* using the ``display_top`` attribute.
Return the snapshot, a ``Snapshot`` instance.
``start(delay: int)`` method:
Start a task using the ``start_timer()`` function calling the
``display()`` method every *delay* seconds.
``stop()`` method:
Stop the task started by the ``start()`` method using the
``stop_timer()`` function.
``count`` attribute:
Maximum number of displayed memory blocks.
``cumulative`` attribute:
If ``True``, cumulate size and count of memory blocks of all frames
of each ``Trace`` instance, not only the most recent frame. The
default value is ``False``.
The option is ignored if the traceback limit is ``1``, see the
``get_traceback_limit()`` function.
``display_top`` attribute:
Instance of ``DisplayTop``.
``file`` attribute:
The top is written into *file*.
``group_by`` attribute:
Determine how memory allocations are grouped: see
``Snapshot.top_by`` for the available values.
``user_data_callback`` attribute:
Optional callback collecting user data (callable, default:
``None``). See ``Snapshot.create()``.
Filter class
------------
``Filter(include: bool, pattern: str, lineno: int=None, traceback:
bool=False)`` class:
Filter to select which memory allocations are traced. Filters can be
used to reduce the memory usage of the ``tracemalloc`` module, which
can be read using ``get_tracemalloc_size()``.
``match_trace(trace)`` method:
Return ``True`` if the ``Trace`` instance must be kept according to
the filter, ``False`` otherwise.
``match(filename: str, lineno: int)`` method:
Return ``True`` if the filename and line number must be kept
according to the filter, ``False`` otherwise.
``match_filename(filename: str)`` method:
Return ``True`` if the filename must be kept according to the
filter, ``False`` otherwise.
``match_lineno(lineno: int)`` method:
Return ``True`` if the line number must be kept according to the
filter, ``False`` otherwise.
``include`` attribute:
If *include* is ``True``, only trace memory blocks allocated in a
file with a name matching filename ``pattern`` at line number
``lineno``. If *include* is ``False``, ignore memory blocks
allocated in a file with a name matching filename :attr`pattern` at
line number ``lineno``.
``pattern`` attribute:
The filename *pattern* can contain one or many ``*`` joker
characters which match any substring, including an empty string. The
``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On
Windows, the comparison is case insensitive and the alternative
separator ``/`` is replaced with the standard separator ``\``.
``lineno`` attribute:
Line number (``int``). If is is ``None`` or lesser than ``1``, it
matches any line number.
``traceback`` attribute:
If *traceback* is ``True``, all frames of the ``traceback``
attribute of ``Trace`` instances are checked. If *traceback* is
``False``, only the most recent frame is checked.
This attribute only has an effect on the ``match_trace()`` method
and only if the traceback limit is greater than ``1``. See the
``get_traceback_limit()`` function.
Frame class
-----------
``Frame`` class:
Trace of a Python frame, used by ``Trace.traceback`` attribute.
``filename`` attribute:
Python filename, ``None`` if unknown.
``lineno`` attribute:
Python line number, ``None`` if unknown.
GroupedStats class
------------------
``GroupedStats(stats: dict, group_by: str, cumulative=False,
timestamp=None, process_memory=None, tracemalloc_size=None)`` class:
Top of allocated memory blocks grouped by on *group_by* as a
dictionary.
The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance.
``compare_to(old_stats: GroupedStats=None)`` method:
Compare to an older ``GroupedStats`` instance. Return a
``StatsDiff`` instance.
``cumulative`` attribute:
If ``True``, cumulate size and count of memory blocks of all frames
of ``Trace``, not only the most recent frame.
``group_by`` attribute:
Determine how memory allocations were grouped. The type of ``stats``
keys depends on *group_by*:
===================== ======================== ==============
group_by description key type
===================== ======================== ==============
``'filename'`` filename ``str``
``'filename_lineno'`` filename and line number ``(str, str)``
``'address'`` memory block address ``int``
===================== ======================== ==============
See the *group_by* parameter of the ``Snapshot.top_by()`` method.
``stats`` attribute:
Dictionary ``{key: stats}`` where the *key* type depends on the
``group_by`` attribute and *stats* type is ``TraceStats``.
``process_memory`` attribute:
Result of the ``get_process_memory()`` function, can be ``None``.
``timestamp`` attribute:
Creation date and time of the snapshot, ``datetime.datetime``
instance.
``tracemalloc_size`` attribute:
The memory usage in bytes of the ``tracemalloc`` module, result of
the ``get_tracemalloc_size()`` function.
Snapshot class
--------------
``Snapshot`` class:
Snapshot of memory blocks allocated by Python.
Use ``TakeSnapshot`` to take regulary snapshots.
``apply_filters(filters)`` method:
Apply a list filters on the ``traces`` and ``stats`` dictionaries,
*filters* is a list of ``Filter`` instances.
``create(\*, with_traces=False, with_stats=True,
user_data_callback=None)`` classmethod:
Take a snapshot of traces and/or statistics of allocated memory
blocks.
If *with_traces* is ``True``, ``get_traces()`` is called and its
result is stored in the ``traces`` attribute. This attribute
contains more information than ``stats`` and uses more memory and
more disk space. If *with_traces* is ``False``, ``traces`` is set to
``None``.
If *with_stats* is ``True``, ``get_stats()`` is called and its
result is stored in the ``Snapshot.stats`` attribute. If
*with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``.
*with_traces* and *with_stats* cannot be ``False`` at the same time.
*user_data_callback* is an optional callable object. Its result
should be serializable by the ``pickle`` module, or
``Snapshot.write()`` would fail. If *user_data_callback* is set, it
is called and the result is stored in the ``Snapshot.user_data``
attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``.
The ``tracemalloc`` module must be enabled to take a snapshot. See
the ``enable()`` function.
``load(filename)`` classmethod:
Load a snapshot from a file.
``top_by(group_by: str, cumulative: bool=False)`` method:
Compute top statistics grouped by *group_by* as a ``GroupedStats``
instance:
===================== ======================== ==============
group_by description key type
===================== ======================== ==============
``'filename'`` filename ``str``
``'filename_lineno'`` filename and line number ``(str, str)``
``'address'`` memory block address ``int``
===================== ======================== ==============
If *cumulative* is ``True``, cumulate size and count of memory
blocks of all frames of each ``Trace`` instance, not only the most
recent frame. The *cumulative* parameter is ignored if *group_by* is
``'address'`` or if the traceback limit is ``1``. See the
``traceback_limit`` attribute.
``write(filename)`` method:
Write the snapshot into a file.
``pid`` attribute:
Identifier of the process which created the snapshot, result of
``os.getpid()``.
``process_memory`` attribute:
Memory usage of the current process, result of the
``get_process_memory()`` function. It can be ``None``.
``stats`` attribute:
Statistics on traced Python memory, result of the ``get_stats()``
function, if ``create()`` was called with *with_stats* equals to
``True``, ``None`` otherwise.
``tracemalloc_size`` attribute:
The memory usage in bytes of the ``tracemalloc`` module, result of
the ``get_tracemalloc_size()`` function.
``traceback_limit`` attribute:
The maximum number of frames stored in the ``traceback`` attribute
of a ``Trace``, result of the ``get_traceback_limit()`` function.
``traces`` attribute:
Traces of Python memory allocations, result of the ``get_traces()``
function, if ``create()`` was called with *with_traces* equals to
``True``, ``None`` otherwise.
The ``traceback`` attribute of each ``Trace`` instance is limited to
``traceback_limit`` frames.
``timestamp`` attribute:
Creation date and time of the snapshot, ``datetime.datetime``
instance.
``user_data`` attribute:
Result of *user_data_callback* called in ``Snapshot.create()``
(default: ``None``).
StatsDiff class
---------------
``StatsDiff(differences, old_stats, new_stats)`` class:
Differences between two ``GroupedStats`` instances. By default, the
``differences`` list is unsorted: call ``sort()`` to sort it.
The ``GroupedStats.compare_to()`` method creates a ``StatsDiff``
instance.
``sort()`` method:
Sort the ``differences`` list from the biggest allocation to the
smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and
then by *key*.
``differences`` attribute:
Differences between ``old_stats`` and ``new_stats`` as a list of
``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*,
*size*, *count_diff* and *count* are ``int``. The key type depends
on the ``group_by`` attribute of ``new_stats``:
===================== ======================== ==============
group_by description key type
===================== ======================== ==============
``'filename'`` filename ``str``
``'filename_lineno'`` filename and line number ``(str, str)``
``'address'`` memory block address ``int``
===================== ======================== ==============
See the ``group_by`` attribute of the ``GroupedStats`` class.
``old_stats`` attribute:
Old ``GroupedStats`` instance, can be ``None``.
``new_stats`` attribute:
New ``GroupedStats`` instance.
Trace class
-----------
``Trace`` class:
Debug information of a memory block allocated by Python.
``size`` attribute:
Size in bytes of the memory block.
``traceback`` attribute:
Traceback where the memory block was allocated as a list of
``Frame`` instances, most recent first.
The list can be empty or incomplete if the ``tracemalloc`` module
was unable to retrieve the full traceback.
The traceback is limited to ``get_traceback_limit()`` frames. Use
``set_traceback_limit()`` to store more frames.
TraceStats class
----------------
``TraceStats`` class:
Statistics on Python memory allocations.
``size`` attribute:
Total size in bytes of allocated memory blocks.
``count`` attribute:
Number of allocated memory blocks.
Links
=====
tracemalloc:
* `#18874: Add a new tracemalloc module to trace Python
memory allocations <http://bugs.python.org/issue18874>`_
* `pytracemalloc on PyPI
<https://pypi.python.org/pypi/pytracemalloc>`_
Similar projects:
* `Meliae: Python Memory Usage Analyzer
<https://pypi.python.org/pypi/meliae>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
<http://guppy-pe.sourceforge.net/>`_
* `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_
* `pympler <http://code.google.com/p/pympler/>`_
* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of
the CherryPy memory leak debugger
* `objgraph <http://mg.pov.lt/objgraph/>`_
* `caulk <https://github.com/smartfile/caulk/>`_
Copyright
=========
This document has been placed into the public domain.
Victor
More information about the Python-Dev
mailing list