[Python-checkins] peps: PEP 454: update the API to its implementation
victor.stinner
python-checkins at python.org
Thu Oct 3 16:03:07 CEST 2013
http://hg.python.org/peps/rev/67ecc9e4a969
changeset: 5162:67ecc9e4a969
user: Victor Stinner <victor.stinner at gmail.com>
date: Thu Oct 03 16:00:07 2013 +0200
summary:
PEP 454: update the API to its implementation
* remove Frame, Trace and TraceStats classes
* add Metric class
* replace start_timer()/stop_timer() with a new Task class
* Task can now be scheduled using a threshold on the size of the traced memory,
and only called repeat times
* add DisplayTop.display() method
* a lot of other (minor) changes
files:
pep-0454.txt | 1094 ++++++++++++++++++++++++-------------
1 files changed, 696 insertions(+), 398 deletions(-)
diff --git a/pep-0454.txt b/pep-0454.txt
--- a/pep-0454.txt
+++ b/pep-0454.txt
@@ -71,150 +71,69 @@
API
===
-To trace most memory blocks allocated by Python, the module should be
-enabled as early as possible by calling ``tracemalloc.enable()``
-function, by setting the ``PYTHONTRACEMALLOC`` environment variable to
-``1``, or by using ``-X tracemalloc`` command line option.
+To trace most memory blocks allocated by Python, the module should be enabled
+as early as possible by setting the ``PYTHONTRACEMALLOC`` environment
+variable to ``1``, or by using ``-X tracemalloc`` command line
+option. The ``tracemalloc.enable`` function can also be called to start
+tracing Python memory allocations.
-By default, the ``Trace.traceback`` attribute only stores one ``Frame``
-instance per allocated memory block. Use ``set_traceback_limit()`` to
-store more frames.
+By default, a trace of an allocated memory block only stores one frame. Use the
+``set_traceback_limit`` function to store more frames.
+Python memory blocks allocated in the ``tracemalloc`` module are ignored by
+default using a filter. Use the ``clear_filters`` function to see trace
+also these memory allocations.
-Functions
----------
+At fork, the module is automatically disabled in the child process.
-``add_filter(filter)`` function:
- Add a new filter on Python memory allocations, *filter* is a
- ``Filter`` instance.
+Main Functions
+--------------
- All inclusive filters are applied at once, a memory allocation is
- only ignored if no inclusive filter match its trace. A memory
- allocation is ignored if at least one exclusive filter matchs its
- trace.
+``cancel_tasks()`` function:
- The new filter is not applied on already collected traces. Use
- ``clear_traces()`` to ensure that all traces match the new filter.
+ Cancel all scheduled tasks.
-
-``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
-
- Add an inclusive filter: helper for ``add_filter()`` creating a
- ``Filter`` instance with ``include`` attribute set to ``True``.
-
- Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)``
- only includes memory blocks allocated by the ``tracemalloc`` module.
-
-
-``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
-
- Add an exclusive filter: helper for ``add_filter()`` creating a
- ``Filter`` instance with ``include`` attribute set to ``False``.
-
- Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)``
- ignores memory blocks allocated by the ``tracemalloc`` module.
-
-
-``clear_filters()`` function:
-
- Reset the filter list.
+ See also the ``get_tasks`` function.
``clear_traces()`` function:
- Clear all traces and statistics on Python memory allocations, and
- reset the ``get_traced_memory()`` counter.
+ Clear all traces and statistics on Python memory allocations, and reset the
+ ``get_arena_size`` and ``get_traced_memory`` counters.
``disable()`` function:
- Stop tracing Python memory allocations and stop the timer started by
- ``start_timer()``.
+ Stop tracing Python memory allocations and cancel scheduled tasks.
- See also ``enable()`` and ``is_enabled()`` functions.
+ See also ``enable`` and ``is_enabled`` functions.
``enable()`` function:
- Start tracing Python memory allocations.
+ Start tracing Python memory allocations.
- See also ``disable()`` and ``is_enabled()`` functions.
+ At fork, the module is automatically disabled in the child process.
-
-``get_filters()`` function:
-
- Get the filters on Python memory allocations as list of ``Filter``
- instances.
-
-
-``get_traceback_limit()`` function:
-
- Get the maximum number of ``Frame`` instances stored in the
- ``traceback`` attribute of a ``Trace`` instance.
-
- Use ``set_traceback_limit()`` to change the limit.
-
-
-``get_object_address(obj)`` function:
-
- Get the address of the memory block of the specified Python object.
-
-
-``get_object_trace(obj)`` function:
-
- Get the trace of a Python object *obj* as a ``Trace`` instance.
-
- The function only returns the trace of the memory block directly
- holding to object. The ``size`` attribute of the trace is smaller
- than the total size of the object if the object is composed of more
- than one memory block.
-
- Return ``None`` if the ``tracemalloc`` module did not trace the
- allocation of the object.
-
- See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.
-
-
-``get_process_memory()`` function:
-
- Get the memory usage of the current process as a meminfo namedtuple
- with two attributes:
-
- * ``rss``: Resident Set Size in bytes
- * ``vms``: size of the virtual memory in bytes
-
- Return ``None`` if the platform is not supported.
+ See also ``disable`` and ``is_enabled`` functions.
``get_stats()`` function:
- Get statistics on traced Python memory blocks as a dictionary
- ``{filename (str): {line_number (int): stats}}`` where *stats* in a
- ``TraceStats`` instance, *filename* and *line_number* can be
- ``None``.
+ Get statistics on traced Python memory blocks as a dictionary ``{filename
+ (str): {line_number (int): stats}}`` where *stats* in a
+ ``(size: int, count: int)`` tuple, *filename* and *line_number* can
+ be ``None``.
- Return an empty dictionary if the ``tracemalloc`` module is
- disabled.
+ Return an empty dictionary if the ``tracemalloc`` module is disabled.
+ See also the ``get_traces`` function.
-``get_traced_memory()`` function:
- Get the total size of all traced memory blocks allocated by Python.
+``get_tasks()`` function:
-
-``get_tracemalloc_size()`` function:
-
- Get the memory usage in bytes of the ``tracemalloc`` module.
-
-
-``get_traces(obj)`` function:
-
- Get all traces of Python memory allocations as a dictionary
- ``{address (int): trace}`` where *trace* is a ``Trace`` instance.
-
- Return an empty dictionary if the ``tracemalloc`` module is
- disabled.
+ Get the list of scheduled tasks, list of ``Task`` instances.
``is_enabled()`` function:
@@ -222,485 +141,864 @@
``True`` if the ``tracemalloc`` module is tracing Python memory
allocations, ``False`` otherwise.
- See also ``enable()`` and ``disable()`` functions.
+ See also ``enable`` and ``disable`` functions.
-``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function:
+Trace Functions
+---------------
- Start a timer calling ``func(*args, **kwargs)`` every *delay*
- seconds. Enable the ``tracemalloc`` module if it is disabled. The
- timer is based on the Python memory allocator, it is not real time.
- *func* is called after at least *delay* seconds, it is not called
- exactly after *delay* seconds if no Python memory allocation
- occurred. The timer has a resolution of 1 second.
+``get_traceback_limit()`` function:
- If the ``start_timer()`` function is called twice, previous
- parameters are replaced. Call the ``stop_timer()`` function to stop
- the timer.
+ Get the maximum number of frames stored in the traceback of a trace of a
+ memory block.
- The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods
- use the ``start_timer()`` function to run regulary a task.
+ Use the ``set_traceback_limit`` function to change the limit.
-``set_traceback_limit(limit: int)`` function:
+``get_object_address(obj)`` function:
- Set the maximum number of ``Frame`` instances stored in the
- ``traceback`` attribute of a ``Trace`` instance. Clear all traces
- and statistics on Python memory allocations if the ``tracemalloc``
- module is enabled,
+ Get the address of the memory block of the specified Python object.
- Storing the traceback of each memory allocation has an important
- overhead on the memory usage. Example with the Python test suite:
- tracing all memory allocations increases the memory usage by
- ``+50%`` when storing only 1 frame and ``+150%`` when storing 10
- frames. Use ``get_tracemalloc_size()`` to measure the overhead and
- ``add_filter()`` to select which memory allocations are traced.
+ A Python object can be composed by multiple memory blocks, the function only
+ returns the address of the main memory block.
- Use ``get_traceback_limit()`` to get the current limit.
+ See also ``get_object_trace`` and ``gc.get_referrers`` functions.
-``stop_timer()`` function:
+``get_object_trace(obj)`` function:
- Stop the timer started by ``start_timer()``.
+ Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple
+ where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples,
+ *filename* and *lineno* can be ``None``.
+ The function only returns the trace of the main memory block of the object.
+ The *size* of the trace is smaller than the total size of the object if the
+ object is composed by more than one memory block.
-DisplayTop class
+ Return ``None`` if the ``tracemalloc`` module did not trace the
+ allocation of the object.
+
+ See also ``get_object_address``, ``get_trace``, ``get_traces``,
+ ``gc.get_referrers`` and ``sys.getsizeof`` functions.
+
+
+``get_trace(address)`` function:
+
+ Get the trace of a memory block as a ``(size: int, traceback)`` tuple where
+ *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples,
+ *filename* and *lineno* can be ``None``.
+
+ Return ``None`` if the ``tracemalloc`` module did not trace the
+ allocation of the memory block.
+
+ See also ``get_object_trace``, ``get_stats`` and ``get_traces``
+ functions.
+
+
+``get_traces()`` function:
+
+ Get all traces of Python memory allocations as a dictionary ``{address
+ (int): trace}`` where *trace* is a
+ ``(size: int, traceback)`` and *traceback* is a list of
+ ``(filename: str, lineno: int)``.
+ *traceback* can be empty, *filename* and *lineno* can be None.
+
+ Return an empty dictionary if the ``tracemalloc`` module is disabled.
+
+ See also ``get_object_trace``, ``get_stats`` and ``get_trace``
+ functions.
+
+
+``set_traceback_limit(nframe: int)`` function:
+
+ Set the maximum number of frames stored in the traceback of a trace of a
+ memory block.
+
+ Storing the traceback of each memory allocation has an important overhead on
+ the memory usage. Use the ``get_tracemalloc_memory`` function to measure
+ the overhead and the ``add_filter`` function to select which memory
+ allocations are traced.
+
+ Use the ``get_traceback_limit`` function to get the current limit.
+
+
+Filter Functions
----------------
+``add_filter(filter)`` function:
+
+ Add a new filter on Python memory allocations, *filter* is a ``Filter``
+ instance.
+
+ All inclusive filters are applied at once, a memory allocation is only
+ ignored if no inclusive filter match its trace. A memory allocation is
+ ignored if at least one exclusive filter matchs its trace.
+
+ The new filter is not applied on already collected traces. Use the
+ ``clear_traces`` function to ensure that all traces match the new
+ filter.
+
+``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
+
+ Add an inclusive filter: helper for the ``add_filter`` method creating a
+ ``Filter`` instance with the ``Filter.include`` attribute set to
+ ``True``.
+
+ Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only
+ includes memory blocks allocated by the ``tracemalloc`` module.
+
+
+``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
+
+ Add an exclusive filter: helper for the ``add_filter`` method creating a
+ ``Filter`` instance with the ``Filter.include`` attribute set to
+ ``False``.
+
+ Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores
+ memory blocks allocated by the ``tracemalloc`` module.
+
+
+``clear_filters()`` function:
+
+ Reset the filter list.
+
+ See also the ``get_filters`` function.
+
+
+``get_filters()`` function:
+
+ Get the filters on Python memory allocations as list of ``Filter``
+ instances.
+
+ See also the ``clear_filters`` function.
+
+
+Metric Functions
+----------------
+
+The following functions can be used to add metrics to a snapshot, see
+the ``Snapshot.add_metric`` method.
+
+``get_allocated_blocks()`` function:
+
+ Get the current number of allocated memory blocks.
+
+
+``get_arena_size()`` function:
+
+ Get the size in bytes of traced arenas.
+
+ See also the ``get_pymalloc_stats`` function.
+
+
+``get_process_memory()`` function:
+
+ Get the memory usage of the current process as a ``(rss: int, vms: int)``
+ tuple, *rss* is the "Resident Set Size" in bytes and *vms* is the size of
+ the virtual memory in bytes
+
+ Return ``None`` if the platform is not supported.
+
+
+``get_pymalloc_stats()`` function:
+
+ Get statistics on the ``pymalloc`` allocator as a dictionary.
+
+ +---------------------+-------------------------------------------------------+
+ | Key | Description |
+ +=====================+=======================================================+
+ | ``alignment`` | Alignment of addresses returned to the user. |
+ +---------------------+-------------------------------------------------------+
+ | ``threshold`` | Small block threshold in bytes: pymalloc uses |
+ | | PyMem_RawMalloc() for allocation greater than |
+ | | threshold. |
+ +---------------------+-------------------------------------------------------+
+ | ``nalloc`` | Number of times object malloc called |
+ +---------------------+-------------------------------------------------------+
+ | ``arena_size`` | Arena size in bytes |
+ +---------------------+-------------------------------------------------------+
+ | ``total_arenas`` | Number of calls to new_arena(): total number of |
+ | | allocated arenas, including released arenas |
+ +---------------------+-------------------------------------------------------+
+ | ``max_arenas`` | Maximum number of arenas |
+ +---------------------+-------------------------------------------------------+
+ | ``arenas`` | Number of arenas currently allocated |
+ +---------------------+-------------------------------------------------------+
+ | ``allocated_bytes`` | Number of bytes in allocated blocks |
+ +---------------------+-------------------------------------------------------+
+ | ``available_bytes`` | Number of bytes in available blocks in used pools |
+ +---------------------+-------------------------------------------------------+
+ | ``pool_size`` | Pool size in bytes |
+ +---------------------+-------------------------------------------------------+
+ | ``free_pools`` | Number of unused pools |
+ +---------------------+-------------------------------------------------------+
+ | ``pool_headers`` | Number of bytes wasted in pool headers |
+ +---------------------+-------------------------------------------------------+
+ | ``quantization`` | Number of bytes in used and full pools wasted due to |
+ | | quantization, i.e. the necessarily leftover space at |
+ | | the ends of used and full pools. |
+ +---------------------+-------------------------------------------------------+
+ | ``arena_alignment`` | Number of bytes for arena alignment padding |
+ +---------------------+-------------------------------------------------------+
+
+ The function is not available if Python is compiled without ``pymalloc``.
+
+ See also ``get_arena_size`` and ``sys._debugmallocstats`` functions.
+
+
+``get_traced_memory()`` function:
+
+ Get the current size and maximum size of memory blocks traced by the
+ ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.
+
+
+``get_tracemalloc_memory()`` function:
+
+ Get the memory usage in bytes of the ``tracemalloc`` module as a
+ tuple: ``(size: int, free: int)``.
+
+ * *size*: total size of bytes allocated by the module,
+ including *free* bytes
+ * *free*: number of free bytes available to store data
+
+
+``get_unicode_interned()`` function:
+
+ Get the size in bytes and the length of the dictionary of Unicode interned
+ strings as a ``(size: int, length: int)`` tuple.
+
+
+DisplayTop
+----------
+
``DisplayTop()`` class:
Display the top of allocated memory blocks.
-``display_snapshot(snapshot, count=10, group_by="filename_lineno", cumulative=False, file=None)`` method:
+``display(count=10, group_by="line", cumulative=False, file=None, callback=None)`` method:
- Display a snapshot of memory blocks allocated by Python, *snapshot*
- is a ``Snapshot`` instance.
+ Take a snapshot and display the top *count* biggest allocated memory
+ blocks grouped by *group_by*.
+
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the newly
+ created snapshot instance. Use the ``Snapshot.add_metric`` method to
+ add new metric.
+
+ Return the snapshot, a ``Snapshot`` instance.
+
+``display_snapshot(snapshot, count=10, group_by="line", cumulative=False, file=None)`` method:
+
+ Display a snapshot of memory blocks allocated by Python, *snapshot* is a
+ ``Snapshot`` instance.
``display_top_diff(top_diff, count=10, file=None)`` method:
- Display differences between two ``GroupedStats`` instances,
- *top_diff* is a ``StatsDiff`` instance.
+ Display differences between two ``GroupedStats`` instances,
+ *top_diff* is a ``StatsDiff`` instance.
``display_top_stats(top_stats, count=10, file=None)`` method:
- Display the top of allocated memory blocks grouped by the
- ``group_by`` attribute of *top_stats*, *top_stats* is a
- ``GroupedStats`` instance.
+ Display the top of allocated memory blocks grouped by the
+ ``GroupedStats.group_by`` attribute of *top_stats*, *top_stats* is a
+ ``GroupedStats`` instance.
+
+``average`` attribute:
+
+ If ``True`` (default value), display the average size of memory blocks.
``color`` attribute:
- If ``True``, always use colors. If ``False``, never use colors. The
- default value is ``None``: use colors if the *file* parameter is a
- TTY device.
+ If ``True``, always use colors. If ``False``, never use colors. The
+ default value is ``None``: use colors if the *file* parameter is a TTY
+ device.
-``compare_with_previous`` attribute:
+``compare_to_previous`` attribute:
- If ``True`` (default value), compare with the previous snapshot. If
- ``False``, compare with the first snapshot.
+ If ``True`` (default value), compare to the previous snapshot. If
+ ``False``, compare to the first snapshot.
+
+``count`` attribute:
+
+ If ``True`` (default value), display the number of allocated memory
+ blocks.
``filename_parts`` attribute:
- Number of displayed filename parts (int, default: ``3``). Extra
- parts are replaced with ``'...'``.
+ Number of displayed filename parts (int, default: ``3``). Extra parts
+ are replaced with ``'...'``.
-``show_average`` attribute:
+``metrics`` attribute:
- If ``True`` (default value), display the average size of memory blocks.
+ If ``True`` (default value), display metrics: see
+ ``Snapshot.metrics``.
-``show_count`` attribute:
+``previous_top_stats`` attribute:
- If ``True`` (default value), display the number of allocated memory
- blocks.
+ Previous ``GroupedStats`` instance, or first ``GroupedStats``
+ instance if ``compare_to_previous`` is ``False``, used to display the
+ differences between two snapshots.
-``show_size`` attribute:
+``size`` attribute:
- If ``True`` (default value), display the size of memory blocks.
+ If ``True`` (default value), display the size of memory blocks.
-DisplayTopTask class
---------------------
+DisplayTopTask
+--------------
-``DisplayTopTask(count=10, group_by="filename_lineno", cumulative=False, file=sys.stdout, user_data_callback=None)`` class:
+``DisplayTopTask(count=10, group_by="line", cumulative=False, file=sys.stdout, callback=None)`` class:
- Task taking temporary snapshots and displaying the top *count*
- memory allocations grouped by *group_by*.
+ Task taking temporary snapshots and displaying the top *count* memory
+ allocations grouped by *group_by*.
- Call the ``start()`` method to start the task.
+ ``DisplayTopTask`` is based on the ``Task`` class and so inherit
+ all attributes and methods, especially:
+
+ * ``Task.cancel``
+ * ``Task.schedule``
+ * ``Task.set_delay``
+ * ``Task.set_memory_threshold``
+
+ Modify the ``display_top`` attribute to customize the display.
``display()`` method:
- Take a snapshot and display the top *count* biggest allocated memory
- blocks grouped by *group_by* using the ``display_top`` attribute.
+ Take a snapshot and display the top ``count`` biggest allocated
+ memory blocks grouped by ``group_by`` using the ``display_top``
+ attribute.
- Return the snapshot, a ``Snapshot`` instance.
+ Return the snapshot, a ``Snapshot`` instance.
-``start(delay: int)`` method:
+``callback`` attribute:
- Start a task using the ``start_timer()`` function calling the
- ``display()`` method every *delay* seconds.
-
-``stop()`` method:
-
- Stop the task started by the ``start()`` method using the
- ``stop_timer()`` function.
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the newly
+ created snapshot instance. Use the ``Snapshot.add_metric`` method to
+ add new metric.
``count`` attribute:
- Maximum number of displayed memory blocks.
+ Maximum number of displayed memory blocks.
``cumulative`` attribute:
- If ``True``, cumulate size and count of memory blocks of all frames
- of each ``Trace`` instance, not only the most recent frame. The
- default value is ``False``.
+ If ``True``, cumulate size and count of memory blocks of all frames of
+ each trace, not only the most recent frame. The default value is
+ ``False``.
- The option is ignored if the traceback limit is ``1``, see the
- ``get_traceback_limit()`` function.
+ The option is ignored if the traceback limit is less than ``2``, see
+ the ``get_traceback_limit`` function.
``display_top`` attribute:
- Instance of ``DisplayTop``.
+ Instance of ``DisplayTop``.
``file`` attribute:
- The top is written into *file*.
+ The top is written into *file*.
``group_by`` attribute:
- Determine how memory allocations are grouped: see
- ``Snapshot.top_by`` for the available values.
+ Determine how memory allocations are grouped: see ``Snapshot.top_by``
+ for the available values.
-``user_data_callback`` attribute:
- Optional callback collecting user data (callable, default:
- ``None``). See ``Snapshot.create()``.
-
-
-Filter class
-------------
+Filter
+------
``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class:
- Filter to select which memory allocations are traced. Filters can be
- used to reduce the memory usage of the ``tracemalloc`` module, which
- can be read using ``get_tracemalloc_size()``.
-
-``match_trace(trace)`` method:
-
- Return ``True`` if the ``Trace`` instance must be kept according to
- the filter, ``False`` otherwise.
+ Filter to select which memory allocations are traced. Filters can be used to
+ reduce the memory usage of the ``tracemalloc`` module, which can be read
+ using the ``get_tracemalloc_memory`` function.
``match(filename: str, lineno: int)`` method:
- Return ``True`` if the filename and line number must be kept
- according to the filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the filename and line number,
+ ``False`` otherwise.
``match_filename(filename: str)`` method:
- Return ``True`` if the filename must be kept according to the
- filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the filename, ``False`` otherwise.
``match_lineno(lineno: int)`` method:
- Return ``True`` if the line number must be kept according to the
- filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the line number, ``False``
+ otherwise.
+
+``match_traceback(traceback)`` method:
+
+ Return ``True`` if the filter matchs the *traceback*, ``False``
+ otherwise.
+
+ *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples.
``include`` attribute:
- If *include* is ``True``, only trace memory blocks allocated in a
- file with a name matching filename ``pattern`` at line number
- ``lineno``. If *include* is ``False``, ignore memory blocks
- allocated in a file with a name matching filename :attr`pattern` at
- line number ``lineno``.
+ If *include* is ``True``, only trace memory blocks allocated in a file
+ with a name matching filename ``pattern`` at line number
+ ``lineno``.
+
+ If *include* is ``False``, ignore memory blocks allocated in a file with
+ a name matching filename :attr`pattern` at line number ``lineno``.
+
+``lineno`` attribute:
+
+ Line number (``int``). If is is ``None`` or less than ``1``, it matches
+ any line number.
``pattern`` attribute:
- The filename *pattern* can contain one or many ``*`` joker
- characters which match any substring, including an empty string. The
- ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On
- Windows, the comparison is case insensitive and the alternative
- separator ``/`` is replaced with the standard separator ``\``.
-
-``lineno`` attribute:
-
- Line number (``int``). If is is ``None`` or lesser than ``1``, it
- matches any line number.
+ The filename *pattern* can contain one or many ``*`` joker characters
+ which match any substring, including an empty string. The ``.pyc`` and
+ ``.pyo`` file extensions are replaced with ``.py``. On Windows, the
+ comparison is case insensitive and the alternative separator ``/`` is
+ replaced with the standard separator ``\``.
``traceback`` attribute:
- If *traceback* is ``True``, all frames of the ``traceback``
- attribute of ``Trace`` instances are checked. If *traceback* is
- ``False``, only the most recent frame is checked.
+ If *traceback* is ``True``, all frames of the traceback are checked. If
+ *traceback* is ``False``, only the most recent frame is checked.
- This attribute only has an effect on the ``match_trace()`` method
- and only if the traceback limit is greater than ``1``. See the
- ``get_traceback_limit()`` function.
+ This attribute is ignored if the traceback limit is less than ``2``.
+ See the ``get_traceback_limit`` function.
-Frame class
------------
+GroupedStats
+------------
-``Frame`` class:
+``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class:
- Trace of a Python frame, used by ``Trace.traceback`` attribute.
+ Top of allocated memory blocks grouped by *group_by* as a dictionary.
-``filename`` attribute:
-
- Python filename, ``None`` if unknown.
-
-``lineno`` attribute:
-
- Python line number, ``None`` if unknown.
-
-
-GroupedStats class
-------------------
-
-``GroupedStats(stats: dict, group_by: str, cumulative=False, timestamp=None, process_memory=None, tracemalloc_size=None)`` class:
-
- Top of allocated memory blocks grouped by on *group_by* as a
- dictionary.
-
- The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance.
+ The ``Snapshot.top_by`` method creates a ``GroupedStats`` instance.
``compare_to(old_stats: GroupedStats=None)`` method:
- Compare to an older ``GroupedStats`` instance. Return a
- ``StatsDiff`` instance.
+ Compare to an older ``GroupedStats`` instance.
+ Return a ``StatsDiff`` instance.
+
+ The ``StatsDiff.differences`` list is not sorted: call
+ the ``StatsDiff.sort`` method to sort the list.
+
+ ``None`` values are replaced with an empty string for filenames or zero
+ for line numbers, because ``str`` and ``int`` cannot be
+ compared to ``None``.
``cumulative`` attribute:
- If ``True``, cumulate size and count of memory blocks of all frames
- of ``Trace``, not only the most recent frame.
+ If ``True``, cumulate size and count of memory blocks of all frames of
+ the traceback of a trace, not only the most recent frame.
+
+``metrics`` attribute:
+
+ Dictionary storing metrics read when the snapshot was created:
+ ``{name (str): metric}`` where *metric* type is ``Metric``.
``group_by`` attribute:
- Determine how memory allocations were grouped. The type of ``stats``
- keys depends on *group_by*:
-
- ===================== ======================== ==============
- group_by description key type
- ===================== ======================== ==============
- ``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
- ``'address'`` memory block address ``int``
- ===================== ======================== ==============
-
- See the *group_by* parameter of the ``Snapshot.top_by()`` method.
+ Determine how memory allocations were grouped: see
+ ``Snapshot.top_by`` for the available values.
``stats`` attribute:
- Dictionary ``{key: stats}`` where the *key* type depends on the
- ``group_by`` attribute and *stats* type is ``TraceStats``.
+ Dictionary ``{key: stats}`` where the *key* type depends on the
+ ``group_by`` attribute and *stats* is a ``(size: int, count: int)``
+ tuple.
-``process_memory`` attribute:
-
- Result of the ``get_process_memory()`` function, can be ``None``.
+ See the ``Snapshot.top_by`` method.
``timestamp`` attribute:
- Creation date and time of the snapshot, ``datetime.datetime``
- instance.
+ Creation date and time of the snapshot, ``datetime.datetime``
+ instance.
-``tracemalloc_size`` attribute:
- The memory usage in bytes of the ``tracemalloc`` module, result of
- the ``get_tracemalloc_size()`` function.
+Metric
+------
+``Metric(name: str, value: int, format: str)`` class:
-Snapshot class
---------------
+ Value of a metric when a snapshot is created.
-``Snapshot`` class:
+``name`` attribute:
- Snapshot of memory blocks allocated by Python.
+ Name of the metric.
- Use ``TakeSnapshot`` to take regulary snapshots.
+``value`` attribute:
+
+ Value of the metric.
+
+``format`` attribute:
+
+ Format of the metric:
+
+ * ``int``: a number
+ * ``percent``: percentage (1.0 means 100%)
+ * ``size``: a size in bytes
+
+
+Snapshot
+--------
+
+``Snapshot(timestamp: datetime.datetime, pid: int, traces: dict=None, stats: dict=None, metrics: dict=None)`` class:
+
+ Snapshot of traces and statistics on memory blocks allocated by Python.
+
+ Use ``TakeSnapshotTask`` to take regulary snapshots.
+
+``add_gc_metrics()`` method:
+
+ Add a metric on garbage collector:
+
+ * ``gc.objects``: total number of Python objects
+
+ See the ``gc`` module.
+
+
+``add_metric(name: str, value: int, format: str)`` method:
+
+ Helper to add a ``Metric`` instance to ``Snapshot.metrics``.
+ Return the newly created ``Metric`` instance.
+
+ Raise an exception if the name is already present in
+ ``Snapshot.metrics``.
+
+
+``add_process_memory_metrics()`` method:
+
+ Add metrics on the process memory:
+
+ * ``process_memory.rss``: Resident Set Size
+ * ``process_memory.vms``: Virtual Memory Size
+
+ These metrics are only available if the ``get_process_memory``
+ function is available on the platform.
+
+
+``add_pymalloc_metrics()`` method:
+
+ Add metrics on the Python memory allocator (``pymalloc``):
+
+ * ``pymalloc.blocks``: number of allocated memory blocks
+ * ``pymalloc.size``: size of ``pymalloc`` arenas
+ * ``pymalloc.max_size``: maximum size of ``pymalloc`` arenas
+ * ``pymalloc.allocated``: number of allocated bytes
+ * ``pymalloc.free``: number of free bytes
+ * ``pymalloc.fragmentation``: fragmentation percentage of the arenas
+
+ These metrics are only available if Python is compiled in debug mode,
+ except ``pymalloc.blocks`` which is always available.
+
+
+``add_tracemalloc_metrics()`` method:
+
+ Add metrics on the ``tracemalloc`` module:
+
+ * ``tracemalloc.traced.size``: size of memory blocks traced by the
+ ``tracemalloc`` module
+ * ``tracemalloc.traced.max_size``: maximum size of memory blocks traced
+ by the ``tracemalloc`` module
+ * ``tracemalloc.traces``: number of traces of Python memory blocks
+ * ``tracemalloc.module.size``: total size of bytes allocated by the
+ ``tracemalloc`` module, including free bytes
+ * ``tracemalloc.module.free``: number of free bytes available for
+ the ``tracemalloc`` module
+ * ``tracemalloc.module.fragmentation``: percentage of fragmentation of
+ the memory allocated by the ``tracemalloc`` module
+ * ``tracemalloc.arena_size``: size of traced arenas
+
+ ``tracemalloc.traces`` metric is only present if the snapshot was created
+ with traces.
+
+
+``add_unicode_metrics()`` method:
+
+ Add metrics on the Unicode interned strings:
+
+ * ``unicode_interned.size``: size of the dictionary, excluding size
+ of strings
+ * ``unicode_interned.len``: length of the dictionary
+
``apply_filters(filters)`` method:
- Apply a list filters on the ``traces`` and ``stats`` dictionaries,
- *filters* is a list of ``Filter`` instances.
+ Apply filters on the ``traces`` and ``stats`` dictionaries,
+ *filters* is a list of ``Filter`` instances.
-``create(\*, with_traces=False, with_stats=True, user_data_callback=None)`` classmethod:
- Take a snapshot of traces and/or statistics of allocated memory
- blocks.
+``create(traces=False, metrics=True)`` classmethod:
- If *with_traces* is ``True``, ``get_traces()`` is called and its
- result is stored in the ``traces`` attribute. This attribute
- contains more information than ``stats`` and uses more memory and
- more disk space. If *with_traces* is ``False``, ``traces`` is set to
- ``None``.
+ Take a snapshot of traces and/or statistics of allocated memory blocks.
- If *with_stats* is ``True``, ``get_stats()`` is called and its
- result is stored in the ``Snapshot.stats`` attribute. If
- *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``.
+ If *traces* is ``True``, ``get_traces`` is called and its result
+ is stored in the ``Snapshot.traces`` attribute. This attribute
+ contains more information than ``Snapshot.stats`` and uses more
+ memory and more disk space. If *traces* is ``False``,
+ ``Snapshot.traces`` is set to ``None``.
- *with_traces* and *with_stats* cannot be ``False`` at the same time.
+ If *metrics* is ``True``, fill ``Snapshot.metrics`` with metrics
+ using the following methods:
- *user_data_callback* is an optional callable object. Its result
- should be serializable by the ``pickle`` module, or
- ``Snapshot.write()`` would fail. If *user_data_callback* is set, it
- is called and the result is stored in the ``Snapshot.user_data``
- attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``.
+ * ``add_gc_metrics``
+ * ``add_process_memory_metrics``
+ * ``add_pymalloc_metrics``
+ * ``add_tracemalloc_metrics``
+ * ``add_unicode_metrics``
- The ``tracemalloc`` module must be enabled to take a snapshot. See
- the ``enable()`` function.
+ If *metrics* is ``False``, ``Snapshot.metrics`` is set to an empty
+ dictionary.
-``load(filename)`` classmethod:
+ Tracebacks of traces are limited to ``traceback_limit`` frames. Call
+ ``set_traceback_limit`` before calling ``Snapshot.create`` to
+ store more frames.
- Load a snapshot from a file.
+ The ``tracemalloc`` module must be enabled to take a snapshot. See the
+ the ``enable`` function.
+
+``get_metric(name, default=None)`` method:
+
+ Get the value of the metric called *name*. Return *default* if the metric
+ does not exist.
+
+
+``load(filename, traces=True)`` classmethod:
+
+ Load a snapshot from a file.
+
+ If *traces* is ``False``, don't load traces.
+
``top_by(group_by: str, cumulative: bool=False)`` method:
- Compute top statistics grouped by *group_by* as a ``GroupedStats``
- instance:
+ Compute top statistics grouped by *group_by* as a ``GroupedStats``
+ instance:
- ===================== ======================== ==============
- group_by description key type
- ===================== ======================== ==============
- ``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
- ``'address'`` memory block address ``int``
- ===================== ======================== ==============
+ ===================== ======================== ==============
+ group_by description key type
+ ===================== ======================== ==============
+ ``'filename'`` filename ``str``
+ ``'line'`` filename and line number ``(str, int)``
+ ``'address'`` memory block address ``int``
+ ===================== ======================== ==============
- If *cumulative* is ``True``, cumulate size and count of memory
- blocks of all frames of each ``Trace`` instance, not only the most
- recent frame. The *cumulative* parameter is ignored if *group_by* is
- ``'address'`` or if the traceback limit is ``1``. See the
- ``traceback_limit`` attribute.
+ If *cumulative* is ``True``, cumulate size and count of memory blocks of
+ all frames of the traceback of a trace, not only the most recent frame.
+ The *cumulative* parameter is ignored if *group_by* is ``'address'`` or
+ if the traceback limit is less than ``2``.
+
``write(filename)`` method:
- Write the snapshot into a file.
+ Write the snapshot into a file.
+
+
+``metrics`` attribute:
+
+ Dictionary storing metrics read when the snapshot was created:
+ ``{name (str): metric}`` where *metric* type is ``Metric``.
``pid`` attribute:
- Identifier of the process which created the snapshot, result of
- ``os.getpid()``.
-
-``process_memory`` attribute:
-
- Memory usage of the current process, result of the
- ``get_process_memory()`` function. It can be ``None``.
+ Identifier of the process which created the snapshot, result of
+ ``os.getpid``.
``stats`` attribute:
- Statistics on traced Python memory, result of the ``get_stats()``
- function, if ``create()`` was called with *with_stats* equals to
- ``True``, ``None`` otherwise.
-
-``tracemalloc_size`` attribute:
-
- The memory usage in bytes of the ``tracemalloc`` module, result of
- the ``get_tracemalloc_size()`` function.
+ Statistics on traced Python memory, result of the ``get_stats``
+ function.
``traceback_limit`` attribute:
- The maximum number of frames stored in the ``traceback`` attribute
- of a ``Trace``, result of the ``get_traceback_limit()`` function.
+ Maximum number of frames stored in a trace of a memory block allocated by
+ Python.
``traces`` attribute:
- Traces of Python memory allocations, result of the ``get_traces()``
- function, if ``create()`` was called with *with_traces* equals to
- ``True``, ``None`` otherwise.
-
- The ``traceback`` attribute of each ``Trace`` instance is limited to
- ``traceback_limit`` frames.
+ Traces of Python memory allocations, result of the ``get_traces``
+ function, can be ``None``.
``timestamp`` attribute:
- Creation date and time of the snapshot, ``datetime.datetime``
- instance.
+ Creation date and time of the snapshot, ``datetime.datetime``
+ instance.
-``user_data`` attribute:
- Result of *user_data_callback* called in ``Snapshot.create()``
- (default: ``None``).
-
-
-StatsDiff class
----------------
+StatsDiff
+---------
``StatsDiff(differences, old_stats, new_stats)`` class:
- Differences between two ``GroupedStats`` instances. By default, the
- ``differences`` list is unsorted: call ``sort()`` to sort it.
+ Differences between two ``GroupedStats`` instances.
- The ``GroupedStats.compare_to()`` method creates a ``StatsDiff``
- instance.
+ The ``GroupedStats.compare_to`` method creates a ``StatsDiff``
+ instance.
``sort()`` method:
- Sort the ``differences`` list from the biggest allocation to the
- smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and
- then by *key*.
+ Sort the ``differences`` list from the biggest difference to the
+ smallest difference. Sort by ``abs(size_diff)``, *size*,
+ ``abs(count_diff)``, *count* and then by *key*.
``differences`` attribute:
- Differences between ``old_stats`` and ``new_stats`` as a list of
- ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*,
- *size*, *count_diff* and *count* are ``int``. The key type depends
- on the ``group_by`` attribute of ``new_stats``:
-
- ===================== ======================== ==============
- group_by description key type
- ===================== ======================== ==============
- ``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
- ``'address'`` memory block address ``int``
- ===================== ======================== ==============
-
- See the ``group_by`` attribute of the ``GroupedStats`` class.
+ Differences between ``old_stats`` and ``new_stats`` as a list of
+ ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*,
+ *size*, *count_diff* and *count* are ``int``. The key type depends on the
+ ``GroupedStats.group_by`` attribute of ``new_stats``: see the
+ ``Snapshot.top_by`` method.
``old_stats`` attribute:
- Old ``GroupedStats`` instance, can be ``None``.
+ Old ``GroupedStats`` instance, can be ``None``.
``new_stats`` attribute:
- New ``GroupedStats`` instance.
+ New ``GroupedStats`` instance.
-Trace class
------------
+Task
+----
-``Trace`` class:
+``Task(func, \*args, \*\*kw)`` class:
- Debug information of a memory block allocated by Python.
+ Task calling ``func(*args, **kw)``. When scheduled, the task is called when
+ the traced memory is increased or decreased by more than *threshold* bytes,
+ or after *delay* seconds.
-``size`` attribute:
+``call()`` method:
- Size in bytes of the memory block.
+ Call ``func(*args, **kw)`` and return the result.
-``traceback`` attribute:
- Traceback where the memory block was allocated as a list of
- ``Frame`` instances, most recent first.
+``cancel()`` method:
- The list can be empty or incomplete if the ``tracemalloc`` module
- was unable to retrieve the full traceback.
+ Cancel the task.
- The traceback is limited to ``get_traceback_limit()`` frames. Use
- ``set_traceback_limit()`` to store more frames.
+ Do nothing if the task is not scheduled.
-TraceStats class
+``get_delay()`` method:
+
+ Get the delay in seconds. If the delay is ``None``, the timer is
+ disabled.
+
+
+``get_memory_threshold()`` method:
+
+ Get the threshold of the traced memory. When scheduled, the task is
+ called when the traced memory is increased or decreased by more than
+ *threshold* bytes. The memory threshold is disabled if *threshold* is
+ ``None``.
+
+ See also the ``set_memory_threshold`` method and the
+ ``get_traced_memory`` function.
+
+
+``schedule(repeat: int=None)`` method:
+
+ Schedule the task *repeat* times. If *repeat* is ``None``, the task is
+ rescheduled after each call until it is cancelled.
+
+ If the method is called twice, the task is rescheduled with the new
+ *repeat* parameter.
+
+ The task must have a memory threshold or a delay: see ``set_delay``
+ and ``set_memory_threshold`` methods. The ``tracemalloc`` must be
+ enabled to schedule a task: see the ``enable`` function.
+
+ The task is cancelled if the ``call`` method raises an exception.
+ The task can be cancelled using the ``cancel`` method or the
+ ``cancel_tasks`` function.
+
+
+``set_delay(seconds: int)`` method:
+
+ Set the delay in seconds before the task will be called. Set the delay to
+ ``None`` to disable the timer.
+
+ The timer is based on the Python memory allocator, it is not real time.
+ The task is called after at least *delay* seconds, it is not called
+ exactly after *delay* seconds if no Python memory allocation occurred.
+ The timer has a resolution of 1 second.
+
+ The task is rescheduled if it was scheduled.
+
+
+``set_memory_threshold(size: int)`` method:
+
+ Set the threshold of the traced memory. When scheduled, the task is
+ called when the traced memory is increased or decreased by more than
+ *threshold* bytes. Set the threshold to ``None`` to disable it.
+
+ The task is rescheduled if it was scheduled.
+
+ See also the ``get_memory_threshold`` method and the
+ ``get_traced_memory`` function.
+
+
+``func`` attribute:
+
+ Function, callable object.
+
+``func_args`` attribute:
+
+ Function arguments, ``tuple``.
+
+``func_kwargs`` attribute:
+
+ Function keyword arguments, ``dict``. It can be ``None``.
+
+
+TakeSnapshotTask
----------------
-``TraceStats`` class:
+``TakeSnapshotTask(filename_template: str="tracemalloc-$counter.pickle", traces: bool=False, metrics: bool=True, callback: callable=None)`` class:
- Statistics on Python memory allocations.
+ Task taking snapshots of Python memory allocations and writing them into
+ files.
-``size`` attribute:
+ ``TakeSnapshotTask`` is based on the ``Task`` class and so inherit
+ all attributes and methods, especially:
- Total size in bytes of allocated memory blocks.
+ * ``Task.cancel``
+ * ``Task.schedule``
+ * ``Task.set_delay``
+ * ``Task.set_memory_threshold``
-``count`` attribute:
+``take_snapshot()`` method:
- Number of allocated memory blocks.
+ Take a snapshot and write it into a file.
+ Return ``(snapshot, filename)`` where *snapshot* is a ``Snapshot``
+ instance and filename type is ``str``.
+
+``callback`` attribute:
+
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the newly
+ created snapshot instance. Use the ``Snapshot.add_metric`` method to
+ add new metric.
+
+``filename_template`` attribute:
+
+ Template to create a filename. The template supports the following
+ variables:
+
+ * ``$pid``: identifier of the current process
+ * ``$timestamp``: current date and time
+ * ``$counter``: counter starting at 1 and incremented at each snapshot,
+ formatted as 4 decimal digits
+
+ The default template is ``tracemalloc-$counter.pickle``.
+
+``metrics`` attribute:
+
+ Parameter passed to the ``Snapshot.create`` function.
+
+``traces`` attribute:
+
+ Parameter passed to the ``Snapshot.create`` function.
Links
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list