[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Victor Stinner]

Le jeu. 11 avr. 2019 à 17:33, Serhiy Storchaka <storchaka at gmail.com> a écrit :
> If reducing the Python memory footprint is an argument for disabling
> Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> the debug build.

The "serialno" field of debug memory allocators is documented as: "an
excellent way to set a breakpoint on the next run, to capture the
instant at which this block was passed out." I'm debugging crashes and
memory leaks in CPython for 10 years, and I simply never had to use
"serialno". I wrote https://bugs.python.org/issue36611 to remove the
serialno field of debug hooks on Python memory allocators: it reduces
the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running
test_asyncio).

Python is used on devices with low memory (ex: 256 MiB for the whole
system). Allowing developers to use a debug build on such devices seem
to be a legit rationale for such change. The debug build is very
useful to identify bugs in C extensions.


> On other hand, since using the debug allocator doesn't cause problems
> with compatibility, it may be possible to use similar technique for the
> objects double list. Although this is not easy because of objects placed
> at static memory.

I'm not sure of what you means by "objects placed at static memory":
the double linked list of all Python objects is created at runtime.
_ob_next and _ob_prev are initialized statically to NULL.

I would be interested if Py_TRACE_REFS could be reimplemented in a
more dynamic fashion. Even if it would still require a debug build, it
would be nice to be able to "opt-in" for this feature (have it
disabled by default, again, to reduce the overhead and reduce the
memory footprint), as tracemalloc which plugs itself into memory
allocators to attach traces to memory blocks.

Except Guido who wrote "I recall finding memory leaks using this.
(E.g. I remember a leak in Zope due to a cache that was never pruned.)
But presumably gc.get_objects() would have been sufficient. (IIRC it
didn't exist at the time.)", at this point, nobody said that they use
Py_TRACE_REFS. So I'm not sure that it's worth it to invest time on a
feature if nobody uses it?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.