[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

That's the follow-up of a thread that I started on python-dev in June 2018:

  [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
  https://mail.python.org/pipermail/python-dev/2018-June/153857.html

When an application crash during a garbage collection, we are usually clueless about the cause of the crash. The crash usually occur in visit_decref() on a corrupted Python object. Sadly, not only there are too many possible reasons which can explain why a Python object is corrupted, but the crash usually occur too late after the object is corrupted. Using a smaller GC threshold can help, but it's not enough.

It would help to be able to enable a builtin checker for corrupted objects. Something that we would triggered by the GC with a threshold specified by the user and that would have zero impact on performance when it's not used.

The implementation would be to iterate on objects and ensure that they are consistent.

Attached PR is an implementation of this idea. It uses new API that I wrote recently:

* _PyObject_ASSERT()
* _PyObject_IsFreed()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()
* _PyDict_CheckConsistency()

If an inconsistency is detected, _PyObject_ASSERT() will call _PyObject_Dump() to dump info about the object. This function can crash, but well, anything can crash on a memory corruption...

----------
components: Interpreter Core
messages: 338536
nosy: pablogsal, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Add gc.enable_object_debugger(): detect corrupted Python objects in the GC
type: enhancement
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36389>
_______________________________________