Strategies when hunting for dictionary corruption

[dieter]

Skip Montanaro <skip.montanaro at gmail.com> writes:

> I'm wrapping some C++ libraries using pybind11. On the pybind11 side of
> things, I've written a simple type converter between Python's datetime.date
> objects and our internal C++ date objects. The first thing the type
> converter needs to do is to insure that the datetime C API is initialized:
>
>     if (!PyDateTimeAPI) { PyDateTime_IMPORT; }
>
> This is failing because a dictionary internal to the import mechanism is
> being corrupted, the extensions dict in import.c. I highly doubt this is a
> Python issue. It's much more likely to be a bug in our C++ libraries or in
> pybind11.
>
> The problem is, dictionaries being rather dynamic objects (growing and
> sometimes shrinking), the actual data location which might be corrupted
> isn't fixed, but can move around. I've got a Python interpreter configured
> using --with-pydebug, it's compiled with -ggdb -O0, and I'm ready to watch
> some memory locations, but it's not clear what exactly I should watch or
> how/when to move my watchpoint(s) around. Should I not be thinking in terms
> of watchpoints? Is there a better way to approach this problem? (I also
> have valgrind at my disposal, but am not very skilled in its use.)

I would use a form of binary search approach -- unless there is
an obvious cause, such as GIL not hold.

For the "binary search approach", you would need a non destructive way
to detect the corruption during a "gdb" debugging session.
There is a set of "gdb" macros to manage (e.g. print out) Python
objects from the debugger. Based on them, you might develop similar
macros to access dicts and check for their validity.
Once you have a way to detect the corruption, you can step into
the "PyDateTime_IMPORT" and find out where the corruption happens.