Access violation in Python garbage collector (visit_decref) - how to debug?

[Geoff Bache]

Hi all,

We are running Python embedded in our C++ product and are now experiencing
crashes (access violation reading 0xffffffffff on Windows) in the Python
garbage collector.

We got this on Python 3.6.4 originally, but I can reproduce it with both
Python 3.6.8 and Python 3.7.4.

The chances of producing a minimal example that reproduces it reliably are
currently small I would say. All attempts to simplify the set up seem to
cause the problem to go away.
Indeed I can only reproduce it by sending a fairly large amount of data 2
or 3 times to our server - sending either half of the data does not
reproduce it.

I note that there have been a lot of crashes in this method reported over
the years, and that efforts are underway in Python 3.8 to make it easier to
find issues like this, i.e. https://bugs.python.org/issue36389.
I won't be able to use Python 3.8 yet but I could potentially use it as a
debugging tool I guess, if there was documentation available how to use
these new features.

I tried setting PYTHONMALLOC=debug but didn't get any more information that
way.

The python code is calling "json.loads" on a fairly large json unicode
string with plenty of Swedish characters in it. It works at least once, and
then crashes on the second or third send of the same data.

I paste the stacktrace from Python 3.7.4 below. Please let me know how I
can debug this further. I'm using Visual Studio 2017 on Windows 10 if that
helps.

Regards,
Geoff Bache

> [Inline Frame] python37.dll!visit_decref(_object *) Line 271 C
  python37.dll!dict_traverse(_object * op, int(*)(_object *, void *) visit,
void * arg) Line 2982 C
  python37.dll!subtract_refs(_gc_head * containers) Line 296 C
  python37.dll!collect(int generation, __int64 * n_collected, __int64 *
n_uncollectable, int nofail) Line 861 C
  python37.dll!collect_with_callback(int generation) Line 1029 C
  python37.dll!collect_generations() Line 1051 C
  [Inline Frame] python37.dll!_PyObject_GC_Alloc(int) Line 1708 C
  [Inline Frame] python37.dll!_PyObject_GC_Malloc(unsigned __int64
basicsize) Line 1718 C
  python37.dll!_PyObject_GC_New(_typeobject * tp) Line 1730 C
  python37.dll!new_dict(_dictkeysobject * keys, _object * * values) Line
584 C
  python37.dll!PyDict_New() Line 678 C
  python37.dll!_parse_object_unicode(_PyScannerObject * s, _object * pystr,
__int64 idx, __int64 * next_idx_ptr) Line 730 C
  python37.dll!scan_once_unicode(_PyScannerObject * s, _object * pystr,
__int64 idx, __int64 * next_idx_ptr) Line 1162 C
  python37.dll!_parse_array_unicode(_PyScannerObject * s, _object * pystr,
__int64 idx, __int64 * next_idx_ptr) Line 869 C
  python37.dll!scan_once_unicode(_PyScannerObject * s, _object * pystr,
__int64 idx, __int64 * next_idx_ptr) Line 1101 C
  python37.dll!scanner_call(_object * self, _object * args, _object * kwds)
Line 1188 C
  python37.dll!_PyObject_FastCallKeywords(_object * callable, _object *
const * stack, __int64 nargs, _object * kwnames) Line 199 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4619 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3095 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals,
_object * locals, _object * const * args, __int64 argcount, _object * const
* kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object *
const * defs, __int64 defcount, _object * kwdefs, _object * closure,
_object * name, _object * qualname) Line 3930 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 433 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4621 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3140 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals,
_object * locals, _object * const * args, __int64 argcount, _object * const
* kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object *
const * defs, __int64 defcount, _object * kwdefs, _object * closure,
_object * name, _object * qualname) Line 3930 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 433 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4621 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3111 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals,
_object * locals, _object * const * args, __int64 argcount, _object * const
* kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object *
const * defs, __int64 defcount, _object * kwdefs, _object * closure,
_object * name, _object * qualname) Line 3930 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 433 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4621 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3095 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  [Inline Frame] python37.dll!function_code_fastcall(PyCodeObject * args,
_object * const *) Line 283 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 408 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4616 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3125 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  [Inline Frame] python37.dll!function_code_fastcall(PyCodeObject * args,
_object * const *) Line 283 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 408 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4616 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3125 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  [Inline Frame] python37.dll!function_code_fastcall(PyCodeObject * args,
_object * const *) Line 283 C
  [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack,
_object * const *) Line 408 C
  python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object
* kwnames) Line 4616 C
  python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line
3125 C
  [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C
  [Inline Frame] python37.dll!function_code_fastcall(PyCodeObject * args,
_object * const *) Line 283 C
  python37.dll!_PyFunction_FastCallDict(_object * func, _object * const *
args, __int64 nargs, _object * kwargs) Line 322 C
  python37.dll!PyObject_Call(_object * callable, _object * args, _object *
kwargs) Line 226 C
  estx.exe!PyObject_CallObject(_object * obj, _object * args) Line 191 C++
.....our server's C++ code in stack frames above this