[issue32209] Crash in set_traverse Within the Garbage Collector's collect_generations()

[Connor W Fitzgerald]

New submission from Connor W Fitzgerald <connorwadefitzgerald at gmail.com>:

This is my first time submitting a bug report, so forgive me if I miss on important information. I am writing code that makes heavy use of sets (8GB+ of them). It segfaults semi-randomly during the processing of them. I've used versions 3.5.3, 3.6.3, and 3.7.0rc2. I debugged the process and came across the error in object/setobject.c:742. I don't really know what information to provide here but this is the best I can do.

Error:
Unhandled exception thrown: read access violation.
entry->**key** was 0x8000.

Stack Trace:
python37.dll!set_traverse(PySetObject * so, int(*)(_object *, void *) visit, void * arg) Line 742
python37.dll!subtract_refs(_gc_head * containers) Line 295
python37.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 862
python37.dll!collect_with_callback(int generation) Line 1027
python37.dll!collect_generations() Line 1049
python37.dll!PyType_GenericAlloc(_typeobject * type, __int64 nitems) Line 969
[Inline Frame] python37.dll!make_new_set(_typeobject *) Line 1055
python37.dll!set_new(_typeobject * type, _object * args, _object * kwds) Line 1127
python37.dll!type_call(_typeobject * type, _object * args, _object * kwds) Line 928
python37.dll!_PyObject_FastCallKeywords(_object * callable, _object * * stack, __int64 nargs, _object * kwnames) Line 199
python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object * kwnames) Line 4705
python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 3182
[Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 549
python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * * args, __int64 argcount, _object * * kwnames, _object * * kwargs, __int64 kwcount, int kwstep, _object * * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4011
python37.dll!_PyFunction_FastCallDict(_object * func, _object * * args, __int64 nargs, _object * kwargs) Line 376
[Inline Frame] python37.dll!PyEval_CallObjectWithKeywords(_object *) Line 819
python37.dll!defdict_missing(defdictobject * dd, _object * key) Line 1993
python37.dll!_PyMethodDef_RawFastCallDict(PyMethodDef * method, _object * self, _object * * args, __int64 nargs, _object * kwargs) Line 496
[Inline Frame] python37.dll!_PyCFunction_FastCallDict(_object *) Line 580
python37.dll!_PyObject_FastCallDict(_object * callable, _object * * args, __int64 nargs, _object * kwargs) Line 101
[Inline Frame] python37.dll!object_vacall(_object * callable, char *) Line 1194
python37.dll!PyObject_CallFunctionObjArgs(_object * callable, ...) Line 1259
python37.dll!dict_subscript(PyDictObject * mp, _object * key) Line 1984
python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 1316
[Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 549
python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * * args, __int64 argcount, _object * * kwnames, _object * * kwargs, __int64 kwcount, int kwstep, _object * * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4011
[Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack, _object * *) Line 433
python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object * kwnames) Line 4703
python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 3182
[Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 549
python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * * args, __int64 argcount, _object * * kwnames, _object * * kwargs, __int64 kwcount, int kwstep, _object * * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4011
python37.dll!PyEval_EvalCodeEx(_object * _co, _object * globals, _object * locals, _object * * args, int argcount, _object * * kws, int kwcount, _object * * defs, int defcount, _object * kwdefs, _object * closure) Line 4045
python37.dll!PyEval_EvalCode(_object * co, _object * globals, _object * locals) Line 532
python37.dll!run_mod(_mod * mod, _object * filename, _object * globals, _object * locals, PyCompilerFlags * flags, _arena * arena) Line 987
python37.dll!PyRun_FileExFlags(_iobuf * fp, const char * filename_str, int start, _object * globals, _object * locals, int closeit, PyCompilerFlags * flags) Line 939
python37.dll!PyRun_SimpleFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 402
python37.dll!PyRun_AnyFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 84
python37.dll!run_file(_iobuf * fp, const wchar_t * filename, PyCompilerFlags * p_cf) Line 340
python37.dll!Py_Main(int argc, wchar_t * * argv) Line 894
[Inline Frame] python.exe!invoke_main() Line 79
python.exe!__scrt_common_main_seh() Line 253
kernel32.dll!0000000077a959cd()
ntdll.dll!0000000077bca561()

I'm not sure what I can do to reproduce this because I don't even know what line of code my program was running before it segfaulted. I'm using sqlite3, json, and pickle as my main extensions. Nothing outside of what is provided in the standard library is used. If you need any more information, I'll gladly provide it.

----------
components: Interpreter Core
messages: 307539
nosy: connorwfitzgerald
priority: normal
severity: normal
status: open
title: Crash in set_traverse Within the Garbage Collector's collect_generations()
type: crash
versions: Python 3.5, Python 3.6, Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32209>
_______________________________________