[issue28199] Compact dict resizing is doing too much work

[Xiang Zhang]

Xiang Zhang added the comment:

I use gdb to run setuptools test suite and find the assumption, split tables are always dense is broken for both dictresize3 and dictresize4.

#0  0x00007ffff71171c7 in __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff7118e2a in __GI_abort () at abort.c:89
#2  0x00007ffff71100bd in __assert_fail_base (fmt=0x7ffff7271f78 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion at entry=0x5e4b90 "oldvalues[i] != ((void *)0)", file=file at entry=0x5e4aa0 "Objects/dictobject.c", line=line at entry=1270, 
    function=function at entry=0x5e59f0 <__PRETTY_FUNCTION__.12083> "dictresize") at assert.c:92
#3  0x00007ffff7110172 in __GI___assert_fail (assertion=assertion at entry=0x5e4b90 "oldvalues[i] != ((void *)0)", 
    file=file at entry=0x5e4aa0 "Objects/dictobject.c", line=line at entry=1270, function=function at entry=0x5e59f0 <__PRETTY_FUNCTION__.12083> "dictresize")
    at assert.c:101
#4  0x000000000048bddc in dictresize (mp=mp at entry=0x7ffff219d2b0, minused=<optimized out>) at Objects/dictobject.c:1270
#5  0x000000000048bf93 in insertion_resize (mp=mp at entry=0x7ffff219d2b0) at Objects/dictobject.c:1100
#6  0x000000000048c5fd in insertdict (mp=mp at entry=0x7ffff219d2b0, key=key at entry=0x7ffff579c3c0, hash=-3681610201421769281, 
    value=value at entry=0x7ffff07f56e8) at Objects/dictobject.c:1136
#7  0x000000000048fdfd in PyDict_SetItem (op=op at entry=0x7ffff219d2b0, key=key at entry=0x7ffff579c3c0, value=value at entry=0x7ffff07f56e8)
    at Objects/dictobject.c:1572
#8  0x0000000000492cb5 in _PyObjectDict_SetItem (tp=tp at entry=0xd52548, dictptr=0x7ffff080cbd8, key=key at entry=0x7ffff579c3c0, 
    value=value at entry=0x7ffff07f56e8) at Objects/dictobject.c:4274
#9  0x000000000049df8a in _PyObject_GenericSetAttrWithDict (obj=0x7ffff080cbb8, name=0x7ffff579c3c0, value=0x7ffff07f56e8, dict=dict at entry=0x0)
    at Objects/object.c:1172
#10 0x000000000049e0cf in PyObject_GenericSetAttr (obj=<optimized out>, name=<optimized out>, value=<optimized out>) at Objects/object.c:1194
#11 0x000000000049d80e in PyObject_SetAttr (v=v at entry=0x7ffff080cbb8, name=name at entry=0x7ffff579c3c0, value=value at entry=0x7ffff07f56e8)
    at Objects/object.c:932

Thanks to Victor's _PyDict_CheckConsistency, it's easy then to find even without dictresize3 and dictresize4 (the current version), the test suite still fails (#define DEBUG_PYDICT).

#0  0x00007ffff71171c7 in __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff7118e2a in __GI_abort () at abort.c:89
#2  0x00007ffff71100bd in __assert_fail_base (fmt=0x7ffff7271f78 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion at entry=0x5e53a0 "mp->ma_values[i] != ((void *)0)", file=file at entry=0x5e4d00 "Objects/dictobject.c", line=line at entry=498, 
    function=function at entry=0x5e5dd0 <__PRETTY_FUNCTION__.11869> "_PyDict_CheckConsistency") at assert.c:92
#3  0x00007ffff7110172 in __GI___assert_fail (assertion=assertion at entry=0x5e53a0 "mp->ma_values[i] != ((void *)0)", 
    file=file at entry=0x5e4d00 "Objects/dictobject.c", line=line at entry=498, 
    function=function at entry=0x5e5dd0 <__PRETTY_FUNCTION__.11869> "_PyDict_CheckConsistency") at assert.c:101
#4  0x000000000048ba17 in _PyDict_CheckConsistency (mp=mp at entry=0x7ffff0806e68) at Objects/dictobject.c:498
#5  0x00000000004927a3 in PyDict_SetDefault (d=d at entry=0x7ffff0806e68, key=0x7ffff2ffcdd8, defaultobj=0x8abf20 <_Py_NoneStruct>)
    at Objects/dictobject.c:2807
#6  0x0000000000492854 in dict_setdefault (mp=0x7ffff0806e68, args=<optimized out>) at Objects/dictobject.c:2824
#7  0x0000000000499469 in _PyCFunction_FastCallDict (func_obj=func_obj at entry=0x7ffff0f2f8c8, args=args at entry=0x105afe8, nargs=nargs at entry=2, 
    kwargs=kwargs at entry=0x0) at Objects/methodobject.c:234
#8  0x0000000000499815 in _PyCFunction_FastCallKeywords (func=func at entry=0x7ffff0f2f8c8, stack=stack at entry=0x105afe8, nargs=nargs at entry=2, 
    kwnames=kwnames at entry=0x0) at Objects/methodobject.c:295
#9  0x0000000000537b6f in call_function (pp_stack=pp_stack at entry=0x7fffffff5cd0, oparg=oparg at entry=2, kwnames=kwnames at entry=0x0)
    at Python/ceval.c:4793

>From the backtrace we can see PyDict_SetDefault breaks the invariant. And reading the code, yes, it doesn't handle split table separately.

I simply replace the logic in PyDict_SetDefault with insertdict to make a test. It doesn't fail, even with dictresize4.

An easy example to reproduce:

>>> class C:
...     pass
... 
>>> c1, c2 = C(), C()
>>> c1.a, c1.b = 1, 2
>>> c2.__dict__.setdefault('b', None)
python: Objects/dictobject.c:498: _PyDict_CheckConsistency: Assertion `mp->ma_values[i] != ((void *)0)' failed.
Aborted (core dumped)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28199>
_______________________________________