[Python-Dev] Showstopper

[Tim Peters]

Guido & I talked out A Plan, and he's going to implement it while I take a
nap.  Outline:

1. It sucks that PyDict_Next() can resize a dict.  And while staring
   at all this in the debugger, it was plain appalling that the mere
   act of doing gc ran around turning empty dicts into non-empty
   ones because of it (not a bug, just irksome waste).

2. It sucks that PyDict_SetItem() can resize a dict even when
   the number of active slots doesn't change.

The plan for addressing those:

A. Rip out the PyDict_Next() resizing hack.

B. In PyDict_SetItem(), first look at the number of free slots,
   and resize the dict if it would be impossible to add a new
   active slot (I *suspect* this can be reduced to making a minimal
   dict when the incoming dict is empty).
   Remember the number of used slots.
   Do the insert.
   Look at the number of used slots now:  do "the usual" resize
   logic if and only the number of used slots changed across
   the insert call.

That much should suffice to stop Jeremy's old bugs, and the bug I bumped into
here.

It's not enough, though.  Allocating a tuple (or any other gc'ed type) can
still cause gc to run, then gc can delete __del__-free cycles, then deleting
those can cause objects with __del__ methods to become unreachable too, and
then any Python code whatsoever can run, incl. but not limited to code that
dicts, and also incl. allowing other threads to run.

So code inside dict methods can't assume much of anything across container
allocations, even after fixing all the bugs we've bumped into so far.  So at
least dict_items() and dict_popitem() remain unsafe after these changes,
although we don't have a concrete test case to prove that and it would be
mondo difficult to create one.  Nevertheless, Python users are effective
proof of the million monkeys hypothesis <wink>.  These remaining problems
require case-by-case analysis and rewriting.

could-be-the-biggest-one-line-fix-in-history-ly y'rs  - tim