[Python-Dev] Re: test_bsddb blocks testing popitem - reason

[Alex Martelli]

On Monday 27 October 2003 10:40 am, Gregory P. Smith wrote:
   ...
> The big difference i see between 2.3cvs and 2.4cvs that could "explain"
> it is that Lib/bsddb/__init__.py has been updated to use a private
> (in memory, single process only) DBEnv with locking and thread support
> enabled.  That explains why db->del() would be doing locking.  But not
> why it would deadlock.

*AH*!  I wasn't looking in the right place, silly me.  Good job!!!  Yes,
now that you've pointed it out, the change from 2.3's
    d = db.DB()
to 2.4's
    e = _openDBEnv()
    d = db.DB(e)
must be the culprit.

I still don't quite see how the lock ends up being "held", but, don't mind
me -- the intricacy of mixins and wrappings and generators and delegations
in those modules is making my head spin anyway, so it's definitely not
surprising that I can't quite see what's going on.

> How do python dictionaries deal with modifications to the dictionary
> intermixed with iteration?

In general, Python doesn't deal well with modifications to any
iterable in the course of a loop using an iterator on that iterable.

The one kind of "modification during the loop" that does work is:

for k in somedict:
    somedict[k] = ...whatever...

i.e. one can change the values corresponding to keys, but not
change the set of keys in any way -- any changes to the set of
keys can cause unending loops or other such misbehavior (not
deadlocks nor crashes, though...).

However, on a real Python dict,
    k, v = thedict.iteritems().next()
doesn't constitute "a loop" -- the iterator object returned by
the iteritems call is dropped since there are no outstanding
references to it right after this statement.  So, following up
with
    del thedict[k]
is quite all right -- the dictionary isn't being "looped on" at
that time.

Given that in bsddb's case that iteritems() first [and only]
next() boils down to a self.first() which in turn does a 
self.dbc.first() I _still_ don't see exactly what's holding the
lock.  But the simplest fix would appear to be in __delitem__,
i.e., if we have a cursor we should delete through it:

    def __delitem__(self, key):
        self._checkOpen()
        if self.dbc is not None:
            self.dbc.set(key)
            self.dbc.delete()
        else:
            del self.db[key]

...but this doesn't in fact remove the deadlock on the
unit-test for popitem, which just confirms I don't really
grasp what's going on, yet!-)


Alex