"RuntimeError: dictionary changed ... & Ruby

[Alex Martelli]

robert <no-spam at no-spam-no-spam.com> wrote:
   ...
> What? When I add/del an item to a dict or list, this is not an atomic
> thread-safe operation?

Exactly: there is no such guarantee in the Python language.

> E.g.:
> One thread does things like  d['x']='y'
> Another thread reads d['z'] or sets  d['z']='w' or dels something.
> 
> If those operations are not atomic, then you'd have to use locks all the
> time to not get RuntimeErrors and worse !?

If you want to be writing correct Python, yes. A preferred approach is
to simply avoid sharing objects among threads, except for objects
designed to be thread-safe (chiefly Queue.Queue).

> Infact I rely on that all the time and standard python modules also do
> so AFAIK

You're relying on an accident of a specific, particular implementation;
if any Python-coded standard library module does likewise, and I'm not
aware of any, that's somewhat different (since that module is PART of
the implementation, it may rely on all kinds of implementation details,
correctly if maybe not wisely). The situation is quite different for
C-coded modules in the CPython implementation, Java-coded ones in the
Jython one, C#-coded one in the IronPython one; each of these is subject
to specific constraints that it's perfectly wise to rely on (since each
implementation, as the language specification fully allows it to do,
adopts a different locking strategy at these low levels).

> I think cPickle has not necessarily to iterate free over native dicts.

It's not forced to by language specification, but neither is it
forbidden. It would be an absurd implementation strategy to waste time
and space to extract a dict's keys() first, as it would NOT buy
"atomicity" anyway -- what if some other thread deletes keys while
you're looping, or calls any nonatomic method on the very value you're
in the process of serializing?!

In some Python implementations, a C-coded module may count on some
atomicity as long as it doesn't explicitly allow other threads nor ever
call back into ANY python-coded part, but obviously cpickle cannot avoid
doing that, so even in those implementations it will never be atomic.

> Whats does copy/deepcopy/[:] ?

Roughly the same situation.


If as you indicate you want to stick with a Python-like language but do
not want to change your style to make it correct Python, you could
perhaps fork the implementation into an "AtomicPython" in which you
somehow fix all nonatomicities (not sure how that would even be possible
if pickling, deepcopying or anything else ever needs to fork into Python
coded parts, but perhaps you might make the GIL into a reentrant lock
and somehow hack it to work, with some constraints). Or perhaps you
might be able to write an extension containing atomicset, atomicget,
atomicpickle, and other operations you feel you need to be atomic (again
barring the difficulties due to possible callbacks into Python) and use
those instead of bare Python primitives.


Alex