"RuntimeError: dictionary changed size during iteration" ; Good atomic copy operations?

[robert]

EleSSaR^ wrote:
> robert si è profuso/a a scrivere su comp.lang.python tutte queste
> elucubrazioni: 
> 
> 
>>own deepcopy: thus, do you already know if the existing deepcopy has the 
>>same problem as cPickle.dump ?    (as the problem araises rarely, it is 
>>difficult for me to test it out)
> 
> I don't know the exact specs of your object, and I don't know what
> operations are you performing on that object, nor the way they're atomic.

There is not much to know. Python object trees consist only of dicts and 
lists as far as variable non-atomic datastructures are concerned. 
(unless you use advanced extension libs like NumPy)

Thus the RuntimeError problem is only about modified dicts/lists during 
Iteration in pickly/copy.


> It seems like you're trying to save periodically the state of such object
> while it is being modified (a sort of backup?), and Python complains about
> that. A self-implemented deepcopy might raise anomalies (i.e. your dumped
> object may be partly a 'before' object and partly an 'after' object ) as
> well. 

Yes, a "backup" / autosave while all threads are running. It doesn't 
matter if 'before' of 'after' another item has been added/deleted 
atomically.


> By the way, you could try employing locks from other threads to dump the
> object as well... this would prevent additional locking.

Don't understand.
The threads work all simulatniously on the object tree, add and detach 
atomically only valid sub-trees.

Regarding what AM said, I would have to lock _each_ dict/list operation 
on the tree, thus almost each change, because even a single attribute 
change "subobj.x='y'" is a dictionary operation. That would make 
threaded programming very arduous.

AFAIK about the current Python implementation: This RuntimeError is only 
thrown "during true Iteration over dict/list, when the number of items 
changes".  (and not when e.g. a single item is changed). Thus a

def rt_save_dict_copy()
   tod={}
   for k in fromd.keys():
     try: tod[k]=fromd[k]
     except: pass
   return tod

without true iteration over the original dict whould copy without 
RuntimeError.

(or maybe equivalent:  "dict(fromd.items())" ? )

I don't know if dict.copy() works so - but I think so, as dict.keys() 
and dict.items() have the same footprint.

The algorithm in cPickle.dump does not work so. Guess it does something 
like "for k in fromd: ..."(!) internally. This might be a "90%-bug"?

Will have to see what copy/deepcopy does ...

Robert