Global Threading Lock 2 - Re: "RuntimeError: dictionary changed size during iteration"..

[Raymond Hettinger]

[robert]
> That queue/passing-through-only-an-extra-global-var communication is
> acceptable for thin thread interaction.
> ( hope this extra global var is thread-safe in future Python's :-) )
>
> But "real" thread-programming should also be possible in Python - and it
> is with the usual discipline in thread programming.

LOL, I'm glad you put "real" in quotes, and I'm glad that you recognize
that apps with intrinsic race conditions are not following "the usual
discipline in thread programming."

Embedded in this discussion is a plausable suggestion for Py2.5 to
offer a way for a thread to temporarily block thread switches while it
does something that needs to be atomic; however, the given use case is
on thin ice as a motivating example (because of the intrinsic race
condition, avoidance of locks, and avoidance of queues).


>This RuntimeError in
> iterations is the (necessary) only compromise, I know of. (Maybe this
> RuntimeError must not even be thrown from Python, when walking through
> variable sequences is done smartly - but smart practice may cost speed,
> so a compromise.)
>
> It can be handled commonly by keys() and some error catching. key
> functions like deepcopy and dump (which cannot easily be subclassed)
> should fit into that "highest common factor" and not "judge" themselves
> about _how_ thread programming has to be done.
 . . .
> In addition Python should define its time atoms, and thus also the
> definite sources of this (unavoidable?) RuntimeError - as explained in
> the later link.

Since others have responded directly to these thoughts, I'll aim at the
bigger picture and make an observation on Python sociology.  Most users
are aware that Python is not cast in stone and that good ideas are
always welcome.  That is usually a good thing, but it sometimes creates
a pitfall.  When someone programs themselves into a corner, they
usually get a cue that something is wrong with their design concept or
that they are not working harmoniously with the language; however, in
the Python world, it is tempting avoid questioning one's own design and
instead start to assume that the language itself is misconcieved.

A good language suggestion should be general purpose,
easy-to-understand, universal across implementations, and solve more
than one use case.  It is bad sign if you have to propose multiple,
simultaneous language changes just to get your script to work.
Likewise, it is a bad sign if the use case is somewhat unusual (i.e.
supporting an app with an intrinsic race-condition).  Also, it is a bad
sign if the proposal is over-specific and ad-hoc (i.e. imposing
memory-intensive requirements on random pieces of library code about
how the code is allowed to loop over dictionaries).

Those criteria for good language proprosals are not met by requests to
start making pieces of pure python code try to fake atomicity.
However, there is a some chance for Py2.5 to introduce a thread-switch
lock (as I proposed in my previous post).  If that is what you want,
feel free to log a SourceForge feature request (preferably with a
sensible use case and no troll phrases like "real" thread programming).


> I have 5 trials max as of now. The error was about once in 3 months in
> my case: that should solve the problem for the rest of the universe ...
> If not, there is another bug going on.

I sure hope your code isn't being used in mission critical apps like
air traffic control :=0


Raymond