CPython 2.7: Weakset data changing size during internal iteration

[Temia Eszteri]

On Fri, 01 Jun 2012 18:42:22 -0400, Terry Reedy <tjreedy at udel.edu>
wrote:

>I gather that the .references attribute is sometimes/always a weakset. 
>To determine its boolean value, it computes its length. For regular 
>sets, this is sensible as .__len__() returns a pre-computed value.

Indeed. Back when I was using 2.6 to develop, it was simply an integer
counter, but that led to some difficulties in maintaining it in case
some sprite objects hadn't been explicitly killed.

>Given that len(weakset) is defined (sensibly) as the number of currently 
>active members, it must count. weakset should really have .__bool__ 
>method that uses any() instead of sum(). That might reduce, but not 
>necessarily eliminate your problem.

Think it might be worth looking into submitting a patch for the next
minor releases for Python if it turns out to solve the problem?
Failing that, I might just have to check the truth value of the data
attribute inside the weak set manually...

>I can think of two reasons:
>
>1. You are using multiple threads and another thread does something to 
>change the size of the set during the iteration. Solution? put a lock 
>around the if-statement so no other thread can change self.data during 
>the iteration.
>
>2. Weakset members remove themselves from the set before returning None. 
>(Just a thought, in case you are not using threads).

It's a multithreaded program to a small extent - I offload I/O
operations, music handling, and a basic, optional debugger console
(which I really wish I could set up to use the real interactive
interpreter instead of the shoddy setup I've got now) to seperate
threads, while the main logic operates in one thread due to OpenGL's
issues with multiple Python threads.

Since the sprite object calls to reference a texture in __init__(),
that means no other thread could even safely reference the texture due
to the potential of making OpenGL calls without the relevant context
kept by the main thread (this has made the loading thread kind of
useless, but the texture strings themselves can still be loaded into
temporary memory, and other data like music still works).

If the weak references removing themselves is the case, it seems like
a kind of silly problem - one would imagine they'd wrap the data check
in _IterationGuard in the _weakrefset.py file like they do for calls
to __iter__(). Very strange.

Anyway, I truly appreciate your input and suggestions. I'll see if
they have any results, and if so, we can work out submitting a patch.
If not, at least reading through this gave me the idea to just call
the data set inside it, so I can use it as an imperfect but functional
solution within the scope of my project.

~Temia
--
When on earth, do as the earthlings do.