[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)

[Andrew Graham]

I confirm the same problem, except that it happens more often for me, about 50 to 60 times each time I run the test. 
It smells like some sort of “unsynchronised acess by different threads to an object problem ”. 

One (very) hacky way round it until the real cause is established is to modify __contains__() in _weakrefset.py line 68 to retry until it gets a proper result

    def __contains__(self, item):
        while 1:
            try:
                try:
                    wr = ref(item)
                except TypeError:
                    return False
                return wr in self.data
            except Exception:
                pass

Regards

Andy Graham

From: Andres Sommerhoff 
Sent: Monday, April 13, 2015 6:39 AM
To: ironpython-users at python.org 
Subject: [Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)

Dear IronPython gurus!

Hopping you can help me with a kind of ramdom bug (unexpected SystemError and ValueError exception) in _weakrefset.py (it would be great if you can replicate it by running my script below. Please let me know). 

The error is random, but I managed to reproduce it by running the "ramdomly" ofending code 100.000 times inside a loop (Script attached). Note that the test takes only a few seconds, and in my PC it throws regularly between 5 to 30 exception for all those cycles). I see there are other people suffering for the same, but without solution or workaround yet (https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html and https://github.com/IronLanguages/main/issues/1187)
In my case, weakref error was related with an intensive use of isinstance() inside "Pulp" library (an optimization python library). Just for your reference: isintance() use internally a WeakSet as a cache for the class types, which in turn use weakref (see ABCMeta.__instancecheck__() in the standard file "abc.py"). 

In my test script I have isolated the problem to WeakSet only (I isolated it to clean the bug from the "abc.py" and "Pulp" library stuff). The exception happens inside WeakSet.__contains__() function (from _weakrefset.py file). As stated, it happens randomly, but apparently only related with a GC collect cycle into a memory "hungry" python script . I ran the test script in two PCs with similar results: Windows 7 64bits and Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The .NET version is 4.0.30319.18444 (32-bit). The test script does: 
  1.. It simulate a "memory intensive" python code (creating a weakref object of 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then the bug don't show up) 
  2.. It manually runs GC.collect() every 1.000 cycles (should collect those weakref objects) 
  3.. ... and it repeat (100.000 times) the following "offending" boolean test:
       test = item in WeakSetObject  #<- Repeated 100.000 times. 
                                      #-> it fails between 10 to 20 times with an unexpected exception

NOTE 1: The "item" is an object added to the WeakSet at the beginning of the script, but "item" should not GC collected as it also keeps a normal (not weak) reference alive. 

NOTE 2: The boolean test should give always True, which is the case 99.9% of the time. Less than 0.01%, the boolean test fails raising an exception of the type "ValueError" (Description:"Index was out of range") or a bit more frequent "SystemError" (Description:"Handle is not initialized"). This happens 5 to 30 times in 100.000 test cycle (Seems very small, but it is important enough to avoid a practical construction of a medium size optimization problem with "Pulp" library). 
Tracking down the error, the exception ValueError is raised in line 70 and the exception "SystemError" in line 73 of "_weakrefset.py" .

    On "Lib\_weakrefset.py" 

    35 :class WeakSet(object):
    ....
    68 :    def __contains__(self, item):
    69 :        try:
    *70:           wr = ref(item)     # <- here is raised "ValueError" Exception ("Index was out of range")
    71 :        except TypeError:
    72 :            return False
    *73:       return wr in self.data # <- here is raised "SystemError" Exception ("Handle is not initialized")
Continuing after the exception, when executing the same boolean test again, it works fine (same item, same WeakSetObject, same execution, local and global context. Script was not started again!). I.e. if you catch the exception and continue the execution, It is like as if the exception never happened before (it's like a runtime lapsus!). 

I believe to fix the source of the problem, Ironpython should trap or avoid the error in weakref module itself (C# code). I don't know how... can someone kindly help me to fix this?

Cheers, 
Andres Sommerhoff





--------------------------------------------------------------------------------
_______________________________________________
Ironpython-users mailing list
Ironpython-users at python.org
https://mail.python.org/mailman/listinfo/ironpython-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20150413/c3eec56b/attachment.html>