[Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/4/25 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/4/25 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 25.04.2011 11:04:
>>>
>>> 2011/4/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>
>>>> Vitja Makarov, 25.04.2011 08:19:
>>>>>
>>>>> 2011/4/25 Stefan Behnel:
>>>>>>
>>>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>>>
>>>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>>>
>>>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>>>
>>>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>>>> compile
>>>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>>>> would
>>>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>>>> current run time makes it basically impossible to keep these tests
>>>>>>>> enabled.
>>>>>>>
>>>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>>>> Py2.7
>>>>>>> tests take 30-45 minutes, which is very long, but not completely out
>>>>>>> of
>>>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>>>
>>>>>> There seems to be a huge memory leak which almost certainly accounts
>>>>>> for
>>>>>> this. The Python process that runs the pyregr suite ends up with about
>>>>>> 50GB
>>>>>> of memory at the end, also in the latest Py3k builds.
>>>>>>
>>>>>> I have no idea where it may be, but it started to show when we merged
>>>>>> the
>>>>>> generator support. That's where I noticed the instant jump in the
>>>>>> runtime.
>>>>>
>>>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>>>
>>>> There's also a second path that's worth investigating. As part of the
>>>> merge,
>>>> there was another change that came in: the CythonPyregrTestCase
>>>> implementation. This means that the regression tests are now being run
>>>> differently than before. The massive memory consumption may simply be due
>>>> to
>>>> the mass of unit tests being loaded into memory.
>>>
>>>    def run_test():
>>> ..................................
>>>         try:
>>>             module = __import__(self.module)
>>>             if hasattr(module, 'test_main'):
>>>                 module.test_main()
>>>         except (unittest.SkipTest, support.ResourceDenied):
>>>             result.addSkip(self, 'ok')
>>>
>>>
>>> It seems that all the modules stay loaded so may be they should be
>>> unloaded with del sys.modules[module_name]?
>>
>> (Binary) module unloading isn't really supported in CPython. There's PEP
>> 3121 that has the potential to change it, but it's not completely
>> implemented, neither in CPython nor in Cython. A major problem is that
>> unloading a module deletes its globals but not necessarily the code that
>> uses them. For example, instances of types defined in the module can still
>> be alive at that point.
>>
>> The way runtests.py deals with this is forking before loading a module.
>> However, this does not currently work with the "xmlrunner" which we use on
>> Hudson, so we let all tests run in a single process there.
>>
>
>
> Btw when running plain python code with generators total ref counter
> doesn't get back to initial value.
> I tried to trace scope and generator destructors and they are run as
> expected. So I'm not sure about leaks in generators.
>

Recently I've found that pyregr.test_dict (test_mutatingiteration)
test makes it slow:

def test_mutatingiteration():
    d = {}
    d[1] = 1
    for i in d:
        print i
        d[i+1] = 1

test_mutatingiteration()


In CPython this code raises: RuntimeError: dictionary changed size
during iteration
And in Cython you have infinite loop. So we can disable this test for now.


-- 
vitja.