[Cython] Hudson pyregr testing takes too long

[Stefan Behnel]

Vitja Makarov, 25.04.2011 08:19:
> 2011/4/25 Stefan Behnel:
>> Stefan Behnel, 07.04.2011 13:52:
>>>
>>> Stefan Behnel, 07.04.2011 13:46:
>>>>
>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>> minutes for a run to ~4 hours when we added generator support.
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>
>>>> I currently have no idea why that is (well, it's likely because we
>>>> compile
>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>> would
>>>> be great if someone could find the time to analyse this problem. The
>>>> current run time makes it basically impossible to keep these tests
>>>> enabled.
>>>
>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>>> tests take 30-45 minutes, which is very long, but not completely out of
>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>
>> There seems to be a huge memory leak which almost certainly accounts for
>> this. The Python process that runs the pyregr suite ends up with about 50GB
>> of memory at the end, also in the latest Py3k builds.
>>
>> I have no idea where it may be, but it started to show when we merged the
>> generator support. That's where I noticed the instant jump in the runtime.
>
> That's very strange for my branch it takes about 30 minutes that is ok.

Does your branch leak memory when you run a generator? Using a debug build 
of CPython, running a Cython generator for the first time persistently 
increases the reference counts for me (using the generators.pyx test):

    Python 2.7.1+ (2.7:c821d3d335e8, Apr 22 2011, 18:37:12)
    [GCC 4.4.3] on linux2
    >>> import generators as G
    [17021 refs]
    >>> import gc
    [17462 refs]
    >>> gc.collect()
    0
    [17465 refs]
    >>> gc.collect()
    0
    [17465 refs]
    >>> list(G.with_outer(1,2,3,4)())
    [1, 2, 3, 4]
    [17474 refs]
    >>> gc.collect()
    0
    [17470 refs]

It seems like this leaked five references. And it seems to be related to 
the generator being inside of a closure itself:

   >>> list(G.test_first_assignment())
   [5, 10, (5, 10)]
   [17475 refs]
   >>> gc.collect()
   0
   [17470 refs]

back to the last value here, but:

   >>> list(G.generator_nonlocal()(5))
   [2, 3, 4, 5, 6]
   [17481 refs]
   >>> gc.collect()
   0
   [17476 refs]

Another six references leaked. And it's only the first time the generator 
is run, running it a second time doesn't change anything:

   >>> list(G.generator_nonlocal()(5))
   [2, 3, 4, 5, 6]
   [17481 refs]
   >>> gc.collect()
   0
   [17476 refs]

Stefan