[Cython] test crash in Py3.2 (NumPy/memoryview/refcounting related?)

[mark florisson]

On 22 April 2012 13:34, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 22.04.2012 13:54:
>> On 22 April 2012 11:57, Stefan Behnel wrote:
>>> I keep seeing test crashes in Py3.2 debug builds on Jenkins with the latest
>>> master, referring to ref-counting problems. Here, for example:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py32-ext/328/console
>>>
>>> and likewise in the series of builds starting at #312.
>>>
>>> I'm not sure what changed in that corner, it looks like one of those
>>> problems that was there for a while and suddenly starts showing when you
>>> touch that innocent looking brick at the other end of the house that was
>>> holding the balance, until now. In this case, that brick was this commit:
>>>
>>> https://github.com/cython/cython/commit/c8f61a668d0ce8af1020f520253e1b5f623cf349
>>>
>>> I reverted it here and that fixed the problem at least temporarily:
>>>
>>> https://github.com/cython/cython/commit/5aac8caf1ba21933cc85a51af3319c78fc08d675
>>>
>>> but it seems to be back now (after my refnanny optimisations). Before
>>> reverting my changes, I was able to reproduce it somewhat reliably on
>>> sage.math by running these three tests together (non-forking):
>>>
>>>        memslice numpy_bufacc numpy_memoryview
>>>
>>> None of the tests shows the problem when run by itself. I can't tell if
>>> it's also in the latest py3k because I don't have a NumPy lying around that
>>> works there. So 3.2 is basically the latest Python version this can be
>>> tested with, and it doesn't occur in the 3.1 tests. The tests use NumPy
>>> 1.6.1, i.e. the latest official release.
>>>
>>> Mark, Dag, could any of you take a look to see if it appears in any way
>>> more obvious to you than to me?
>>
>> Hm, I think you can try to disable the numpy_memoryview test and just
>> continue development as normal. numpy_memoryview has a testcase
>> function which inserts the test into __test__, so you could just
>> comment out that line.
>
> ... as long as we remember to put it back in ;)
>
>
>> The test seems to fail before it runs though? Is it possible to obtain
>> a backtrace?
>
> When I reproduced it in the Jenking workspace, it crashed while trying to
> clean up objects to free memory, specifically in the deallocation visitor
> function of one of the memory views of numpy_memoryview (IIRC). That didn't
> really tell me where that object was created or what happened to it to get
> it to crash. Needs some more investigation.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Hm, I can't reproduce the issue, but that assertion triggers only for
update_refs and subtract_refs (when the gc is trying to determine
which object may be potentially unreachable). I think the problem here
is that a single memoryview object is traversed multiple times through
different traverse functions, and that the refcount doesn't match the
number of traverses. Indeed, the refcount is only one, as the actual
count is the acquisition count. So we shouldn't traverse the
memoryview objects in memoryview slices, i.e. not
_memoryviewslice.from_slice.memview. I'll come up with a commit
shortly, would you be willing to test it?