[Cython] array expressions

[mark florisson]

On 23 October 2012 10:47, mark florisson <markflorisson88 at gmail.com> wrote:
> On 23 October 2012 02:44, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Sun, Oct 21, 2012 at 5:39 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>>> mark florisson, 14.10.2012 13:59:
>>>>>>> The problem with minivect as a package is that it caters to different
>>>>>>> projects, which have different requirements. Cython and minivect are
>>>>>>> quite closely coupled, and any future change, or in the future any
>>>>>>> older version may not have the functionality Cython needs, it's not
>>>>>>> exactly a stable API at this point.
>>>>>>
>>>>>> Ok, understood.
>>>>>>
>>>>>>
>>>>>>> For instance Numba needs python
>>>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>>>
>>>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>>>> compatibility. And that is really because not all use cases have yet
>>>>>>> been anticipated, and some really require a change, as I've already
>>>>>>> seen with Numba.
>>>>>>>
>>>>>>> We could list minivect as a dependency, which works for
>>>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>>>> into problems that didn't install with pip, and I don't think an
>>>>>>> exclusion of a 300kb addition is worth any of that.
>>>>>>
>>>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>>>> but including it directly and considering it a part of Cython (and Numba
>>>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>>>> package, or at least to support a separate installation and independent
>>>>>> update. Basically, if you can't update it separately, there's no use in
>>>>>> installing it separately.
>>>>>>
>>>>>> As long as we handle this so, we should take care to keep the generic parts
>>>>>> in their separate package directory and the Cython specific parts in
>>>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>>>> as possible, so that we can actually reach a point where both have an
>>>>>> interface. I would guess that the need to support Numba from the same
>>>>>> source base will encourage this kind of separation anyway.
>>>>>
>>>>> Yes, definitely.
>>>>>
>>>>>> Note that this means that minivect will fall under the release schedules of
>>>>>> Cython and Numba (independently), instead of really having its own releases.
>>>>>
>>>>> It can have its own releases as well, but currently there isn't much
>>>>> point :) Minivect can be developed independent of the releases, since
>>>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>>>> habit of squashing the minivect pulls to avoid its history.
>>>>>
>>>>> I'll also wait for Dag and Robert to see if they have a (final)
>>>>> opinion before merging the subtree.
>>>>
>>>> As I mentioned on the pull request, looks good to me. Given the
>>>> (somewhat) tight coupling with the AST but the desire to use the
>>>> codebase for multiple projects, a subtree seems to make the most sense
>>>> (until/if we have some kind of a plugin system).
>>>>
>>>> - Robert
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Great, thanks for the review Robert. There seems to be a refcount
>>> issue with python 3:
>>>
>>> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
>>> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
>>> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
>>> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> from Cython.Compiler import Main
>>> [98632 refs]
>>>>>> ^D
>>> [98632 refs]
>>> python: Modules/gcmodule.c:327: visit_decref: Assertion
>>> `gc->gc.gc_refs != 0' failed.
>>>
>>> The object being visited is a UtilityCode object. I think the error
>>> means the object is being visited more often than it's refcount value,
>>> which means it's not increffed properly somewhere. I'm not sure how/if
>>> my changes introduced this, has this problem been encountered recently
>>> in Cython's master?
>>
>> Not that I'm aware of. The refnanny is supposed to catch this kind of
>> thing, is it enabled?
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Yeah, it doesn't catch it. I'll try tracing the imports to see what it
> is importing that is different from master. Thanks.

I had some time to look into it, and it appeared that the bug was in
CloneNode, it just never got triggered, unless you analyse the types
of the CloneNode. That sets is_temp to True, but the node that's being
cloned may not own its reference, but temporaries are assumed to own
it.