[Cython] array expressions

[mark florisson]

On 14 October 2012 09:32, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 14.10.2012 10:18:
>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>> mark florisson, 13.10.2012 20:30:
>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>
>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>> requires a bit of adaptation.
>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>
>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>> interface than the implementation).
>>>>>
>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>> code and
>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>
>>>>>> I don't see a reason for a new pull request.
>>>>
>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>> to split of, merge, push, and pull subdirectories.
>>>>
>>>> This means when users pull the master project, they get the
>>>> sub-projects as well (without themselves needing newer git versions).
>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>> the subproject's history).
>>>>
>>>> What about using this approach? That way Cython remains stable and
>>>> pinned on the right minivect version now and in the future, with no
>>>> burden on users.
>>>
>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>> either, even though a lot of people use Cython together with it.
>>
>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>> itself and working on the AST level.
>>
>> Plex would be a better comparison (though bad as well, since Plex is not
>> optional while minivect is).
>>
>>> Keeping the two packages separate helps in keeping the interface between
>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>> version of minivect just because it's so easy, and I would like to allow
>>> users to install a new version of either Cython or minivect at any time.
>>
>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>> unrealistic and pointless.
>>
>> I think you should look at minivect as "some AST transform algorithms which
>> numba and Cython are able to share". It doesn't really have a life on its
>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>> won't happen just yet. If it does, we can revisit this.)
>
> Ok, so I gave bad examples, fair enough. But I still don't see why we
> should integrate minivect with Cython more deeply than necessary. Could you
> explain what the advantage would be over having two separate packages?

The problem with minivect as a package is that it caters to different
projects, which have different requirements. Cython and minivect are
quite closely coupled, and any future change, or in the future any
older version may not have the functionality Cython needs, it's not
exactly a stable API at this point. For instance Numba needs python
2.7, whereas Cython needs to be compatible with python 2.4.

Before releasing minivect I'll verify every time that it doesn't break
Cython, but I currently have no real promises for backwards or forward
compatibility. And that is really because not all use cases have yet
been anticipated, and some really require a change, as I've already
seen with Numba.

We could list minivect as a dependency, which works for
easy_install/pip users, but I just foresee numerous people running
into problems that didn't install with pip, and I don't think an
exclusion of a 300kb addition is worth any of that.

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel