[Cython] array expressions

[Dag Sverre Seljebotn]

On 10/14/2012 12:23 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 14.10.2012 11:03:
>> On 10/14/2012 10:32 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 14.10.2012 10:18:
>>>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>>> mark florisson, 13.10.2012 20:30:
>>>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>>>
>>>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>>>> requires a bit of adaptation.
>>>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>>>
>>>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>>>> interface than the implementation).
>>>>>>>
>>>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>>>> code and
>>>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>>>
>>>>>>>> I don't see a reason for a new pull request.
>>>>>>
>>>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>>>> to split of, merge, push, and pull subdirectories.
>>>>>>
>>>>>> This means when users pull the master project, they get the
>>>>>> sub-projects as well (without themselves needing newer git versions).
>>>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>>>> the subproject's history).
>>>>>>
>>>>>> What about using this approach? That way Cython remains stable and
>>>>>> pinned on the right minivect version now and in the future, with no
>>>>>> burden on users.
>>>>>
>>>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>>>> either, even though a lot of people use Cython together with it.
>>>>
>>>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>>>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>>>> itself and working on the AST level.
>>>>
>>>> Plex would be a better comparison (though bad as well, since Plex is not
>>>> optional while minivect is).
>>>>
>>>>> Keeping the two packages separate helps in keeping the interface between
>>>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>>>> version of minivect just because it's so easy, and I would like to allow
>>>>> users to install a new version of either Cython or minivect at any time.
>>>>
>>>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>>>> unrealistic and pointless.
>>>>
>>>> I think you should look at minivect as "some AST transform algorithms which
>>>> numba and Cython are able to share". It doesn't really have a life on its
>>>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>>>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>>>> won't happen just yet. If it does, we can revisit this.)
>>>
>>> Ok, so I gave bad examples, fair enough. But I still don't see why we
>>> should integrate minivect with Cython more deeply than necessary. Could you
>>> explain what the advantage would be over having two separate packages?
>>
>> Mark already raised the issue and we discussed in through in another thread
>
> This is still the same thread, at least for me. I replied to it when asking
> for progress.
>
>
>> where you participated, and me, Mark and Robert at least agreed on
>> something very similar to what Mark is proposing to do now.
>>
>> If you want to reopen the discussion I wish you had rather dug up the old
>> thread again and respond to arguments in it, rather than us having to
>> repeat the arguments here.
>
> The single argument for git based integration was that it simplifies the
> workflow during development. Meaning, it's a development-only thing and
> deployments would use separate package installations.
>
> I think that's ok as long as it's actually helpful for development and
> doesn't start getting in the way. I can't comment on that yet, but I
> wouldn't want to have to deal with it.

You know what -- I didn't think things through and was totally wrong here.

'git subtree' has a very different feel to it than 'git submodule'; it 
means that minivect will be in the tree by default and would have to be 
stripped out when making a release (or included by default).

So revisiting this discussion is indeed in order, as git subtree wasn't 
mentioned last time.

Dag Sverre

>
>
>> (Though please also consider what it does to discussion climate to reopen
>> discussions about trivial details that can also trivially be changed later
>> if better arguments crop up.
>
> It may be a trivial detail as long as it only has a reasonable impact on
> the development. If it has a user impact, it's no longer immediately
> obvious that it's trivial.
>
>
>> Mark had a weekend of spare time to get this
>> merged (yay!), and now who-knows-what happens because you decided to
>> bikeshed something that had, at least to the eyes of rest of us, achieved
>> consensus on this list.)
>
> Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of
> what he's doing now just wasn't clear enough to me.
>
> Mark, could you explain in a bit more detail in which cases the git version
> will be used and in which cases users would (or can) install minivect
> separately?
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>