[Cython] Cython <> pythran collaboration

[Stefan Behnel]

Robert Bradshaw schrieb am 09.12.2014 um 18:02:
> On Mon, Dec 8, 2014 at 11:52 AM, serge Guelton wrote:
>> I am one of the two core developers of the Pythran[0] compiler.
>> We are planning to start a project where Cython would delegate the
>> generation of optimized (i.e. parallel / vectorized / without temporary
>> arrays) numpy expression when it finds one, instead of calling the
>> native API. Something like the following meaningless example:
>>
>>     import numpy as np
>>     cimport numpy as np
>>
>>     def play_with_arrays(np.ndarray arr):
>>         cdef int N = arr.shape[0]
>>         cdef np.ndarray other_arr = np.empty([N], dtype=np.int)
>>
>>         other_arr = arr ** 2 + 1 # pythran would handle this
>>
>>         for i in range(N):
>>             other_arr[i] /=  2 # cython would handle this
>>
>>         return  other_arr
>>
>> Everything would live in a pyx file and it should be transparent to a
>> cython user. Cython would drive the compilation process, using pythran
>> as a numpy expression optimizer.
>>
>> Does that look feasible / appealing to you?
>>
>> [0] http://pythonhosted.org/pythran/
> 
> Yes, that would be very cool; efficient array operations (including
> broadcasting) without intermediates is something we've long wanted to
> have.

+1

There were previous discussions about this on both Cython mailings lists
(dev/users), e.g.

http://thread.gmane.org/gmane.comp.python.cython.devel/15357

http://thread.gmane.org/gmane.comp.python.cython.user/8275

http://article.gmane.org/gmane.comp.python.cython.devel/14296

One of the outcomes was that we'd like to have an external tool doing the
expression optimisation work, so pythran would fit here.

The main idea is to write some special syntax tree nodes that process array
expressions with pythran, and to replace the existing generic tree nodes
(see AddNode, MulNode & friends in ExprNodes.py) with the pythran specific
nodes during expression analysis (see Pipeline.py) whenever we notice that
we are working on memory views. Nodes do their own code generation, which
would be handed to pythran in your case. The specific external tool to use
(i.e. which set of tree nodes to inject) would depend on a compiler
directive, so the pythran integration code should go into a separate module.

All type information regarding the memory views is available to the tree
nodes at expression analysis time, so integrating an external tool should
not be difficult. Node replacement is done by simply returning a different
node instance from its analyse_types() method. There's a helper classmethod
"from_node()" for instantiating an ExprNode subtype from another tree node
(and properly copy over its external tree state).

Further obstacles can be discussed on this list.

Stefan