[Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

[Frédéric Bastien]

Hi,

Up to now, I didn't post frequently to this mailing list. So I present
myself a little bit. I'm Frédéric Bastien, one of Theano senior
developer. I'm not a student nor a professor, but a staff of our lab.
So I more time for Theano then most people in our lab, but I have
other work too.

>From now on,  I'm going to read every weekday email on the numpy
mailing list. I think this will help communication between the Theano
and numpy community. We have many common goal and this can only be
useful to all of us. Also, if possible, I'll try to go to conference
where discussion will occur on the big change coming to numpy.


So now, here is my comments/questions to those change.

Currently Theano include some type of functionality that I didn't saw
in the this thread or others related to the refactoring of Numpy.

1) Theano do numerical stability optimization. (simple example: log(1
+ x) -> log1p(x))
2) It allow to do symbolic "graph generation". We use this to do
symbolic differentiation, to compute the R op and L op used in Hessian
free.
3) Automatically make some operation inplace on there input: a + b -> a += b
4) Mathematical optimization like a*2+a -> a * 3
5) Implementation optimization (planed in the new numpy)

Is there any plan to add 1-4 in the new project? Will it be
doable(especially 2). The Theano interface that ask the user to create
a Theano function was caused by the fact it is hard/time consuming to
work on the AST that we looked a little bit. I didn't looked at the
python AST myself, so I don't know if it was possible or just hard. Do
someone know if there is all the information that we need?

I don't recall discussion on having Theano work at the byte-code
level. I never looked at python byte-code, but I remember having read
that it change at each version of python? If that is true, how do you
plan to handle this? I don't think that rewriting each optimization
again at each release of python is acceptable. Event if python change
the byte-code only at each X.Y release.

I like the idea of having a common library to do array expression
optimization (the arrayExprOpt idea). Theano optimizations cover much
more type of optimization as said then just element-wise optimization.
Up to now, I only remember having read about the element-wise
optimization. Did I forget something? Do you plan more then this? If
the goal stay at this level, it mean in the end it target to replace
numexpr, but I think we should target more then this as there is many
high level optimization possible that are useful.


There is many projects that do optimized code generation that use SSE*
instruction for tensor operation like blitz++(already named) and
eigen2. Redoing all this work is time consuming. So any reuse can be
useful. Don't forget that many implementation optimization are size
and hardware dependent. Do you plan to go that low level? If there is
such low level optimization, I suppose it will be only for the
frequent case.



I'm not sure if this thread is the right one for this, but we started
a new project for a full tensor on the GPU. It will be the used by
Theano, PyCUDA and PyOpenCL. But this ask change to the ndarray
interface. The current code is a C library that use cython for its
interface. We made a mailing list for it[1]. The current version is in
a fork. I think it will be highly valuable that we have in the end the
same gpu nd array with numpy too. It only have been announced as a
goal for the new Numpy up to now. We have some work done. It will be
usable shortly. But don't expect all Numpy function implemented yet :)


Frédéric Bastien

[1] https://github.com/inducer/compyte/wiki