[Numpy-discussion] Proposed Roadmap Overview

[Matthew Brett]

On Sat, Feb 18, 2012 at 1:40 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Feb 18, 2012 at 2:17 PM, David Cournapeau <cournape at gmail.com>
> wrote:
>>
>> On Sat, Feb 18, 2012 at 8:45 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett
>> >> > <matthew.brett at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi.
>> >> >>
>> >> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire
>> >> >> <cjordan1 at uw.edu> wrote:
>> >> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett
>> >> >> > <matthew.brett at gmail.com> wrote:
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire
>> >> >> >> <cjordan1 at uw.edu> wrote:
>> >> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden
>> >> >> >>> <sturla at molden.no>
>> >> >> >>> wrote:
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout
>> >> >> >>>> <jason-sage at creativetrax.com>:
>> >> >> >>>>
>> >> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote:
>> >> >> >>>>>> We would have to write a C++ programming tutorial that is
>> >> >> >>>>>> based
>> >> >> >>>>>> on
>> >> >> >>>>>> Pyton knowledge instead of C knowledge.
>> >> >> >>>>>
>> >> >> >>>>> I personally would love such a thing.  It's been a while since
>> >> >> >>>>> I
>> >> >> >>>>> did
>> >> >> >>>>> anything nontrivial on my own in C++.
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>>> One example: How do we code multiple return values?
>> >> >> >>>>
>> >> >> >>>> In Python:
>> >> >> >>>> - Return a tuple.
>> >> >> >>>>
>> >> >> >>>> In C:
>> >> >> >>>> - Use pointers (evilness)
>> >> >> >>>>
>> >> >> >>>> In C++:
>> >> >> >>>> - Return a std::tuple, as you would in Python.
>> >> >> >>>> - Use references, as you would in Fortran or Pascal.
>> >> >> >>>> - Use pointers, as you would in C.
>> >> >> >>>>
>> >> >> >>>> C++ textbooks always pick the last...
>> >> >> >>>>
>> >> >> >>>> I would show the first and the second method, and perhaps
>> >> >> >>>> intentionally forget the last.
>> >> >> >>>>
>> >> >> >>>> Sturla
>> >> >> >>>>
>> >> >> >>
>> >> >> >>> On the flip side, cython looked pretty...but I didn't get the
>> >> >> >>> performance gains I wanted, and had to spend a lot of time
>> >> >> >>> figuring
>> >> >> >>> out if it was cython, needing to add types, buggy support for
>> >> >> >>> numpy,
>> >> >> >>> or actually the algorithm.
>> >> >> >>
>> >> >> >> At the time, was the numpy support buggy?  I personally haven't
>> >> >> >> had
>> >> >> >> many problems with Cython and numpy.
>> >> >> >>
>> >> >> >
>> >> >> > It's not that the support WAS buggy, it's that it wasn't clear to
>> >> >> > me
>> >> >> > what was going on and where my performance bottleneck was. Even
>> >> >> > after
>> >> >> > microbenchmarking with ipython, using timeit and prun, and using
>> >> >> > the
>> >> >> > cython code visualization tool. Ultimately I don't think it was
>> >> >> > cython, so perhaps my comment was a bit unfair. But it was
>> >> >> > unfortunately difficult to verify that. Of course, as you say,
>> >> >> > diagnosing and solving such issues would become easier to resolve
>> >> >> > with
>> >> >> > more cython experience.
>> >> >> >
>> >> >> >>> The C files generated by cython were
>> >> >> >>> enormous and difficult to read. They really weren't meant for
>> >> >> >>> human
>> >> >> >>> consumption.
>> >> >> >>
>> >> >> >> Yes, it takes some practice to get used to what Cython will do,
>> >> >> >> and
>> >> >> >> how to optimize the output.
>> >> >> >>
>> >> >> >>> As Sturla has said, regardless of the quality of the
>> >> >> >>> current product, it isn't stable.
>> >> >> >>
>> >> >> >> I've personally found it more or less rock solid.  Could you say
>> >> >> >> what
>> >> >> >> you mean by "it isn't stable"?
>> >> >> >>
>> >> >> >
>> >> >> > I just meant what Sturla said, nothing more:
>> >> >> >
>> >> >> > "Cython is still 0.16, it is still unfinished. We cannot base
>> >> >> > NumPy
>> >> >> > on
>> >> >> > an unfinished compiler."
>> >> >>
>> >> >> Y'all mean, it has a zero at the beginning of the version number and
>> >> >> it is still adding new features?  Yes, that is correct, but it seems
>> >> >> more reasonable to me to phrase that as 'active development' rather
>> >> >> than 'unstable', because they take considerable care to be backwards
>> >> >> compatible, have a large automated Cython test suite, and a major
>> >> >> stress-tester in the Sage test suite.
>> >> >>
>> >> >
>> >> > Matthew,
>> >> >
>> >> > No one in their right mind would build a large performance library
>> >> > using
>> >> > Cython, it just isn't the right tool. For what it was designed for -
>> >> > wrapping existing c code or writing small and simple things close to
>> >> > Python
>> >> > - it does very well, but it was never designed for making core C/C++
>> >> > libraries and in that role it just gets in the way.
>> >>
>> >> I believe the proposal is to refactor the lowest levels in pure C and
>> >> move the some or most of the library superstructure to Cython.
>> >
>> >
>> > Go for it.
>>
>> The proposal of moving to a core C + cython has been discussed by
>> multiple contributors. It is certainly a valid proposal. *I* have
>> worked on this (npymath, separate compilation), although certainly not
>> as much as I would have wanted to. I think much can be done in that
>> vein. Using the "shut up if you don't do it" is a straw man (and
>> uncalled for).
>
>
> OK, I was annoyed.

By what?

Best,

Matthew