[Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[Travis Oliphant]

Believe me, I'm all for incremental changes if it is actually possible and
doesn't actually cost more.  It's also why I've been silent until now about
anything we are doing being a candidate for a NumPy 2.0.  I understand the
challenges of getting people to change.  But, features and solid
improvements *will* get people to change --- especially if their new
library can be used along with the old library and the transition can be
done gradually. Python 3's struggle is the lack of features.

At some point there *will* be a NumPy 2.0.   What features go into NumPy
2.0, how much backward compatibility is provided, and how much porting is
needed to move your code from NumPy 1.X to NumPy 2.X is the real user
question --- not whether it is characterized as "incremental" change or
"re-write".     What I call a re-write and what you call an
"incremental-change" are two points on a spectrum and likely overlap
signficantly if we really compared what we are thinking about.

One huge benefit that came out of the numeric / numarray / numpy transition
that we mustn't forget about was actually the extended buffer protocol and
memory view objects.  This really does allow multiple array objects to
co-exist and libraries to use the object that they prefer in a way that did
not exist when Numarray / numeric / numpy came out.    So, we shouldn't be
afraid of that world.   The existence of easy package managers to update
environments to try out new features and have applications on a single
system that use multiple versions of the same library is also something
that didn't exist before and that will make any transition easier for
users.

One thing I regret about my working on NumPy originally is that I didn't
have the foresight, skill, and understanding to work more on a more
extended and better designed multiple-dispatch system so that multiple
array objects could participate together in an expression flow.   The
__numpy_ufunc__ mechanism gives enough capability in that direction that it
may be better now.

Ultimately, I don't disagree that NumPy can continue to exist in
"incremental" change mode ( though if you are swapping out whole swaths of
C-code for Cython code --- it sounds a lot like a "re-write") as long as
there are people willing to put the effort into changing it.   I think this
is actually benefited by the existence of other array objects that are
pushing the feature envelope without the constraints --- in much the same
way that the Python standard library is benefitted by many versions of
different capabilities being tried out before moving into the standard
library.

I remain optimistic that things will continue to improve in multiple ways
--- if a little "messier" than any of us would conceive individually.   It
*is* great to see all the PR's coming from multiple people on NumPy and all
the new energy around improving things whether great or small.

Best,

-Travis

​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140604/4f5193c2/attachment.html>