[Numpy-discussion] SciPy Sprint results

[Travis E. Oliphant]

Hi all,

We had a great Sprint at Berkeley over last weekend.  Jarrod deserves a 
huge hand for organizing it and Fernando should be also congradulated 
for making the Sprint a productive communication session with a lot of 
different people. 

Going forward, there will be a relatively informal SciPy board whose 
purpose is to keep SciPy (and NumPy) moving forward.   Currently, this 
board consists of  (alphabetically)

Eric Jones
Robert Kern
Jarrod Millman
Travis Oliphant

Our goal is to clean up SciPy and get it ready for 1.0 release over the 
next year or so (which will need lots of help from the community).   If 
anybody else is interested in serving on this board, just send me email. 

As part of this goal, we will be having regular "sprints" as well 
virtual "bug-days" and "doc-days" where people who want to participate 
using IRC can join in and coordinate efforts.   There will be at least 
one bug-day or doc-day every month over the next year (on the last 
Friday of the month).   The first one is a "doc-day" which will be held 
Friday on December 28, 2007 (getting started on New Year's resolutions 
early).   This doc-day will be virtual where anyone with an internet 
connection can join in on the scipy channel on  irc.freenode.net.

At least one board member will be available at each "doc-day" or 
"bug-day" (even if we have to recruit board members to make it happen :-) )

The recent Sprint was very helpful.  Jarrod is putting together some 
material from the Sprint.  But, I wanted to provide over-view 
information for those who may be interested in what happend.

Summary:

A lot of great discussion took place (and some fine actual coding by a 
few) which resulted in the following plans:

Schedule
------------
  * NumPy 1.0.5 in mid January
  * SciPy 0.7.0 in mid March to April
  * NumPy 1.1 by August 2008 (may slip a bit depending on what is wanted 
to be included)

The plans below are for NumPy 1.0.5 and SciPy 0.7.0 unless otherwise noted.

IO
----
  * scipy.io will be gutted and what functionality remains will be 
placed in numpy.io.
  * scipy.io will be a place for file readers and writers for various 
data formats (data, audio, video, images, matlab, excel, etc.)
  * NumPy will get a standard binary file format (.npy/.npz) for 
arrays/groups_of_arrays.
  * NumPy will be trying to incorporate some of matplotlib's csv2rec and 
rec2csv functionality.
  * Pickling arrays will be discouraged (you will still be able to do 
it, we will just try to not make it seem that it is the "default" way to 
save arrays).

Testing
---------
  * scipy unit-testing will be "nose-compliant" and therefore nose will 
be required to run the SciPy tests. 
  * NumPy will still use the current testing framework but will support 
SciPy's desire to be nose-compliant.  NumPy 1.1 tests may move to just 
being "nose-compliant"
  * The goal is to make tests easier for contributors to write.

Weave
---------
  * weave will not move into NumPy yet, but possibly at NumPy 1.1, there 
could be a separate package containing all the "wrapping" support code 
for NumPy in a more unified fashion (if somebody is interested in this, 
it is a great time to jump in).

Sandbox
-----------
  * the scipy sandbox is disappearing (except for user playgrounds) and 
useful code in it will be placed in other areas.

Python versions
--------------------
  * SciPy 0.7.0 will require Python 2.4 (we can now use decorators for 
SciPy).
  * NumPy will still be useful with Python 2.3 until at least 1.1

Other discussions
----------------------
  * numpy-scons will be a separate package for now for building 
extensions with scons (we need experience to figure out what to do with 
it).
  * fixes to repr for numpy float scalars were put in place
  * Thanks to Rob Falck scipy.optimize grew slsqp  (sequential 
least-squares programming) method (allows for equality and inequality 
constraints).   The code by Dieter Kraft was wrapped.
  * We will be working to coordinate efforts with William Stein (of SAGE 
fame) in the future.  Sage developers will be coming to Austin at the 
end of February to do some cooperative sprinting.
  * Brian Granger is working on a parallel version of NumPy that is very 
interesting.


Deprecation approaches
-------------------------------

Functions in SciPy that are disappearing will be "deprecated" with 
appendages to the docstring to explain how to do it differently.  The 
deprecation will issue a warning when the function is run.  In the next 
version, the function will disappear.   Once SciPy hits 1.0, the 
deprecation paradigm will be a bit more conservative.


A lot of fabulous things are happening with SciPy.  It is an exciting 
time to be a part of it.   There are a lot of ways to jump in and 
participate so feel free.   If there is something you think needs 
addressing, then please share it.   We may have a simple PEP process in 
the future, but for now the rule is basically "working code."


Best regards,

-Travis O.