[SciPy-dev] The future of SciPy and its development infrastructure

[David Cournapeau]

Hi Matthew

>> I think Stefan's position is that, as more people start using and
>> contributing to Scipy, it's become near impossible to maintain in a
>> release-worthy way (Stefan - is that right)?    That if we want to
>> keep going without collapsing we need a more formal process.

FWIW, that's my position as well. Several people complained about slow
releases - but getting faster releases is only possible with a non
linear timeline, where you don't accept random code near a release
date. Even linux itself has merge windows to limit things a few weeks
before a release, and Linux is not known to have a very formal
process.

To avoid red-herring, I will only work on real-case examples, as they
happened recently. I don't think anybody has been happy with the 18
months between 0.6 and 0.7. IMO, the bare minimum to do for a release
is:
 - check that it builds on windows, mac os X and Linux (both 32 and 64 bits)
 - check that it runs the test-suite
 - check that no blocker issue is kept opened

This cannot be done if we don't have tests for new features. If code
is pure python and pure computation, I don't mind so much, as long as
it passes basic sanity check, but if it involves C or worse Fortran,
it is almost guaranteed to break somewhere. Most people only code on
one platform, and don't know that it may break something on python
2.4, or python 2.6, or on windows x64, or visual studio, or solaris,
etc... Again, a concrete example: kdtree code, through its use of
cython, was broken on python 2.4 on linux 64 bits, because of a cython
bug. When this happens a few days before a planned release, it is
enough to break one more RC, which takes several hours of work for
release managers (to rebuild the binaries, set up things on
sourceforge, etc...). To be clear, I do not blame the author of the
code, I don't expect every contributor to check for those things. But
I expect people to care that it is work for other people, and that
somebody else has to check those things.

Already having "merge windows", and blocked windows (nobody can commit
anything without approval from the release management team) would be a
huge gain. It would really help for making releases - I don't know a
single big open source project which does not use this process in one
way or the other. I don't feel like our process is on par with the
size of scipy at this point.

David