[SciPy-Dev] Discontinued Rackspace Open Compute Discount

[Olivier Grisel]

Status update:

- I got a reply from Rackspace and and they agree to continue the discount
through April, 2020. This should give us enough time to move
operations. I also deleted many useless costly resources on that
account so that the running cost for the coming month should not
exceed $100/month even if we go beyond end of April.

- I already upgraded the
https://github.com/MacPython/scikit-learn-wheels infrastructure to
upload nightly builds and temporary release artifacts to anaconda.org,
details below.

- OpenBLAS builds used for numpy and scipy will be pushed to github
releases: https://github.com/MacPython/openblas-libs/issues/14

- We still need to update the MacPython/*-wheels for the rest of the
projects but I can help prepare some pull requests.

Based on past observed CDN traffic on the Rackspace account (around
800 GB/month), it should be fine to use the free anaconda.org package
hosting service to host the nightly builds for all the community
projects that currently rely on http://wheels.scipy.org.

For scikit-learn, I decided to keep the release staging artifacts
separated from the nightly build artifacts:

- https://anaconda.org/scipy-wheels-nightly/ as a shared host for all
nightly builds
- https://anaconda.org/scikit-learn-wheels-staging/ as a temporary
store for scikit-learn wheels prior to final upload to pypi.org when
making a release.

Here is the configuration in scikit-learn CI that does the wheel
upload to anaconda.org:

https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0db1d7a93b7435408f4/azure/posix.yml#L108-L141

## Shared nightly build hosting

The scipy-wheels-nightly organization is meant to make it easier for
numpy, scipy, pandas, scikit-learn and other projects with long build
time to publish binaries for their development branch on a daily basis
so that continuous integration can efficiently run their tests against
the dev branch of their dependencies, so as to spot regressions, ASAP
before making a release.

For instance, once numpy and scipy have been configured to upload
nightly builds there it will be possible to reconfigure the scheduled
tests of a project that depends on scikit-learn to run against the dev
branch of the full stack using:

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple
scikit-learn
pip install --no-build-isolation -e .
pytest

If you are a maintainer of one of the projects that publishes
nightly-builds on http://wheels.scipy.org, **please create an account
on https://anaconda.org and reply to this email with your identifier**
so that I can grant you permissions on this shared organization.

>From there you will be able to generate tokens to use as secret
variable in your CI servers. To use those access tokens in a CI setup
will need to enable permissions for the token:

- Write access to API
- Upload pypi packages

## Staging hosting for release artifacts

Each project is free to use its own staging area. Staging areas are
just useful as a synchronization step in a multi-CI release automation
setup. There is really no need to publish those wheels to end-users.
The end users will grab the wheels of the released versions from the
main https://pypi.org index once the release is done.

That being said, I plan to also create a default staging area for
convenience, e.g.:

- https://anaconda.org/multibuild-wheels-staging/

for projects who do not really care and want to use the
https://github.com/matthew-brett/multibuild scripts with minimal admin
efforts.

Thanks for your attention, let me know if you have any comment on this plan.

-- 
Olivier