[Numpy-discussion] reducing effort spent on wheel builds?

[Ralf Gommers]

On Thu, Jul 15, 2021 at 2:41 PM Matti Picus <matti.picus at gmail.com> wrote:

> On 15/7/21 1:21 pm, Ralf Gommers wrote:
>
> > Hey all,
> >
> > I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo
> > again recently, and I've done the same for SciPy. ...
> >
> > Cheers,
> > Ralf
>
>
> Since my name was mentioned, the things I have spent time on for wheel
> packaging in order of time spent (not scientifically measured rather
> looking back at the closed PRs and thinking "oh yeah, that was painful")
> have been
>
> - Updating manylnux to move away from 10 year old glibc on linux (still
> stuck and not clear how to finish it [0])
>
> - Moving from travis-ci.org to travis-ci.com (with the panic around
> build credits) and from Appveyor to github actions/azure pipelines
>
> - Moving from rackspace's wheel hosting to anaconda.org
>
> - Working around CI failures with aarch64 for linux, mainly due to
> shortcomings in the free CI providers
>
> - Bugs in dependencies: avoiding buggy Accelerate when building NumPy
> and debugging Windows/aarch64 problems with OpenBLAS
>
> - Updating OpenBLAS versions
>
> - Shepherding Apple M1 hardware through the manylinux/multibuild/wheel
> pipeline (all the hard work was done by others)
>
> - Trying to make 64-bit interfaces for OpenBLAS work (99.9% done by Chuck)
>
> - Updating PyPy versions
>

Thanks Matti! This list is super helpful.


> Only the last one, which was actually the least painful, would be helped
> by Ralf's list.
>

Not so sure about that - probably the single biggest pain points are CI
providers (especially the exotic ones) and OpenBLAS - a dependency we
struggle with mostly because of wheels. Without ppc64le and s390x (forgot
that one) we wouldn't need TravisCI at all for example, and we would have
less work on https://github.com/MacPython/openblas-libs too.

On the other hand, packaging is made harder as more technologies go into
> a wheel build. The twitter thread started with "SciPy added a required
> dependency on a technology which broke things, but people stepped up to
> fix the problem quickly" and morphed into "lets drop wheels for lesser
> used platforms". I am not sure the discussion should have moved away
> from the first point so quickly. Perhaps there should be some discussion
> of the cost of adding new build dependencies and somehow making the
> dependency conditional for a release or two until all the worst kinks
> are worked out.
>

Not quite the right mailing list, but: it *is* an optional dependency in
scipy 1.7.0, and there has been no proposal so far to make it required.


>
> For the record, I am +1 on removing sdists from PyPI until pip changes
> its default to --only-binary :all: [1]
>
> Matti
>
>
> [0] https://github.com/pypa/manylinux/issues/1012


Ah yes:( That just seems like a major oversight in the perennial manylinux
spec. If no one fixes it, guess we wait till we can jump to `_2_??` with a
higher GCC requirement.

Cheers,
Ralf


> [1] https://github.com/pypa/pip/issues/9140
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20210715/e5562fe8/attachment.html>