[SciPy-Dev] proposal to use SciPy project funds for build and CI improvements

Mon Jul 5 01:53:47 EDT 2021

+1 for both the project and the new repository.

(Glad to read about the log as well!)

Thanks Ralf.

Cheers,
Pamphile 

> On 5 Jul 2021, at 00:01, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
> 
> 
> +1
> 
>> On Sun, 4 Jul 2021 at 15:47, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> Hi all,
>> 
>> I'd like to make a proposal to use some project funds to make faster progress on moving to a new build system and improving our CI timeout issues. 
>> 
>> Because this is the first such proposal, let me first deal with the meta issue of _how_ to deal with this. At the start of the year I co-wrote NEP 48 (https://numpy.org/neps/nep-0048-spending-project-funds.html) to have a framework for NumPy for spending project funds. I'd like to follow that NEP also for SciPy. So I'll make the proposal here, in accordance with that NEP, and then the other thing is to figure out where to put it up as a PR (or wiki page, or ...). For that my proposal is: create a new repo called something like `project-mgmt`, where we can put such proposals as well as meeting minutes, presentations, and other content. And have a single summary page in Markdown format with a list of all proposals made, linking out to the full proposals.
>> 
>> On to the actual proposal, keeping in mind https://numpy.org/neps/nep-0048-spending-project-funds.html#defining-fundable-activities-and-projects.
>> 
>> Title: Accelerating the move to Meson as the SciPy build system
>> Cost: $12,000
>> Time duration: 3 months
>> Team members: Matthew Brett (funded), Ralf Gommers (unfunded), Gagandeep Singh (unfunded), Smit Lunagariya (funded)
>> 
>> Context:
>> Multiple people recently told me that our current build system and CI issues are a pain point for working on SciPy. We have CI timeouts on our main repo, scipy-wheels is a pain, conda-forge is also timing out on aarch64 and ppc64le build. And I've found myself spending a whole day or weekend several times in the past months to fix broken CI jobs due to build/packaging issues.
>> About 4 months ago, I wrote an RFC to move to Meson as a build system, see https://github.com/scipy/scipy/issues/13615. Since then I have spent a lot of time on this, and also used Quansight Labs funding to get one of my colleagues, Gagandeep Singh, to help me move this forward faster (and he has been super helpful). Getting a Linux + OpenBLAS dev build is close to done (16 of 20 modules complete, all that's left is `misc`, `io`, `integrate` and `signal` - probably those can be completed in the coming week). 
>> Doing everything else, like adding support for Windows/aarch64/MKL/ILP64/etc. is still a lot of work. There's a tracking issue at https://github.com/rgommers/scipy/issues/22. Some of this stuff is pretty specialized, e.g., building wheels, and Fortran-on-Windows. It will take a long time to complete if it's just me spending time on the weekends. 
>> Matthew Brett is available to help soon. He has written multibuild and dealt with build/packaging topics for a long time, so I think he would be able to make progress quickly. He wants to work on this topic as an independent contractor (I won't add rates here, that's for the core team to look at - but they're very reasonable).
>> The overall goals of this effort are:
>> Get a (mostly) feature-complete Meson based build system in place by the end of October, so we can ship it in the 1.8.0 release. If all goes well, get rid of the `numpy.distutils` based build 1 release later.
>> Resolve all our CI timeout issues. If the build system alone isn't enough (it should be though), analyze the remaining issues and figure out how to tackle whatever is still problematic.
>> The deliverables are, in order of priority:
>> Add capability to build release artifacts (wheels, sdist). I did some initial tests at the start with mesonpep517, but I have no idea if it works for SciPy and what bugs are lurking there.
>> macOS support
>> Add support for things we need to a dev.py interface (basically, like runtests.py in that it can do everything, like building docs, running benchmarks, etc.)
>> Contribute a few things we need, or that would make the build setup better, to Meson. For example, I think we'd want to be able to use `py3.install_sources` on generated targets. Of course I'm still new-ish to Meson, so it could well be that after making a proposal on the Meson issue tracker, another solution will be preferred.
>> Windows support
>> Improve build dependency detection and configuration - in particular BLAS/LAPACK flavors.
>> Clean up as many build warnings as possible, and silence the rest. The goal is to have a CI job that is completely silent (passes with `-Werror`).
>> Optimize build time further, including on CI (e.g., reuse build cache between jobs).
>> I'd like to use 75% of the funds to pay Matthew, and 25% of the funds to take on a talented intern, Smit Lunagariya. He is comfortable working in Python, C and C++, and will be able to help with deliverables 7 and 8, as well as with testing and other tasks.
>> 
>> Overall I'm not certain that this amount of funds is enough to complete every last deliverable on this list; it's hard to estimate. But we should certainly get close, and be able to ship a Meson build in 1.8.0.
>> 
>> For a quick sketch of the benefits:
>> On my machine the build time is currently 13 minutes with numpy.distutils and 90 seconds with Meson
>> The build log goes from >10,000 lines (mostly noise) to ~100 lines (mostly configuration info).
>> Why should this be a funded activity? For two reasons: (1) build and CI issues are probably the number one maintenance issue we have, and (2) most people find working on build/CI boring and painful, and hence it doesn't happen with volunteer-only activities. Even regular maintenance of our CI is hard - recently jobs have been broken for 1-2 weeks at a time a couple of times. And the scipy-wheels repo is in even worse shape.
>> 
>> My proposal is to designate this topic as high priority, let the core team approve the compensation levels for the funded people, and get the work started. For reporting on progress and to have some accountability about how we spent the funds, I propose a short report (say 1 page, giving a status update and linking the main PRs made or issues closed) when half the funds are spent, and another such report at the end.
>> 
>> In addition I'll plan to write a small development grant for this when the next call for that opens up, because I think we want Matthew to continue working on this after this $12,000 has been spent. I'd like to get everything as good as it can be, so other projects can easily adopt Meson as well - to also achieve my not-so-secret goal of not having to integrate numpy.distutils capabilities into setuptools once distutils is removed from Python.
>> 
>> Thoughts?
>> 
>> Cheers,
>> Ralf
>> 
>> 
>> 
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210705/cc1ff143/attachment.html>