[SciPy-Dev] win32 binaries and scipy 0.17.0 release

[Tony Kelman]

Matthew Brett <matthew.brett <at> gmail.com> writes:

> The problem we're trying to solve, is making a toolchain that will
> allow any project to build their own Windows .whl installers without
> having to worry about how other projects were built.

You are making an implicit assumption though that all other projects
were built for compatibility with the python.org builds of CPython,
and that all users will absolutely want to do this. For people who
want to embed Python, e.g. in GDB, Blender, IJulia, etc this isn't
a requirement that adds much value.

> My own feeling is having one central place building all these packages
> is not going to scale, partly because continuous testing is so
> important, and this can most efficiently be done and responded to by
> the projects themselves.

I agree, this is one of my issues with MSYS2. An automatic build service
would be ideal, and we heavily leverage the OpenSUSE build service to
cross-compile Windows binaries of many libraries for use with Julia
packages. (There are other reasons I can get into that we don't use
MSYS2's infrastructure and toolchain more heavily for Julia, but it
would be slightly off-topic.) Right now the only mature solution for
binary distribution of the scientific Python stack on Windows is Anaconda,
which is not only "one central place" but also not reproducible by other
open source projects - as far as we're aware they are using MKL for
NumPy and Intel Fortran for SciPy (and any other Fortran dependencies),
correct? Anaconda.org still does not provide hosted automatic Windows
build VM's on open source plans, do they?

> We scientific Python persons are particularly concerned with packages
> that need compilation.
> 
> That does put us at risk of DLL hell because many packages need to
> link to external libraries, and two packages may need the same
> library.  Or two versions of one package may get installed which have
> different and incompatible versions of the same library.

Which compiler you're using is another dimension of compatibility here,
but I don't see how it changes matters at all with respect to library
versioning and API changes.

> Yes, it is unfortunate, it would be good to find some way of sharing
> effort, and / or lobbying for the Python / pypi folks.

It might be worth pinging a few people who are working on porting the
scientific Python stack to ARM/Android, as cross compilation is also an
important tool there. Distutils and many other Python tools like Cython
and conda have some pretty deeply ingrained assumptions that the build
system equals the runtime system (and Windows equals MSVC). MSYS[2] is
fundamentally a path-mangling hack that allows you to use autotools,
bash, and gmake within a posix system while pretending that you aren't
cross-compiling to a non-posix MinGW target. If your build system knows
how to cross compile properly, it can be cleaner to just do that from
either Cygwin or Linux (and in the latter case, build times are often
substantially faster).