[SciPy-Dev] win32 binaries and scipy 0.17.0 release

[Matthew Brett]

On Sat, Jan 2, 2016 at 8:48 PM, Tony Kelman <tony at kelman.net> wrote:
> Matthew Brett <matthew.brett <at> gmail.com> writes:
>
>> The problem we're trying to solve, is making a toolchain that will
>> allow any project to build their own Windows .whl installers without
>> having to worry about how other projects were built.
>
> You are making an implicit assumption though that all other projects
> were built for compatibility with the python.org builds of CPython,
> and that all users will absolutely want to do this. For people who
> want to embed Python, e.g. in GDB, Blender, IJulia, etc this isn't
> a requirement that adds much value.

No, you misunderstand me, it's possible to imagine this system built
on any easily available Python, I was only saying that it's important
that it should be obvious and easy how to build each individual
project without having to worry about - for example - the DLL
configuration of other projects.  If you have one central place that
decides what to do - like Anaconda or the MSYS2 packages - then that
place can take care that the DLLs do not tread on each others toes -
but once you have distributed package building it becomes a lot harder
to do that.

>> My own feeling is having one central place building all these packages
>> is not going to scale, partly because continuous testing is so
>> important, and this can most efficiently be done and responded to by
>> the projects themselves.
>
> I agree, this is one of my issues with MSYS2. An automatic build service
> would be ideal, and we heavily leverage the OpenSUSE build service to
> cross-compile Windows binaries of many libraries for use with Julia
> packages. (There are other reasons I can get into that we don't use
> MSYS2's infrastructure and toolchain more heavily for Julia, but it
> would be slightly off-topic.) Right now the only mature solution for
> binary distribution of the scientific Python stack on Windows is Anaconda,
> which is not only "one central place" but also not reproducible by other
> open source projects - as far as we're aware they are using MKL for
> NumPy and Intel Fortran for SciPy (and any other Fortran dependencies),
> correct? Anaconda.org still does not provide hosted automatic Windows
> build VM's on open source plans, do they?

I don't know, maybe someone else can comment?  But in any case, there
are other reasons, including the ones you give, to build up an
alternative solution to packaging using standard tools like pip and
wheels.

>> We scientific Python persons are particularly concerned with packages
>> that need compilation.
>>
>> That does put us at risk of DLL hell because many packages need to
>> link to external libraries, and two packages may need the same
>> library.  Or two versions of one package may get installed which have
>> different and incompatible versions of the same library.
>
> Which compiler you're using is another dimension of compatibility here,
> but I don't see how it changes matters at all with respect to library
> versioning and API changes.

I'm sorry if I gave that impression, I was only trying to say that
these problems get worse when switching from a centralized build to a
distributed one.

>> Yes, it is unfortunate, it would be good to find some way of sharing
>> effort, and / or lobbying for the Python / pypi folks.
>
> It might be worth pinging a few people who are working on porting the
> scientific Python stack to ARM/Android, as cross compilation is also an
> important tool there. Distutils and many other Python tools like Cython
> and conda have some pretty deeply ingrained assumptions that the build
> system equals the runtime system (and Windows equals MSVC). MSYS[2] is
> fundamentally a path-mangling hack that allows you to use autotools,
> bash, and gmake within a posix system while pretending that you aren't
> cross-compiling to a non-posix MinGW target. If your build system knows
> how to cross compile properly, it can be cleaner to just do that from
> either Cygwin or Linux (and in the latter case, build times are often
> substantially faster).

We could surely benefit from cross-compilation - is there a good
central place to get going with mingw-w64 cross compiling?

Cheers,

Matthew