[Numpy-discussion] ANN: NumPy 1.8.1 release

[Matthew Brett]

Hi,

On Fri, Apr 4, 2014 at 8:19 AM, Carl Kleffner <cmkleffner at gmail.com> wrote:
>
> I'ts time for me to come back to the discussion after a longer break.
>
> some personal history: I was looking for a 64bit mingw more than a year ago
> (unrelated to python) for Fortran development and tried out quite some mingw
> toolchain variants based on the mingw-w64 project. In a nutshell: the most
> appropriate and best documentated solution are the toolchains provided by
> the mingw builds project IMHO.
>
> The next step was the idea to use this toolchain for compiling python
> extensions (C and Fortran) and then to try out compiling numpy and scipy
> with OpenBLAS.
>
> Despite the fact, that a mingw-w64 based toolchain is rock solid today the
> following possible issues should be considered for Python development:
>
> (1) deploy problem: mingw runtime DLLs can not be found at runtime
> Solution: use flags for static linking or use a dedicated 'static' GCC
> toolchain for compiling and linking. Both solutions should work.
>
> (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since
> GCC4.6, MSVC uses 4 bytes
> Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64
> X86_64 is not affected. This issue is the major cause for segment faults on
> 32bit systems.
>
> (3) Import library problem: numpy distutils does not play well with
> mingw-w64
> Solution: create a Python import library with the mingw-w64 tools. Use a
> patched numpy distutils. A detailed description can be found here:
> http://article.gmane.org/gmane.comp.python.numeric.general/56749 .
>
> (4) linking against the correct msvcrXXX Version.
> Solution: create a 'specs' file (howto see
> http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes
> necessary informations.
>
> (5) manifest resources
> Solution: extend the GCC toolchain with the Manifest resource files and
> ensure linkage with the help of the 'specs' file.
>
> (6) Blas Lapack for numpy scipy
> There is no silver bullet! A trade-off between licence acceptance,
> performance and stability remains to be found. OpenBLAS on Win32 seems to be
> quite stable. Some OpenBLAS issues on Win64 can be adressed with a single
> threaded version of that library.
>
> On my google drive:
> https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharing
> I provide the necessary parts to try the procedures described at
> http://article.gmane.org/gmane.comp.python.numeric.general/56749 and
> http://article.gmane.org/gmane.comp.python.numeric.general/56767
>
> Up to now I didn't find time to put a comprehensive description on the Web
> and to update all that stuff (MSVCR100 support for the toolchain still
> missing), so I add my incomplete, not yet published mingw-w64 FAQ at the end
> of my longish E-Mail for further discussions.
>
> Carl
>
> ---
>
> my personal mingw-w64 FAQ
> =========================
>
> what is mingw-w64
> -----------------
>
> mingw-w64 is a fork of the mingw32 project
> - http://sourceforge.net/apps/trac/mingw-w64/wiki/History
>
> why choose mingw-w64 over mingw
> -------------------------------
>
> - 32 AND 64bit support
> - large file support
> - winpthread pthreads implementation, MIT licenced.
> - cross compiler toolchains availabe for Linux
>
> official mingw-w64 releases
> ---------------------------
>
> source releases of the mingw-64 repository
> -
> http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/
>
> official mingw-w64 GCC toolchains
> ---------------------------------
>
> 'recommened' builds are available from the mingw-builds project
> http://mingw-w64.sourceforge.net/download.php#mingw-builds for example
> -
> http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/
> -
> http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/
>
> These are common combinations of exception and thread models. You can find
> other combinations as well. Exception handling affects C++ development.
> Don't ever link object code with different types of exception and/or thread
> handling!
>
> threads concerning the question 'where to find mingw-w64 builds'
> - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700
> - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484
>
> how to build a mingw-w64 based GCC toolchain on Windows
> -------------------------------------------------------
>
> "mingw-builds" is a set of scripts and patches for compiling the GCC
> toolchain under Windows with the help of msys2 POSIX enviroment
> - https://github.com/niXman/mingw-builds/tree/develop
> recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64
> sf.net (4)
>
> what is msys2
> -------------
>
> msys2 is the successor of msys. Msys2 is necessary as enviroment for the
> mingw build process on Windows.
> - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
>
> where to get precompiled mingw-w64 compiled libraries
> -----------------------------------------------------
>
> recent mingw-w64 based tools and library packages together with sources and
> patches are available from archlinux as well as from the msys2 maintainers.
> - http://sourceforge.net/projects/mingw-w64-archlinux/files/  (i686: Sjlj |
> x86_64: SEH)
> - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/  (i686: Dwarf |
> x86_64: SEH)
>
> what is a static GCC toolchain
> ------------------------------
>
> GCC as well as the all neccessary libraries for a toolchain can be compiled
> with "-disabled-shared". This is supported by the mingw-builds scripts as an
> option. All the necessary object code from the GCC runtimes will be
> statically linked into the binaries. As a consequence the binary size will
> be increased in comparison to the standard toolchains. The advantage is,
> that there will be no dependancy to external GCC runtime libraries, so the
> deployment of python extensions is greatly improved. Using such a toolchain
> is more reliable than using -static-XXX flags.
> However, exception heavy C++ programms (i.e. QT) should be compiled with
> shared runtimes to avoid  problems with exceptions handling over DLL
> boundaries.
> For building typically Python extensions a customized static GCC toolchain
> is the best compromise IMHO.
>
> customizations over standard mingw-builds releases
> --------------------------------------------------
>
> - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf
> exceptions) and 64 bit (posix threads, SEH exceptions)
> - statically  build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0
> - languages: C, C++, gfortran, LTO
> - customized 'specs' file for MSVCR90 linkage and manifest support (MSVCR100
> linkage coming soon)
> - additional ftime64 patch to allow winpthreads and OpenMP to work with
> MSVCR90 linkage
> - openblas-2.9rc1 with windows thread support (OpenMP disabled) included

Thanks very much for this.

Would you consider putting this up as a numpy wiki page?

https://github.com/numpy/numpy/wiki

I think it would be very valuable...

Cheers,

Matthew