[Numpy-discussion] ANN: NumPy 1.8.1 release

Fri Apr 4 11:19:05 EDT 2014

I'ts time for me to come back to the discussion after a longer break.

some personal history: I was looking for a 64bit mingw more than a year ago
(unrelated to python) for Fortran development and tried out quite some
mingw toolchain variants based on the mingw-w64 project. In a nutshell: the
most appropriate and best documentated solution are the toolchains provided
by the mingw builds project IMHO.

The next step was the idea to use this toolchain for compiling python
extensions (C and Fortran) and then to try out compiling numpy and scipy
with OpenBLAS.

Despite the fact, that a mingw-w64 based toolchain is rock solid today the
following possible issues should be considered for Python development:

(1) deploy problem: mingw runtime DLLs can not be found at runtime
Solution: use flags for static linking or use a dedicated 'static' GCC
toolchain for compiling and linking. Both solutions should work.

(2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since
GCC4.6, MSVC uses 4 bytes
Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64
X86_64 is not affected. This issue is the major cause for segment faults on
32bit systems.

(3) Import library problem: numpy distutils does not play well with
mingw-w64
Solution: create a Python import library with the mingw-w64 tools. Use a
patched numpy distutils. A detailed description can be found here:
http://article.gmane.org/gmane.comp.python.numeric.general/56749 .

(4) linking against the correct msvcrXXX Version.
Solution: create a 'specs' file (howto see
http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes
necessary informations.

(5) manifest resources
Solution: extend the GCC toolchain with the Manifest resource files and
ensure linkage with the help of the 'specs' file.

(6) Blas Lapack for numpy scipy
There is no silver bullet! A trade-off between licence acceptance,
performance and stability remains to be found. OpenBLAS on Win32 seems to
be quite stable. Some OpenBLAS issues on Win64 can be adressed with a
single threaded version of that library.

On my google drive:
https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharingI
provide the necessary parts to try the procedures described at
http://article.gmane.org/gmane.comp.python.numeric.general/56749 and
http://article.gmane.org/gmane.comp.python.numeric.general/56767

Up to now I didn't find time to put a comprehensive description on the Web
and to update all that stuff (MSVCR100 support for the toolchain still
missing), so I add my incomplete, not yet published mingw-w64 FAQ at the
end of my longish E-Mail for further discussions.

Carl

---

my personal mingw-w64 FAQ
=========================

what is mingw-w64
-----------------

mingw-w64 is a fork of the mingw32 project
- http://sourceforge.net/apps/trac/mingw-w64/wiki/History

why choose mingw-w64 over mingw
-------------------------------

- 32 AND 64bit support
- large file support
- winpthread pthreads implementation, MIT licenced.
- cross compiler toolchains availabe for Linux

official mingw-w64 releases
---------------------------

source releases of the mingw-64 repository
-
http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/

official mingw-w64 GCC toolchains
---------------------------------

'recommened' builds are available from the mingw-builds project
http://mingw-w64.sourceforge.net/download.php#mingw-builds for example
-
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/
-
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/

These are common combinations of exception and thread models. You can find
other combinations as well. Exception handling affects C++ development.
Don't ever link object code with different types of exception and/or thread
handling!

threads concerning the question 'where to find mingw-w64 builds'
- http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700
- http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484

how to build a mingw-w64 based GCC toolchain on Windows
-------------------------------------------------------

"mingw-builds" is a set of scripts and patches for compiling the GCC
toolchain under Windows with the help of msys2 POSIX enviroment
- https://github.com/niXman/mingw-builds/tree/develop
recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64
sf.net (4)

what is msys2
-------------

msys2 is the successor of msys. Msys2 is necessary as enviroment for the
mingw build process on Windows.
- http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/

where to get precompiled mingw-w64 compiled libraries
-----------------------------------------------------

recent mingw-w64 based tools and library packages together with sources and
patches are available from archlinux as well as from the msys2 maintainers.
- http://sourceforge.net/projects/mingw-w64-archlinux/files/  (i686: Sjlj |
x86_64: SEH)
- http://sourceforge.net/projects/msys2/files/REPOS/MINGW/  (i686: Dwarf |
x86_64: SEH)

what is a static GCC toolchain
------------------------------

GCC as well as the all neccessary libraries for a toolchain can be compiled
with "-disabled-shared". This is supported by the mingw-builds scripts as
an option. All the necessary object code from the GCC runtimes will be
statically linked into the binaries. As a consequence the binary size will
be increased in comparison to the standard toolchains. The advantage is,
that there will be no dependancy to external GCC runtime libraries, so the
deployment of python extensions is greatly improved. Using such a toolchain
is more reliable than using -static-XXX flags.
However, exception heavy C++ programms (i.e. QT) should be compiled with
shared runtimes to avoid  problems with exceptions handling over DLL
boundaries.
For building typically Python extensions a customized static GCC toolchain
is the best compromise IMHO.

customizations over standard mingw-builds releases
--------------------------------------------------

- two dedicated GCC toolchains for both 32bit (posix threads, Dwarf
exceptions) and 64 bit (posix threads, SEH exceptions)
- statically  build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0
- languages: C, C++, gfortran, LTO
- customized 'specs' file for MSVCR90 linkage and manifest support
(MSVCR100 linkage coming soon)
- additional ftime64 patch to allow winpthreads and OpenMP to work with
MSVCR90 linkage
- openblas-2.9rc1 with windows thread support (OpenMP disabled) included

2014-04-02 1:46 GMT+02:00 David Cournapeau <cournape at gmail.com>:

>
>
>
> On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau <cournape at gmail.com>
>> wrote:
>> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> >>
>> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett <matthew.brett at gmail.com
>> >
>> >> wrote:
>> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL
>> >> > loaded via:
>> >> >
>> >> > hDLL = LoadLibraryEx(pathname, NULL,  LOAD_WITH_ALTERED_SEARCH_PATH);
>> >> >
>> >> > will in turn (by default) search for its dependent DLLs in their own
>> >> > directory.    Or maybe in the directory of the first DLL to be loaded
>> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the
>> >> > documentation.  Looking forward to doing my tax return after this.
>> >> >
>> >> > But - anyway - that means that any extensions in the DLLs directory
>> >> > will get their dependencies from the DLLs directory, but that is only
>> >> > true for extensions in that directory.
>> >>
>> >> So in conclusion, if we just drop our compiled dependencies next to
>> >> the compiled module files then we're good, even on older Windows
>> >> versions? That sounds much simpler than previous discussions, but good
>> >> news if it's true...
>> >
>> >
>> > That does not work very well in my experience:
>> >
>> >   - numpy has extension modules in multiple directories, so we would
>> need to
>> > copy the dlls in multiple subdirectories
>> >   - copying dlls means that windows will load that dll multiple times,
>> with
>> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen
>> > serious issues when doing something similar for hdf5 dll and
>> pytables/h5py).
>>
>> We could just ship all numpy's extension modules in the same directory
>> if we wanted. It would be pretty easy to stick some code at the top of
>> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them
>> into the appropriate places in the package namespace.
>>
>> Of course scipy and numpy will still both have to ship BLAS etc., and
>> so I guess it will get loaded at least twice in *any* binary install
>> system. I'm not sure why this would be a problem (Windows, unlike
>> Unix, carefully separates DLL namespaces, right?)
>
>
> It does not really matter here. For pure blas/lapack, that may be ok
> because the functions are "stateless", but I would not count on it either.
>
> The cleanest solution I can think of is to have 'privately shared DLL',
> but that would AFAIK require patching python, so not really an option.
>
>
> , but if it is a
>> problem then it's a very fundamental one for any binaries we ship.
>>
>> Do the binaries we ship now have this problem? Or are we currently
>> managing to statically link everything?
>>
>
> We currently statically link everything. The main challenge is that 'new'
> (>= 4) versions of mingw don't easily allow statically linking all the
> mingw-related dependencies. While the options are there, everytime I tried
> to do it with an official build of mingw, I had some weird, very hard to
> track crashes. The other alternative that has been suggested is to build
> one own's toolchain where everything is static by default. I am not sure
> why that works, and that brings the risk of depending on a toolchain that
> we can't really maintain.
>
> David
>
>>
>> -n
>>
>> --
>> Nathaniel J. Smith
>> Postdoctoral researcher - Informatics - University of Edinburgh
>> http://vorpus.org
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140404/9255f0c8/attachment.html>