[Numpy-discussion] Compile NumPy with ifort, MSVC and MKL - DLL load failed

[Christoph Gohlke]

Hello,

On 2/15/2021 7:38 AM, Pietro Fontana wrote:
> Hi all,
> 
> I've been trying to compile NumPy from source on Windows 10, with MSVC 
> compiler and Intel MKL. Whenever I link to MKL it fails at loading DLLs.
> I am running Windows 10.0.18363 with Microsoft Visual Studio 2019 
> (16.8.5) and Intel MKL 2017.8.275.
> 
> I managed to reproduce the issue with a minimal setup, using latest 
> Python and NumPy.
> 
>  1. Download latest Python (3.9.1) and latest NumPy (1.20.1) source.
>  2. Open a VS command prompt, unpack Python source, build with
>     PCbuild\build.bat
>  3. Run mklvars.bat intel64 to get the right environment variables set.
>  4. Add the Intel compilers (needed for ifort) to PATH:
> 
> set PATH=C:\Program Files 
> (x86)\IntelSWTools\compilers_and_libraries_2017\windows\bin\intel64;%PATH%
> 
>  5. Create a virtual env, copy a few files from the Python build and
>     activate the virtual env:
> 
> copy Python\PCbuild\amd64\python39.dll venv\Scripts
> copy Python\PC\pyconfig.h venv\Include
> 
>  6. Build NumPy from source and install: pip install . -v
>  7. Try to import NumPy: python -c "import numpy"
> 
> The error message appears as follows:
> 
> |Traceback (most recent |call||last|):|
> 
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py"|, line 
> |22|, |in|<|module|>|
> 
> ||from|. |import|multiarray|
> 
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\multiarray.py"|, 
> line |12|, |in|<|module|>|
> 
> ||from|. |import|overrides|
> 
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\overrides.py"|, 
> line |7|, |in|<|module|>|
> 
> ||from|numpy.core._multiarray_umath |import|(|
> 
> |ImportError: DLL |load||failed||while|importing _multiarray_umath: The specified |module|could |not|be found.|
> 
> ||
> 
> |During handling |of|the above |exception|, another |exception|occurred:|
> 
> ||
> 
> |Traceback (most recent |call||last|):|
> 
> ||File||"<string>"|, line |1|, |in|<|module|>|
> 
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\__init__.py"|, line |145|, |in|<|module|>|
> 
> ||from|. |import|core|
> 
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py"|, line 
> |48|, |in|<|module|>|
> 
> ||raise|ImportError(msg)|
> 
> |ImportError:|
> 
> |[… useful suggestions that however did not lead to a solution…]|
> 
> |Original |error|was: DLL |load||failed||while|importing _multiarray_umath: The specified |module|could |not|be found.|
> 
>   
> 
> The MKL libraries are picked up during compilation since it returns:
> 
> |FOUND:|
> 
> |        libraries = [|'mkl_rt'|]|
> 
> |        library_dirs = [|'C:\\Program Files 
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib\\intel64'|]|
> 
> |        define_macros = [(|'SCIPY_MKL_H'|, |None|), (|'HAVE_CBLAS'|, |None|)]|
> 
> |        include_dirs = [|'C:\\Program Files 
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl'|, |'C:\\Program Files 
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\include'|, |'C:\\Program Files 
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib'|]|
> 
> I tried to analyze the DLL resolution on |_multiarray_umath.pyd| with 
> Dependencies (the newer version of Dependency Walker) but it seems that 
> the MKL DLL loads fine. There are some DLLs that appear as not correctly 
> loaded, but as far as I understand it is caused by the inspection 
> software limit with Windows API sets (api-ms-win-core-*, 
> ext-ms-onecore-*, ext-ms-win-*, and similar), not by actual problems 
> with this DLLs, so I think the system is correctly setup.
> 
> If I skip the initialization of MKL environment variables, then the MKL 
> libraries are not picked and NumPy is compiled to a functional state.
> 
> In the past this setup used to work with Python 3.6, VS2015 and a 
> similar version of Intel MKL.
> I was able to reproduce the issue with NumPy 1.16.2, 1.17 and 1.20.1; 
> with Python 3.8.6 and Python 3.9.1; with Intel MKL 2017 and oneAPI 2020.
> 
> Am I missing any obvious step to succeed in this adventure?
> 
> 

Python >= 3.8 will no longer use PATH for resolving dependencies of 
extension modules. Use os.add_dll_directory(mkl_bin_path) 
<https://docs.python.org/3/library/os.html#os.add_dll_directory> in all 
your scripts before importing numpy or add the call to a 
_distributor_init.py file in the numpy package directory.

Christoph