[SciPy-User] Install Scipy with Anaconda's MKL libraries

David Hagen david at drhagen.com
Thu May 4 08:16:13 EDT 2017 

Am I correct that there is no free version of ifort and icc? Will this
break after the 30-trial ends?

On Thu, May 4, 2017 at 2:58 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> Ok, I followed instructions on this page:
>
> https://software.intel.com/en-us/articles/building-
> numpyscipy-with-intel-mkl-and-intel-fortran-on-windows
>
> Installed parallel_studio_xe_2016_update4 with mkl, ifort, and icc.
>
> 1. I compiled numpy (python setup.py install) from source with this
> site.cfg file:
>
> [mkl]
> include_dirs = C:\Program Files (x86)\IntelSWTools\parallel_
> studio_xe_2016.4.062\compilers_and_libraries_2016\windows\mkl\include
> library_dirs = C:\Program Files (x86)\IntelSWTools\parallel_
> studio_xe_2016.4.062\compilers_and_libraries_2016\
> windows\mkl\lib\intel64;C:\Program Files (x86)\IntelSWTools\parallel_
> studio_xe_2016.4.062\compilers_and_libraries_2016\
> windows\compiler\lib\intel64
> mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll
> lapack_libs = mkl_lapack95_lp64
>
>
> 2. I replaced site.cfg file in ..\Lib\site-packages\numpy\distutils with
> the one above, since this is used by scipy in step 3.
>
> 3. I compiled scipy (python setup.py install) from source.
>
> Compilation went without errors, both packages import without errors in
> python.
>
>
> On Wed, May 3, 2017 at 8:26 PM, David Hagen <david at drhagen.com> wrote:
>
>> So those instructions get me farther than I have gotten before.
>>
>> Is this actually necessary?:
>>
>> Note: You will need to rebuild Python from source files. This is due to
>> the downloaded Python binary on Windows might be incompatible with the
>> Visual Studio version you used. Otherwise you will encounter runtime crash
>> when run numpy or script tests.
>>
>> This is what I have done so far:
>>
>> 1) Download Scipy zip file from Github mater branch
>> 2) Unzip file
>> 3) Add a site.cfg file with these contents:
>> [mkl]
>> library_dirs = C:\Program Files (x86)\IntelSWTools\compilers_a
>> nd_libraries_2017.2.187\windows\mkl\lib\intel64_win;C:\Program Files
>> (x86)\IntelSWTools\compilers_and_libraries_2017.2.187\window
>> s\compiler\lib\intel64_win
>> include_dirs = C:\Program Files (x86)\IntelSWTools\compilers_a
>> nd_libraries_2017.2.187\windows\mkl\include
>> mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_t
>> hread,mkl_core,libiomp5md
>> lapack_libs = mkl_lapack95_lp64,mkl_blas95_l
>> p64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
>>
>> 4) cd into that directory
>> 5) run: python setup.py config --compiler=intelemw --fcompiler=intelvem
>> build_clib --compiler=intelemw --fcompiler=intelvem build_ext
>> --compiler=intelemw --fcompiler=intelvem install
>>
>> It gets really far into the build but this is the error that results:
>>
>> building extension "scipy.spatial.qhull" sources
>> creating build\src.win-amd64-3.6\scipy\spatial
>> Could not locate executable icc
>> Could not locate executable ecc
>> Traceback (most recent call last):
>>   File "setup.py", line 417, in <module>
>>     setup_package()
>>   File "setup.py", line 413, in setup_package
>>     setup(**metadata)
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\core.py
>> ", line 166, in setup
>>     return old_setup(**new_attr)
>>   File "C:\Anaconda3\envs\scipy_master\lib\distutils\core.py", line 148,
>> in setu
>> p
>>     dist.run_commands()
>>   File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 955,
>> in run_
>> commands
>>     self.run_command(cmd)
>>   File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974,
>> in run_
>> command
>>     cmd_obj.run()
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \build_clib.py", line 74, in run
>>     self.run_command('build_src')
>>   File "C:\Anaconda3\envs\scipy_master\lib\distutils\cmd.py", line 313,
>> in run_c
>> ommand
>>     self.distribution.run_command(command)
>>   File "C:\Anaconda3\envs\scipy_master\lib\distutils\dist.py", line 974,
>> in run_
>> command
>>     cmd_obj.run()
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \build_src.py", line 148, in run
>>     self.build_sources()
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \build_src.py", line 165, in build_sources
>>     self.build_extension_sources(ext)
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \build_src.py", line 324, in build_extension_sources
>>     sources = self.generate_sources(sources, ext)
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \build_src.py", line 377, in generate_sources
>>     source = func(extension, build_dir)
>>   File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config
>>     if config_cmd.check_func('open_memstream', decl=True, call=True):
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \config.py", line 312, in check_func
>>     self._check_compiler()
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\command
>> \config.py", line 51, in _check_compiler
>>     self.compiler.initialize()
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\intelcc
>> ompiler.py", line 86, in initialize
>>     MSVCCompiler.initialize(self, plat_name)
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\msvc9co
>> mpiler.py", line 53, in initialize
>>     os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
>>   File "C:\Anaconda3\envs\scipy_master\lib\site-packages\numpy\
>> distutils\msvc9co
>> mpiler.py", line 32, in _merge
>>     if new in old:
>> TypeError: argument of type 'NoneType' is not iterable
>>
>> I notice that this is calling into Numpy. Do I have modify Numpy in order
>> to compile Scipy? I skipped the Numpy specific steps in the instructions
>> because I don't need the development version of Numpy.
>>
>> On Tue, May 2, 2017 at 11:15 PM, Denis Akhiyarov <
>> denis.akhiyarov at gmail.com> wrote:
>>
>>> Did you follow these instructions?
>>>
>>> https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
>>>
>>> On Tue, May 2, 2017, 10:00 PM David Hagen <david at drhagen.com> wrote:
>>>
>>>> Is there a recipe for this combination? I installed MKL from that link
>>>> and the latest Visual Studio. Scipy did not find MKL on its own. I'm sure
>>>> there's some environment variables that need to be set, but I don't know
>>>> what they are.
>>>>
>>>> On Tue, May 2, 2017 at 4:00 AM, William Heymann <immudzen at gmail.com>
>>>> wrote:
>>>>
>>>>> Intel has made MKL, TBB, and a few other things completely free to
>>>>> use, even in a commercial project. Visual Studio is also free unless you
>>>>> are a very large company.
>>>>>
>>>>> https://software.intel.com/en-us/articles/free-mkl
>>>>>
>>>>> I have been using that for other projects without any problems and
>>>>> compiling with Visual Studio has been very easy.
>>>>>
>>>>> On Tue, May 2, 2017 at 1:40 AM, David Hagen <david at drhagen.com> wrote:
>>>>>
>>>>>> I'll try to stick with MinGW-w64 for now, but I don't even get to the
>>>>>> compilation phase. If I install lapack and blas from conda-forge, it still
>>>>>> says that lapack/blas are not found, but you indicated that I need to set
>>>>>> some paths. Are there instructions for this? I have no idea what
>>>>>> environment variables to set in order to tell Scipy to use these packages.
>>>>>>
>>>>>> On Mon, May 1, 2017 at 3:44 AM, Denis Akhiyarov <
>>>>>> denis.akhiyarov at gmail.com> wrote:
>>>>>>
>>>>>>> I still suggest Intel+MSVC compilers, since you can use trial
>>>>>>> version or request license for open-source projects from Intel. This is
>>>>>>> what Anaconda team is using. Also this is what Christoph Gohlke wheels are
>>>>>>> based on:
>>>>>>>
>>>>>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
>>>>>>>
>>>>>>> If you end up with m2w64, here is lapack for conda, you may still
>>>>>>> have to modify paths:
>>>>>>>
>>>>>>> https://anaconda.org/conda-forge/lapack
>>>>>>>
>>>>>>> And blas:
>>>>>>>
>>>>>>> https://anaconda.org/search?q=Blas
>>>>>>>
>>>>>>> On Sun, Apr 30, 2017, 5:22 PM Matthieu Brucher <
>>>>>>> matthieu.brucher at gmail.com> wrote:
>>>>>>>
>>>>>>>> Why do you want to pay Intel? You can install the MKL and develop
>>>>>>>> with it, no sweat.
>>>>>>>>
>>>>>>>> 2017-04-30 22:41 GMT+01:00 David Hagen <david at drhagen.com>:
>>>>>>>>
>>>>>>>>> > Welcome to the world of pain with building scientific packages
>>>>>>>>> from source on Windows!
>>>>>>>>>
>>>>>>>>> I am beginning to feel it.
>>>>>>>>>
>>>>>>>>> > You need Fortran and C/C++ compilers on Windows to build scipy
>>>>>>>>> from source
>>>>>>>>>
>>>>>>>>> I have MinGW-w64 installed, which seems to be the recommended
>>>>>>>>> method.
>>>>>>>>>
>>>>>>>>> > I’m pretty sure that anaconda does not come with the development
>>>>>>>>> files for MKL, only the runtime files.
>>>>>>>>>
>>>>>>>>> I understand now. It looks like MKL is not the way to go unless I
>>>>>>>>> want to pay Intel.
>>>>>>>>>
>>>>>>>>> > If you don't need mkl and lapack/blas is good enough, then
>>>>>>>>> m2w64-toolchain from conda should have all necessary dependencies for
>>>>>>>>> building scipy.
>>>>>>>>>
>>>>>>>>> My only goal is to install and use Scipy master somewhere where it
>>>>>>>>> won't break my stable installation. I thought Anaconda would be a good
>>>>>>>>> place to start because once I activate an Anaconda environment, I should be
>>>>>>>>> able to treat like a normal Python installation and follow the normal
>>>>>>>>> install-from-source instructions. I went ahead and installed that
>>>>>>>>> m2w64-toolchain package, but it still doesn't find any BLAS/LAPACK. Maybe I
>>>>>>>>> should change my question to: how do I install Scipy on Windows from
>>>>>>>>> source? Though when I search for this specifically on the web, the answer
>>>>>>>>> seems to be "Don't.". It seems that MinGW-w64 and ATLAS are recommended by
>>>>>>>>> Scipy. Do you know of a conda/pip package that provides ATLAS for building
>>>>>>>>> Scipy or another more suitable BLAS/LAPACK?
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Information System Engineer, Ph.D.
>>>>>>>> Blog: http://blog.audio-tk.com/
>>>>>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at python.org
>>>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at python.org
>> https://mail.python.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at python.org
> https://mail.python.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-user/attachments/20170504/0634c662/attachment-0001.html>

More information about the SciPy-User mailing list