[SciPy-Dev] SciPy in conda-forge env on Windows

Fri Apr 20 01:17:43 EDT 2018

I'm suspicious about the really complex pattern of success/failure,
including things that seem like they shouldn't matter, and the fact
that the test is called "test_random". It does have a setUp method
that attempts to seed the RNG, but maybe that's not working somehow,
and it's actually a non-deterministic test failure? (To be clear: if
it fails at all then that's probably a bug, but it might be a bug
that's happening in all these configurations but only sometimes
results in a test failure.)

You should file a bug at https://github.com/scipy/scipy/issues

-n

On Thu, Apr 19, 2018 at 5:14 PM, Mark Alexander Mikofski
<mikofski at berkeley.edu> wrote:
> Thanks for your response. I tested it again, and I still hit the same issue.
>
> Here's my procedure:
>
> (base)> conda create -n forge
> (base)> conda activate forge
> (forge)> conda config --env --add channels conda-forge
> (forge)> conda install python=3 pip setuptools wheel
> (forge)> conda install nose pytest
> (forge)> conda install numpy
> (forge)>python
>>>> import numpy
>>>> numpy.test()
>
> Ran 6413 tests in 84.363s
>
> OK (KNOWNFAIL=19, SKIP=16)
> <nose.result.TextTestResult run=6413 errors=0 failures=0>
>
>>>> exit()
>
> (forge)> pip install scipy
> (forge)> python
>>>> import scipy
>>>> scipy.test(verbose=3, tests=['scipy.linalg.tests.test_basic'])
>
> scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%]
> scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
>
> I have to kill the process here
>
> ---
>
> Note, the same thing happens if I install scipy from conda, which installs
> scipy from anaconda channel on win_amd64 b/c scipy-feedstock is currently
> disabled on conda-forge
>
> ---
>
> But, everything passes fine if I just install both numpy and scipy from pip.
>
> And everything passes fine if I install scipy first, which then installs
> numpy from the default "anaconda" channel
>
> ---
>
> I posted an issue on the numpy-feedstock, and a comment on the conda-forge
> google group:
>
> * https://github.com/conda-forge/numpy-feedstock/issues/82
>
> *
> https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/conda-forge/PPLWn4U1ZHA/frHthukWCgAJ
>
> ---
>
> I'm really sorry if this is a red-herring or wild goose chase. I didn't mean
> to cause any trouble. I think following Chris Barker's advice, I should have
> installed scipy first, and then I wouldn't have been in this mess.
>
> Thanks!
>
>
> On Thu, Apr 19, 2018 at 1:23 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Mixing wheels and conda packages can cause some problems over time because
>> of pip and conda getting confused about what's installed. But if you made a
>> conda env, and then did a pip install, and that's all you've done, then that
>> should work fine AFAIK.
>>
>> And the scipy wheels should be self contained and work correctly in any
>> environment, including a conda env where numpy is using mkl or a different
>> version of openblas.
>>
>> I don't know what's going wrong in your case, but I can at least say that
>> as far as I know, it *should* work and you're hitting a genuine bug.
>>
>> On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski
>> <mikofski at berkeley.edu> wrote:
>>>
>>> Hi,
>>>
>>> Sorry if this question is redundant. I checked the "install" page in the
>>> SciPy docs, and couldn't find any mention of conda-forge.
>>>
>>> Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the
>>> numpy-feedstock from conda-forge? When I run the tests for scipy, the fail
>>> and hang after "scipy\linalg\tests\test_basic.py".
>>>
>>> The reason I thought this might work is because they both use a variant
>>> of OpenBLAS, but perhaps they are not compatible? I can see that NumPy
>>> openblas.dll is in miniconda/envs/<envname>/Library/bin (31,053KB) where as
>>> the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and
>>> the name is vendorized with a long alphanumeric string.
>>>
>>> Unfortunately if I try to use conda to install SciPy, conda uses the
>>> default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock
>>> has been disabled, and when I run the tests using this mix of
>>> conda-forge:numpy and anaconda:scipy, I get the same failure at
>>> "scipy\linalg\tests\test_basic.py", which I would expect because the
>>> anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need
>>> to be compatible for scipy to work, right?
>>>
>>> The test that fails is "TestDet.test_random()"
>>>
>>> scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%]
>>>
>>> this is the pytest traceback:
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> AssertionError                            Traceback (most recent call
>>> last)
>>> <ipython-input-3-ff0509141d7f> in <module>()
>>> ----> 1 test_det.test_random()
>>>
>>>
>>> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py
>>> in test_random(self)
>>>     907             d1 = det(a)
>>>     908             d2 = basic_det(a)
>>> --> 909             assert_almost_equal(d1, d2)
>>>     910
>>>     911     def test_random_complex(self):
>>>
>>>
>>> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py
>>> in assert_almost_equal(actual, desired, decimal, err_msg, verbose)
>>>     579         pass
>>>     580     if abs(desired - actual) >= 1.5 * 10.0**(-decimal):
>>> --> 581         raise AssertionError(_build_err_msg())
>>>     582
>>>     583
>>>
>>> AssertionError:
>>> Arrays are not almost equal to 7 decimals
>>>  ACTUAL: 0.001303440758814572
>>>  DESIRED: -0.09307217461347188
>>>
>>> Then the next test hangs for several minute, and I have to kill the
>>> process.
>>>
>>> scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
>>>
>>> It also hangs when I call it directly from an interpreter.
>>>
>>> If I use just pip to install both numpy
>>> (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy
>>> (scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I
>>> can see the vendorized version of openBLAS for numpy in
>>> site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll,
>>> the vendor alpha-numeric string is also the same:
>>> "BNVRK7633HSX7YVO2TADGR4A5KEKXJAW"
>>>
>>> Maybe this is very basic, and I shouldn't be asking. But I don't
>>> understand how to use conda-forge then with SciPy and NumPy on Windows. It
>>> seems like users should only use all pip or only the default "anaconda"
>>> channel. Or if using conda-forge on window, then only use NumPy without
>>> SciPy, but that also means other packages like statsmodels are out too. Also
>>> maybe this is a question for conda-forge and not scipy-dev. Again, so sorry
>>> if my question is out of place.
>>>
>>> I'm also sorry if this is received poorly. I am so happy and very
>>> grateful for all of the volunteer work that has gone into making SciPy and
>>> NumPy work on Windows. I remember the old days when I had to refer people to
>>> cgohlke's website. If there is anything I can do to help, I can try. I will
>>> take a look at reactivating the conda-forge build for win_amd64, I believe
>>> that would solve the problem. Also I will bring up the issue of dependency
>>> clashes with conda-forge as well.
>>>
>>> Thanks,
>>> Mark
>>>
>>> --
>>> Mark Mikofski, PhD (2005)
>>> Fiat Lux
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
>
> --
> Mark Mikofski, PhD (2005)
> Fiat Lux
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>

-- 
Nathaniel J. Smith -- https://vorpus.org