[scikit-learn] scikit-learn 0.24 installation fails with ModuleNotFoundError: No module named 'scipy'

[Bertrand B.]

To whom it may concern,

I am trying to install scikit-learn in a PySpark job using the
install_pypi_package PySpark API but the install fails with :

sc.install_pypi_package("scikit-learn")

Collecting scikit-learn
  Using cached https://files.pythonhosted.org/packages/db/e2/9c0bde5f81394b627f623557690536b12017b84988a4a1f98ec826edab9e/scikit-learn-0.24.0.tar.gz
Requirement already satisfied: numpy>=1.13.3 in
/usr/local/lib64/python3.7/site-packages (from scikit-learn)
Collecting scipy>=0.19.1 (from scikit-learn)
  Using cached https://files.pythonhosted.org/packages/58/9d/8296d8211318d690119eba6d293b7a149c1c51c945342dd4c3816f79e1ba/scipy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: joblib>=0.11 in
/usr/local/lib64/python3.7/site-packages (from scikit-learn)
Collecting threadpoolctl>=2.0.0 (from scikit-learn)
  Using cached https://files.pythonhosted.org/packages/f7/12/ec3f2e203afa394a149911729357aa48affc59c20e2c1c8297a60f33f133/threadpoolctl-2.1.0-py3-none-any.whl
Building wheels for collected packages: scikit-learn
  Running setup.py bdist_wheel for scikit-learn: started
  Running setup.py bdist_wheel for scikit-learn: finished with status 'error'
  Complete output from command /tmp/1611000009300-0/bin/python -u -c
"import setuptools,
tokenize;__file__='/mnt/tmp/pip-build-phc6p6gl/scikit-learn/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d
/tmp/tmpry3gf9r0pip-wheel- --python-tag cp37:
  Partial import of sklearn during the build process.
  Traceback (most recent call last):
    File "/mnt/tmp/pip-build-phc6p6gl/scikit-learn/setup.py", line
201, in check_package_status
      module = importlib.import_module(package)
    File "/tmp/1611000009300-0/lib64/python3.7/importlib/__init__.py",
line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'scipy'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/mnt/tmp/pip-build-phc6p6gl/scikit-learn/setup.py", line
306, in <module>
      setup_package()
    File "/mnt/tmp/pip-build-phc6p6gl/scikit-learn/setup.py", line
294, in setup_package
      check_package_status('scipy', min_deps.SCIPY_MIN_VERSION)
    File "/mnt/tmp/pip-build-phc6p6gl/scikit-learn/setup.py", line
227, in check_package_status
      .format(package, req_str, instructions))
  ImportError: scipy is not installed.
  scikit-learn requires scipy >= 0.19.1.

I do not encounter this error with scikit-learn 0.23.2 :

sc.install_pypi_package("scikit-learn==0.23.2")

Collecting scikit-learn==0.23.2
  Using cached https://files.pythonhosted.org/packages/f4/cb/64623369f348e9bfb29ff898a57ac7c91ed4921f228e9726546614d63ccb/scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: scipy>=0.19.1 in
/mnt/tmp/1611000009300-0/lib/python3.7/site-packages (from
scikit-learn==0.23.2)
Requirement already satisfied: numpy>=1.13.3 in
/usr/local/lib64/python3.7/site-packages (from scikit-learn==0.23.2)
Requirement already satisfied: joblib>=0.11 in
/usr/local/lib64/python3.7/site-packages (from scikit-learn==0.23.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in
/mnt/tmp/1611000009300-0/lib/python3.7/site-packages (from
scikit-learn==0.23.2)
Installing collected packages: scikit-learn
Successfully installed scikit-learn-0.23.2


Could you please help me understand why the scikit-learn 0.24 installation
fails ?

Thank you for your help,

Bertrand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scikit-learn/attachments/20210120/c901d266/attachment-0001.html>