`import somemodule` does not check all paths in `sys.path`

[stefand986 at gmail.com]

Hi everyone,
I am currently investigating an interesting bug in homeassistant which is a bit hard to reproduce. I already dug so deep that I ask here, because I think the root cause has to do with how Python locates and loads modules/packages.

The "story" is: In homeassistant (short HA) some modules are installed at runtime during first start ("bootstrapping"), and for some reason loading the modules fails directly after installing the modules. Subsequent starts work fine.

The modules that are installed during bootstrapping are installed to `~/.homeassistant/deps/lib/python3.7/site-packages`. Installing the modules happens in a different thread than where the modules are imported. (However as far as I can tell the importing thread waits for the install to finish.)

Now the interesting thing is that when I start HA in verbose mode, I can see that Python is trying not all of the paths that are in `sys.path` (in the following snippet I cut out some stuff for brevity. Link to the full log at the end). As one can see the `~/.homeassistant/deps/lib/python3.7/site-packages` location is not tried and hence the import fails:

  ```
  # trying /home/pyuser/.homeassistant/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  # trying /home/pyuser/.homeassistant/sqlalchemy.abi3.so
  # trying /home/pyuser/.homeassistant/sqlalchemy.so
  # trying /home/pyuser/.homeassistant/sqlalchemy.py
  # trying /home/pyuser/.homeassistant/sqlalchemy.pyc
  # trying /home/pyuser/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  [...]
  # trying /usr/local/lib/python3.7/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  [...]
  # trying /usr/local/lib/python3.7/lib-dynload/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  [...]
  # trying /home/pyuser/.local/lib/python3.7/site-packages/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  [...]
  # trying /usr/local/lib/python3.7/site-packages/sqlalchemy.cpython-37m-x86_64-linux-gnu.so
  [...]
  Exception in thread Recorder:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
      self.run()
    File "/home/pyuser/.local/lib/python3.7/site-packages/homeassistant/components/recorder/__init__.py", line 187, in run
      from .models import States, Events
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 728, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/home/pyuser/.local/lib/python3.7/site-packages/homeassistant/components/recorder/models.py", line 8, in <module>
      from sqlalchemy import (
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'sqlalchemy'
  ```

I also checked that this path is also really in `sys.path` at the time the import happens. This is `sys.path` printed out directly before the `import sqlalchemy statement`:

  ```
  ['/home/pyuser/.homeassistant',
   '/home/pyuser/.homeassistant/deps/lib/python3.7/site-packages',
   '/home/pyuser',
   '/usr/local/lib/python37.zip',
   '/usr/local/lib/python3.7',
   '/usr/local/lib/python3.7/lib-dynload',
   '/home/pyuser/.local/lib/python3.7/site-packages',
   '/usr/local/lib/python3.7/site-packages']
  ```


What I already tried is to do a importlib.invalidate_caches() after installing the modules. However this does not fix it. 

So what is happening here? Why does the import skip `~/.homeassistant/deps/lib/python3.7/site-packages`? Any tips on how I can investigate further?

Link to the full log: https://gist.github.com/StefanD986/fbf08426ceaad657929b973cd5add25a#file-homeassistant_verbose_first_start-txt-L5787