[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Nick Coghlan]

On 22 October 2017 at 04:03, xoviat <xoviat at gmail.com> wrote:

> Nick:
>
> That's generally a good idea, but one significant problem that can occur
> is that the Python import system will cache certain libraries, people will
> run "pip install," and then they will expect such libraries to be
> available. I don't even know exactly how the caching for the import system
> works, so I don't want to go and make claims about it that may be incorrect
> (maybe you do?). However, it is important to keep that in mind when
> considering an API.
>

Yep, since we switched to the implementation that uses fewer stat calls,
you need to call `importlib.invalidate_caches()` to be sure your current
process will see anything you just installed. However, for modules you
haven't previously imported, that's independent of how you installed them -
the problem is that the granularity on the import system's automated cache
invalidation depends on the granularity of the filesystem's mtime records
for directories, so you either have to sleep for a few seconds (since the
mtime resolution on filesystems like FAT & FAT32 is only a couple of
seconds), or else force the cache invalidation.

The problem with the sys.modules cache and already imported libraries is
different: for those, you either need to use
`importlib.reload(existing_module)` to force an in-place update of the
existing namespace, or else `del sys.modules[existing_module.__name__]` to
force creation of a new module without affecting old references to it.

It's those subtleties that keep "Restart all Python processes if you expect
them to see components you just installed" in place as our default
guidance: it's the only option that's guaranteed to work in all cases.
While hot reloading can be made to work, it has assorted caveats and
restrictions that you need to learn in order to do it correctly and
reliably (many of them are actually pretty similar to the self-imposed
restrictions needed to make lazy loading work reliably).

However, none of that impacts the question of whether `pip.main()` runs
code in the current process or implicitly runs it in a subprocess - `pip`
doesn't import the modules it installs either way, so it will all look the
same as far as the import system is concerned.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171022/a1df9572/attachment-0001.html>