[Python-Dev] PEP-582 and multiple Python installations

[Steve Dower]

On 02Apr2019 0817, Calvin Spealman wrote:
> (I originally posted this to python-ideas, where I was told none of this 
> PEP's authors subscribe so probably no one will see it there, so I'm 
> posting it here to raise the issue where it can get seen and hopefully 
> discussed)

Correct, thanks for posting. (I thought we had a "discussions-to" tag 
with distutils-sig on it, but apparently not.)

> While the PEP does show the version number as part of the path to the 
> actual packages, implying support for multiple versions, this doesn't 
> seem to be spelled out in the actual text. Presumably 
> __pypackages__/3.8/ might sit beside __pypackages__/3.9/, etc. to keep 
> future versions capable of installing packages for each version, the way 
> virtualenv today is bound to one version of Python.
> 
> I'd like to raise a potential edge case that might be a problem, and 
> likely an increasingly common one: users with multiple installations of 
> the *same* version of Python. This is actually a common setup for 
> Windows users who use WSL, Microsoft's Linux-on-Windows solution, as you 
> could have both the Windows and Linux builds of a given Python version 
> installed on the same machine. The currently implied support for 
> multiple versions would not be able to separate these and could create 
> problems if users pip install a Windows binary package through 
> Powershell and then try to run a script in Bash from the same directory, 
> causing the Linux version of Python to try to use Windows python packages.
> 
> I'm not actually sure what the solution here is. Mostly I wanted to 
> raise the concern, because I'm very keen on WSL being a great entry path 
> for new developers and I want to make that a better experience, not a 
> more confusing one. Maybe that version number could include some other 
> unique identify, maybe based on Python's own executable. A hash maybe? I 
> don't know if anything like that already exists to uniquely identify a 
> Python build or installation.

Yes, this is a situation we're aware of, and it's caught in the conflict 
of "who is this feature meant to support".

Since all platforms have a unique extension module suffix (e.g. 
"module.cp38-win32.pyd"), it would be possible to support this with 
"fat" packages that include all binaries (or some clever way of merging 
wheels for multiple platforms).

And since this is already in CPython itself, it leads to about the only 
reasonable solution - instead of "3.8", use the extension module suffix 
"cp38-win32". (Wheel tags are not in core CPython, so we can't use those.)

But while this seems obvious, it also reintroduces problems that this 
has the potential to fix - suddenly, just like installing into your 
global environment, your packages are not project-specific anymore but 
are Python-specific. Which is one of the major confusions people run 
into ("I pip installed X but now can't import it in python").

So the main points of discussion right now are "whose problem does this 
solve" and "when do we tell people they need a full venv". And that 
discussion is mostly happening at 
https://discuss.python.org/t/pep-582-python-local-packages-directory/963/

Cheers,
Steve