[Distutils] Including soabi tag in --user path?

[Nathaniel Smith]

On Fri, Jul 14, 2017 at 9:48 PM, Ned Deily <nad at python.org> wrote:
> On Jul 15, 2017, at 00:41, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily <nad at python.org> wrote:
>>> On Jul 15, 2017, at 00:22, Nathaniel Smith <njs at pobox.com> wrote:
>>>> A user on the pywavelets mailing list seems to have gotten themselves
>>>> into a mess: AFAICT they did 'pip install --user pywavelets' using a
>>>> ucs4 python, and then started a ucs2 python and tried to import it,
>>>> which of course failed because these two python builds have
>>>> incompatible ABIs, even though they share the same --user directory.
>>>>
>>>> In retrospect, this seems like an obvious oversight: right now the
>>>> --user path (on Linux) is something like
>>>>
>>>> ~/.local/lib/python3.5/site-packages
>>>>
>>>> Wouldn't it make more sense for it to be something like
>>>>
>>>> ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
>>>>
>>>> so that Pythons with incompatible ABIs can't see each other's
>>>> packages? (That's sysconfig.get_config_var("SOABI").)
>>>>
>>>> Obviously this can't really be fixed for old versions, but is this a
>>>> change that should be made in 3.7?
>>>
>>> I'm confused.  This shouldn't be an issue for Python 3.*.  First, there is no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string representation.
>>
>> Right, but there are still SOABI variations, e.g. debug builds versus
>> regular builds.
>
> And those are covered in the file names, e.g. a debug build extension module would be:
>
> _psutil_osx.cpython-35dm-darwin.so
>
>> But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension modules have unique names based on ABI, for example:
>>>
>>> _psutil_osx.cpython-35m-darwin.so
>>
>> Right, but this only helps if you install multiple builds of the same
>> package on top of each other. Which Debian is able to arrange, so PEP
>> 3149 solves their problem. But the --user dir is managed by pip, and
>> if you try to install a cpython-35dm package on top of a cpython-35m
>> package, then pip will helpfully remove the first before installing
>> the second. (And this is really the only reasonable thing for pip to
>> do.) So in practice any given package in the --user dir can only
>> support one SOABI variant, even though all the SOABI variants use the
>> same dir.
>
> Well, that sounds like either a pip problem or a packager's problem.  The point of PEP 3149 was exactly to allow multiple builds of extension modules built with different ABIs to co-exist in the same directory.  You would have the same behavior with packages installed to the system site-packergs dir, right?  So what's special about --user?

For system-managed packages, we tend to get away with it because
Debian [1] is able to enforce that there's one .deb file that ships
all of the pure python files and all of the SOABI variants for
extension modules together, so everything remains consistent.

For pip managed packages this doesn't really make sense. You have to
somehow ensure that two different pip runs, using different versions
of python, are both allowed to install two copies of the same package
on top of each other into the same directory, and that the only
difference between them is the extension modules. And what do you do
if one SOABI Python install package X version Y, but then a different
SOABI Python tries to upgrade to package X version (Y+1)?

Another possible option would be to have both an SOABI-tagged
directory and a non-tagged directory on sys.path, and install packages
that contain extensions into one and pure-python packages into the
other. Allegedly this is what the purelib/platlib distinction in wheel
metadata is supposed to enable, I think, but in practice there are
definitely wheels out there that break this (e.g. tensorflow is
allegedly a purelib wheel), so I don't know if it works in practice.

And yeah, it might well make sense to do this for system site packages
too, but then there might be additional complications with Debian etc.
so I figured I'd start with the obvious case :-).

-n

[1] or whoever is distributing system packages, but PEP 3149 came from Debian.

-- 
Nathaniel J. Smith -- https://vorpus.org