Change in cache tag in Python 3.8 - pip confused

[Paul Moore]

On Tue, 7 May 2019 at 22:26, Chris Angelico <rosuav at gmail.com> wrote:
> So the next question is: Is this actually a problem? If it's something
> that can only ever happen to people who build prerelease Pythons, it's
> probably not an issue. Is there any way that a regular installation of
> Python could ever change its cache_tag? What I experienced would be
> extremely confusing if it ever happened live ("but it IS installed, no
> it ISN'T installed...").

I'm probably not the person to answer, because I know *way* to much
about how pip works to judge what's confusing :-)

However, I'd say that:

1. It's mostly about what core Python/distutils does (compiled module
names, and cache tags, are core language things - see PEP 3147 for the
details)..
2. In a normal release, Python wouldn't break compatibility like this,
so this would be a non-issue in a standard x.y.z Python release.
3. Python doesn't guarantee ABI compatibility across minor releases,
and the site-packages locations would normally be different anyway (I
know they are on Windows, and I assume they would be on Unix) so you
won't have this issue in anything other than X.Y.Z and X.Y.W (where W
!= Z). I'd normally recommend reinstalling everything when changing a
point release too, but I know not everyone does that. ABI
compatibility between point releases is maintained, though, so it
shouldn't be a problem.

Short answer, it's only ever going to be a problem for people building
their own Python from git, and they are probably going to know enough
to debug the issue themselves (even if, like you, they find it hard to
work out what happened...)

> When pip installed python-lzo from source, it must have constructed
> the file name somehow. I've tried digging through pip's sources, but
> searching for "cache_tag" came up blank, and I'm not sure where it
> actually figures out what name to save the .so file under. There's a
> lot of code in pip, though, and I freely admit to having gotten quite
> lost in the weeds :|

The build is done in distutils (or maybe, setuptools) so that's where
you should probably look. Pip just gets a bunch of files from
setuptools, and installs them, at least for up to date projects and
versions of pip - it's the "PEP 517" behaviour if you're into recent
packaging standards. For older pip versions, or projects not updated
to recent standards, pip just says "hey, setuptools, install this
stuff and tell me the names of all the files you created".

Good luck digging into the setuptools/distutils code - take rations,
torches and emergency flares, it's dangerous in there :-)

Paul