[Distutils] pip upgrade woes

[Jason R. Coombs]

Setuptools advertises in its description, "Easily download, build, install, upgrade, and uninstall Python packages". It was intended to support the uninstall model, though at the time that was written, "easy" meant easier than finding and removing each of the files by hand.

If you're using eggs, the default install model for Setuptools, they can be uninstalled by removing any reference to it from easy-install.pth (which causes it not to be added to sys.path) or by removing the .egg from the site dir (in which case Setuptools will remove the easy-install.pth reference).

It's not a clean uninstall in all cases, because any scripts will still linger.

Nevertheless, it's still a model that works reasonably well.

Setuptools can't help with distutils-installed packages. Note that Setuptools will always install itself as an egg unless installed by another manager such as pip or apt.

-----Original Message-----
From: pypa-dev at googlegroups.com [mailto:pypa-dev at googlegroups.com] On Behalf Of Paul Moore
Sent: Tuesday, 24 March, 2015 14:29
To: Ionel Cristian Mărieș
Cc: pypa-dev; DistUtils mailing list
Subject: Re: pip upgrade woes

On 24 March 2015 at 18:10, Ionel Cristian Mărieș <contact at ionelmc.ro> wrote:
> There's one issue with pip upgrade that annoys me occasionally, and 
> when it does it's very annoying.
>
> Every so often me or some customer has to upgrade some core packages 
> like pip, setuptools or virtualenv on some machine. Now this becomes 
> very annoying because said packages were installed there with either 
> easy_install or just `setup.py install`. Several upgrades like that 
> and now the machine has a hadful of eggs there. Lots of mistakes were 
> made but what's done is done.
>
> Now, if `pip upgrade pip setuptools virtualenv` it will run around a 
> bit, flap its wings and in the end it's not gonna fly like an eagle, 
> and won't be able to go beyond it's cursed fences. An so, I feel like 
> chicken farmer when I try to upgrade packages and pip can't upgrade 
> them. Cause those old eggs are still going to be first on sys.path. 
> And when I try to run pip it's still that old one.
>
> Sometimes few `pip uninstall` solve the issue, but most of the time I 
> have to manually remove files because pip can't figure out what files to remove.
>
> One big issue is that pip uninstall only uninstalls the first package 
> it finds, and similarly, pip install will only uninstall the first 
> package it finds before coping the new files.
>
> This whole process becomes a whole lot more annoying when you have to 
> explain someone how to cleanup this mess and get latest pip and setuptools.
>
> So I'm wondering if there's a better way to cleanup machines like 
> that. Any ideas?

If I understand your problem correctly, the issue is that these machines have older installs of packages, added by tools that don't have uninstall capabilities, using formats that are not used by pip.
You're not able to get pip uninstall to work either because the uninstall data isn't in a form pip can handle, or because there is no uninstall data. I suspect the "first on sys.path" issue is caused by setuptools' .pth files, which are even messier to tidy up, from what I recall of my infrequent dealings with them.

It should be possible to write some sort of "purge" command that searches out and removes all traces of a package like that. I wouldn't want pip to do anything like that automatically, for obvious reasons.
I'm not even that keen on it being a new pip subcommand, because there's a lot of risk of breakage (I'd imagine the code would need a certain amount of guesswork to identify non-standard files and directories that "belong" to a package).

As a start, I'd suggest looking at writing some sort of independent purge-package command that you could use when you hit problems (pip install -U setuptools... weirdness happens, so purge-package setuptools; pip install setuptools). If all the scenarios that tool handles end up being clearly defined and low-risk, then it might be that there's scope for a "pip purge" command of some sort based on it, that removes all trace of a package even when the standard uninstall metadata isn't available.

Maybe someone else here with more understanding of the history of easy_install and how eggs used to work (maybe still do, for all I
know...) can offer something more specific. Sorry that I can't.
Paul