[Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))

[Steve Dower]

From: Paul Moore
> On 13 July 2013 10:05, Paul Moore <p.f.moore at gmail.com> wrote:
> How robust is the process of upgrading pip using itself? Specifically on
> Windows, where these things typically seem less reliable.
> 
> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason
> for the failure is simple enough to explain - the pip.exe wrapper is held open
> by the OS while it's in use, so that the upgrade cannot replace it.
> 
> The result is a failed upgrade and a partially installed new version of pip. In
> practice, the exe stubs are probably added fairly late in the install (at least
> when installing from sdist, with a wheel that depends on the order of the files
> in the wheel), so it's probably only a little bit broken, but "a little bit
> broken" is still broken :-(
> 
> On the other hand, "python -m pip install -U pip" works fine because it avoids
> the exe wrappers.
> 
> There's a lot of scope for user confusion and frustration in all this. For
> standalone pip I've tended to recommend "don't do that" - manually uninstall and
> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective.
> That sort of advice isn't going to be realistic for a pip bundled with CPython.
> 
> Does anyone have any suggestions?

Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?

Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.

MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.

If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.

So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.

Cheers,
Steve