[Python-Dev] PEP 514: Python registration in the Windows registry

[Steve Dower]

On 16Jul2016 1254, Paul Moore wrote:
> On 15 July 2016 at 23:20, Steve Dower <steve.dower at python.org> wrote:
>> Hi all
>>
>> I'd like to get this PEP approved (status changed to Active, IIUC).
>
> Some comments below.

Awesome, thanks! Posted a pull request at 
https://github.com/python/peps/pull/59 for ease of diff reading, and 
some commentary below (with aggressive snipping).

>> Motivation
>> ==========
>>
>> When installed on Windows, the official Python installer creates a registry
>> key for discovery and detection by other applications. This allows tools such
>> as installers or IDEs to automatically detect and display a user's Python
>> installations.
>
> The PEP seems quite strongly focused on GUI tools ... I'd like to avoid tool
> developers reading this section and  thinking "it only applies to GUI tools or
> OS integration, not to me".

Agreed. I tried to avoid any console/GUI-specific terms, but I can 
probably be more explicit about it being useful to both.

> For example, virtualenv introspects the available Python installations
> - see https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L86
> - to support the "-p <interpreter>" flag. To handle this well, it
> would be useful to allow distributions to register a "short tag", so
> that as well as "-p 3.5" or "-p 2", Virtualenv could support (say) "-p
> conda3.4" or "-p pypy2". (The short tag should be at the Company
> level, so "conda" or "pypy", and the version gets added to that).
>
> Another place where this might be useful is the py.exe launcher (it's
> not in scope for this PEP, but having the data needed to allow the
> launcher to invoke any available installation could be useful for
> future enhancements).

virtualenv would be a great example to use. My thinking was that the Tag 
should be appropriate here (perhaps with the Company to disambiguate 
when necessary), and that is now explicit.

Anaconda currently has "Anaconda_4.1.1_64-bit" as their tag, which would 
not be convenient, so an explicit suggestion here would help ensure this 
is useful.

> Another key motivation for me would be to define clearly what
> information tools can rely on being able to get from the available
> registry entries describing what's installed. Whenever I've needed to
> scan the registry, the things I've needed to find out are where I find
> the Python interpreter, what Python version it is, and whether it's
> 32-bit or 64-bit. The first so that I can run Python, and the latter
> two so that I can tell if this is a version I support *without*
> needing to run the interpreter. For me, everything else in this PEP is
> about UI, but those 3 items plus the "short tag" idea are more about
> what capabilities I can provide.

Good points. I discussed architecture with a colleague at one point and 
I'm not entirely sure it's universally useful (what architecture is 
IronPython when built for Any CPU? what architecture is Jython?), but 
maybe something like the contents of importlib.machinery.IMPORT_SUFFIXES 
would be?

>> On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special
>> key that 32-bit processes transparently read and write to rather than
>> accessing the ``Software`` key directly.
>
> It might be worth being more explicit here that 32-bit and 64-bit
> processes see the registry keys slightly differently. More on this
> below.

I considered this and thought I had a link to the official docs about 
it. I don't want this PEP to be mistaken for documentation on how 
registry redirection works :)

>> Backwards Compatibility
>> -----------------------
>>
> Also, Python 3.5 doesn't appear to include the architecture in
> sys.winver either.
>
> ...
> (Unless it adds -32 for 32-bit, and reserves the bare version for
> 64-bit. I've skimmed the CPython source but can't confirm that). The
> documentation of sys.winver makes no mention of whether it
> distinguishes 32- and 64-bit builds. In fact, it states "The value is
> normally the first three characters of version". If we're relying on
> sys.winver being unique by version/architecture, the docs need to say
> so (so that future changes don't accidentally violate that).

I'll update the docs, but your guess is correct. I changed sys.winver on 
32-bit to be "3.5-32" since that matches what py.exe was already using 
to refer to it. I didn't want to invent 
yet-another-way-to-tag-architectures. (I also updated py.exe to match 
tags directly under PythonCore, so 3.5-32 is matched without scanning 
the binary type.)

Also, sys.winver is defined in PCBuild/python.props, which is how we 
accidentally backported the suffix to 2.7.11 :(

>> It is not possible to detect side-by-side installations of both 64-bit and
>> 32-bit versions of Python prior to 3.5 when they have been installed for the
>> current user. Python 3.5 and later always uses different Tags for 64-bit and
>> 32-bit versions.
>
> From what I can see, this latter isn't true. I presume that 64-bit
> uses no suffix, but 32-bit uses a "-32" suffix? This should probably
> be made explicit. At a minimum, if I were writing a tool to list all
> installed Python versions, with only what I have available to go on
> (the PEP and a 64-bit Python 3.5) I wouldn't be able to write correct
> code, as I don't have all the information I need.
>
> Also, if we expect to be able to distinguish 32 and 64 bit
> implementations in this way, that's putting a new restriction on
> sys.winver, that it returns a different value for 32-bit and 64-bit
> builds. If that's the case, I'd rather see that explicitly documented,
> both here and in the sys.winver documentation.
>
> I'd actually prefer a more explicit mechanism going forward, but as
> this is a "backward compatibility" section I'll save that for later.

I don't want to lock in the actual scheme of the tags used by CPython. 
Granted, without SysArchitecture in the key you can't identify the 
architecture from the information you have, but Tags are supposed to be 
treated as opaque with the exception of those that were released prior 
to 3.5 (also not explicit, so I'll fix that).

>> The Company part of the key is intended to group related environments and to
>> ensure that Tags are namespaced appropriately. The key name should be
>> alphanumeric without spaces and likely to be unique. For example, a
>> trademarked
>> name, a UUID, or a hostname would be appropriate::
>>
>>     HKEY_CURRENT_USER\Software\Python\ExampleCorp
>>     HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60
>>     HKEY_CURRENT_USER\Software\Python\www.example.com
>
> I'd suggest adding "Human-readable Company values are preferred".
> UUIDs seem like a horrible idea in practice.

Maybe, but name-squatting seems equally bad (and I've seen enough 
collaborative registration systems like this to see the value in 
guaranteed uniqueness). I'll recommend trademarked names though.

> It's also worth noting that "Display Name" isn't actually as useful as
> it sounds, in practice. A tool that relies on it would report the
> python.org installers as being provided by "PythonCore", which isn't
> particularly user friendly. Maybe we need something in the "Backward
> Compatibility" section going into a bit more detail as to how tools
> should deal with that, and maybe we need to add a "DisplayName" in
> 3.6+.

I'll specify defaults for PythonCore on each of them and add them to 3.6 
(once the PEP is approved and we agree on the values). It certainly 
doesn't harm the usefulness, but we do want to make sure that tools are 
handling old PythonCore entries in a consistent way.

>> If a string value named ``DisplayName`` exists, it should be used to
>> identify the environment to users. Otherwise, the name of the key should be used.
>
> To an extent there's the same comment here as for DisplayName for
> Company - it needs to be defined with consideration for how it will be
> used. This is, of course, more of a "quality of implementation" matter
> than a standards one. But the PEP might benefit from an example of
> use, maybe showing the output from a hypothetical command line tool
> that lists all installations on the machine.

It's defined as being used to "identify ... to users". Equally, the 
SupportUrl "may be displayed or otherwise used to direct users to 
[support]". I feel that these are strong enough definitions, and that 
showing an hypothetical command line tool output might be seen as too 
prescriptive (or alternatively, swinging the pendulum too far away from 
GUI tools).

>> If a string value named ``Version`` exists, it should be used to identify the
>> version of the environment. This is independent from the version of Python
>> implemented by the environment.
>>
>> If a string value named ``SysVersion`` exists, it must be in ``x.y`` or
>> ``x.y.z`` format matching the version returned by ``sys.version_info`` in the
>> interpreter. Otherwise, if the Tag matches this format it is used. If not,
>> the Python version is unknown.
>
> I'm not too happy with this. [...]
>
>> Note that each of these values is recommended, but optional.
>
> SysVersion and SysArchitecture (or a Tag that works as a fallback)
> should be mandatory. Otherwise I'm OK with this statement.

Snipped most of the details because I agree it's unsatisfying right now, 
but I disagree with enough of the counterproposal that it was getting to 
be messy commenting on each bit.

Basically, I added SysArchitecture (to match platform.architecture()[0], 
typically '32bit' or '64bit' but extensible without having to define all 
potential values in the PEP) with a note that for PythonCore it should 
be inferred from the registry path.

SysVersion no longer allows inferring it from the Tag, except for 
PythonCore and only when SysVersion is missing.

I'm very keen to not force any of this information to be required as it 
is very difficult to know how to deal with interpreters that don't 
include it. Does virtualenv refuse to list/use it, even if the install 
path is valid, just because SysArchitecture was omitted? What if the 
registry becomes corrupt - should Visual Studio refuse to show what 
information it can obtain?

I already stated that all information is recommended. The change I've 
made now is that tools shouldn't work too hard to guess - if SysVersion 
is missing, they can simply say they don't know what version the 
interpreter is. If knowing the version is critically important, they can 
refuse to use it, but for many applications this is not going to be the 
case.

And Python 3.6 will specify all of the keys. Adding it to Python 3.5 is 
only likely to cause issues now with people who test against 3.5.3 and 
not 3.5.2, which didn't have the keys.

>> Beneath the environment key, an ``InstallPath`` key must be created. This key
>> is always named ``InstallPath``, and the default value must match
>> ``sys.prefix``::
>>
>>     HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath
>>         (Default) = "C:\ExampleCorpPy36"
>>
>> If a string value named ``ExecutablePath`` exists, it must be a path to the
>> ``python.exe`` (or equivalent) executable. Otherwise, the interpreter
>> executable is assumed to be called ``python.exe`` and exist in the directory
>> referenced by the default value.
>>
>> If a string value named ``WindowedExecutablePath`` exists, it must be a path
>> to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed
>> interpreter executable is assumed to be called ``pythonw.exe`` and exist in
>> the directory referenced by the default value.
>
> These two items assume implicitly that a Python installation must
> provide python.exe and pythonw.exe. I'm inclined to make this
> explicit. Specifically, I think it's crucial that tools can read the
> (console or windowed) executable path as described here, and run that
> executable with standard Python command line arguments, and expect it
> to work. Otherwise there's little point in the installation
> registering its existence.

Again, there's a backwards compatibility argument here in that Python 
3.4 and earlier did not create full paths to the executables. But that 
can be called out separately.

When you say "assume implicitly ... python.exe and pythonw.exe", do you 
mean executables by those names (counterexample - IronPython includes 
ipy.exe and ipyw.exe, which you either know specially or would discover 
from these keys)? Or BOTH executables (e.g. python.exe AND pythonw.exe)? 
Or executables with equivalent behaviour?

I'd argue that the whole PEP only applies to Python interpreters, and if 
you don't support standard command line arguments you aren't really a 
Python interpreter and shouldn't be registering as one. But I hesitate 
to try and define a hard rule that captures all the possible nuances 
here - I'd rather deal with it by having users file bugs against 
offending interpreters for not working correctly.

> I can see an argument for a distribution providing just python.exe and
> omitting pythonw.exe (or even the other way around). But I can't see
> how I could write generic code to work with such a distribution. So
> let's disallow that possibility until someone comes up with a concrete
> use case

I think in this case, you'd either specify both keys with the same path 
(so tools that want the windowed executable are going to get a console 
window) or omit the key and make sure you don't have a "pythonw.exe" in 
your install directory (which seems unlikely :) ). These keys are mainly 
about the possibility of renaming the executables, as shown in the example.

But I've added a note reminding tools developers that the executable may 
not exist, and attempting to launch it may fail (i.e. business as usual).

>> Other Keys
>> ----------
>>
>> Some other registry keys are used for defining or inferring search paths
>> under certain conditions. A third-party installation is permitted to define
>> these keys under their Company-Tag key, however, the interpreter must be
>> modified and rebuilt in order to read these values. Alternatively, the
>> interpreter may be modified to not use any registry keys for determining
>> search paths. Making such changes is a decision for the third party; this PEP
>> makes no recommendation either way.
>
> I think we need to be clearer here. ...

Great suggestion. I've revised this section.

(I have vague plans to make the PythonPath subkey redundant in more 
cases for 3.6, and I *think* we can probably drop the Modules key 
completely, but I'm not entirely sure it's a good idea. Still thinking 
about it :) )

Cheers,
Steve