From ssbarnea at redhat.com  Mon Sep 24 05:48:12 2018
From: ssbarnea at redhat.com (Sorin Sbarnea)
Date: Mon, 24 Sep 2018 10:48:12 +0100
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python to
 python3 on newer distributions
Message-ID: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>

Hi!

I would like to propose a change on PEP 394 recommendations which were originally made almost 7 years ago when python3 was bleeding edge for most distributions.

Now almost all major distributions are switching to python3 by default and not installing python2 in their base deployment.

They also seem careful to follow PEP 394 which is a good thing to do. Still, the side effect of this is that based on that they do *not* create the symlink between python -> python3 which means a newly spawned machine would give "command not found" when trying to run `python`.

The lack of default breaks lots of tools which only need one python interpreter to run, regardless its version.

One notable issue is Ansible where users would be forced to do extra steps for controlling new machines with only python3 on them.
https://github.com/ansible/ansible/issues/45852 <https://github.com/python/peps/pull/785> -- yep, not being able to cleanly run configuration management may create a chicken and the egg issue, as users would want to use it to configure the machine.

Other examples are any scripts that now will not be able to ship with a generic shebang line. It would be impossible to make a generic python script executable to run on all platforms, even if the code itself would work without problems. 

I may also mention the (in)famous usage of `curl <url> | python` approach which is popular.  Without having a "python" command, developers would have to 
update the documentation to mention the use of python or python3 based on the platform (creating more work which may be avoidable).

If PEP recommendation would be made more generic, and to suggest distribution publishers to provide a python symlink to whichever is the default interpreter on their current OS release, we could easily avoid this cascade of changes that would provide a poor UX.

I raised a first version of that update at https://github.com/python/peps/pull/785 <https://github.com/python/peps/pull/785> 

Arch Linux made the decision to implement this regardless the PEP. Instead of going the path of ignoring it it, wouldn't be much better if we would improve it so it can be followed by everyone without clearly bad side-effects for the endusers?

Thanks
Sorin Sbarnea

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20180924/1b1c9e58/attachment.html>

From fungi at yuggoth.org  Mon Sep 24 10:33:01 2018
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Mon, 24 Sep 2018 14:33:01 +0000
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
Message-ID: <20180924143301.ejiy3ugesn3f4chv@yuggoth.org>

On 2018-09-24 10:48:12 +0100 (+0100), Sorin Sbarnea wrote:
[...]
> Now almost all major distributions are switching to python3 by
> default and not installing python2 in their base deployment.
> 
> They also seem careful to follow PEP 394 which is a good thing to
> do. Still, the side effect of this is that based on that they do
> *not* create the symlink between python -> python3 which means a
> newly spawned machine would give "command not found" when trying
> to run `python`.
> 
> The lack of default breaks lots of tools which only need one
> python interpreter to run, regardless its version.
[...]

At least in Debian, they're looking at dropping the /usr/bin/python
symlink from future python2.7 packages, so as to get out of
appearing to define a "default Python interpreter version" and
allowing users to then supply their own if they see fit.

    https://lists.debian.org/debian-python/2018/05/msg00063.html

There is also strong sentiment within the Debian community against
ever supplying a /usr/bin/python which invokes python3, even once
they cease shipping a python2.7 interpreter entirely.

> One notable issue is Ansible where users would be forced to do
> extra steps for controlling new machines with only python3 on
> them. https://github.com/ansible/ansible/issues/45852
> <https://github.com/python/peps/pull/785> -- yep, not being able
> to cleanly run configuration management may create a chicken and
> the egg issue, as users would want to use it to configure the
> machine.

On servers which don't have Python installed into the global system
context, or where you want Ansible to invoke a particular
non-default interpreter installation, something like
ansible_python_interpreter would still be needed, right? (Think:
RHEL Software Collections.)

> Other examples are any scripts that now will not be able to ship
> with a generic shebang line. It would be impossible to make a
> generic python script executable to run on all platforms, even if
> the code itself would work without problems.

PEP 394 already allows bin/python in a Python 3.x virtual
environment to invoke python3. Distribution packaging similarly
works around these issues by patching shebang lines to refer to an
explicit major interpreter version. This does leave installing
sdists/wheels into the global system context as a broken use case,
but it's become increasingly popular within the Python packaging
community to declare that particular case dangerous and unsupported:

    https://github.com/pypa/pip/issues/4805#issuecomment-381404626

Also some distributions (e.g. Red Hat) have long held that the
system-provided Python interpreter is only intended for use by
Python-based applications shipped in the distribution itself, and
that if you have a need for a generic Python environment to run
arbitrary scripts you should install one in a non-system path.
They've traditionally introduced non-upstream behaviors into their
system Python interpreter in service of backporting newer versions
of software they've committed to support, so there's no guarantee it
will match the featureset for the claimed interpreter version
anyway.

> I may also mention the (in)famous usage of `curl <url> | python`
> approach which is popular.  Without having a "python" command,
> developers would have to update the documentation to mention the
> use of python or python3 based on the platform (creating more work
> which may be avoidable).
[...]

Maybe it's my security background, but I'd rather see that
anti-pattern die in a fire. If people don't even know what version
of Python they should tack on the end of that pipe, they almost
certainly shouldn't be running random scripts they find on the Web
through it. Also, if the idea is that more and more distros are no
longer shipping Python 2.x by default but _are_ shipping 3.x then
changing that to `... | python3` as the default recommendation seems
pretty simple to me (does still make my skin crawl though).

> Arch Linux made the decision to implement this regardless the PEP.
> Instead of going the path of ignoring it it, wouldn't be much
> better if we would improve it so it can be followed by everyone
> without clearly bad side-effects for the endusers?

I think most of the Python community has traditionally considered
the choice Arch foisted on their users to be a terrible and not at
all recommended configuration. I certainly wouldn't want to see it
propagated further.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20180924/69f12c21/attachment.sig>

From barry at python.org  Tue Sep 25 10:47:57 2018
From: barry at python.org (Barry Warsaw)
Date: Tue, 25 Sep 2018 10:47:57 -0400
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
Message-ID: <2F73FB2E-4D76-415D-80BF-F0B52E5A2709@python.org>

On Sep 24, 2018, at 05:48, Sorin Sbarnea <ssbarnea at redhat.com> wrote:
> 
> I would like to propose a change on PEP 394 recommendations which were originally made almost 7 years ago when python3 was bleeding edge for most distributions.

The last time this topic came up, we essentially deferred changing PEP 394 until after Python 2?s official EOL date.  I think that?s still appropriate.  Once Python 2 is for good and ever retired, then it?s worth considering changes to PEP 394?s language.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20180925/3ca6245a/attachment.sig>

From doko at ubuntu.com  Tue Sep 25 11:29:31 2018
From: doko at ubuntu.com (Matthias Klose)
Date: Tue, 25 Sep 2018 17:29:31 +0200
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <20180924143301.ejiy3ugesn3f4chv@yuggoth.org>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
 <20180924143301.ejiy3ugesn3f4chv@yuggoth.org>
Message-ID: <5167a891-8df4-d413-96fc-76abe4d7bc81@ubuntu.com>

On 24.09.2018 16:33, Jeremy Stanley wrote:
> On 2018-09-24 10:48:12 +0100 (+0100), Sorin Sbarnea wrote:
> [...]
>> Now almost all major distributions are switching to python3 by
>> default and not installing python2 in their base deployment.
>>
>> They also seem careful to follow PEP 394 which is a good thing to
>> do. Still, the side effect of this is that based on that they do
>> *not* create the symlink between python -> python3 which means a
>> newly spawned machine would give "command not found" when trying
>> to run `python`.
>>
>> The lack of default breaks lots of tools which only need one
>> python interpreter to run, regardless its version.
> [...]
> 
> At least in Debian, they're looking at dropping the /usr/bin/python
> symlink from future python2.7 packages, so as to get out of
> appearing to define a "default Python interpreter version" and
> allowing users to then supply their own if they see fit.
> 
>     https://lists.debian.org/debian-python/2018/05/msg00063.html
> 
> There is also strong sentiment within the Debian community against
> ever supplying a /usr/bin/python which invokes python3, even once
> they cease shipping a python2.7 interpreter entirely.

Right, Debian doesn't want to re-use the "python" binary name pointing to
python3.  For Ubuntu, I can propose a change, once we stop shipping packages
relying on the python shebang.  However that wouldn't be before within a few years.

>> One notable issue is Ansible where users would be forced to do
>> extra steps for controlling new machines with only python3 on
>> them. https://github.com/ansible/ansible/issues/45852
>> <https://github.com/python/peps/pull/785> -- yep, not being able
>> to cleanly run configuration management may create a chicken and
>> the egg issue, as users would want to use it to configure the
>> machine.
> 
> On servers which don't have Python installed into the global system
> context, or where you want Ansible to invoke a particular
> non-default interpreter installation, something like
> ansible_python_interpreter would still be needed, right? (Think:
> RHEL Software Collections.)
> 
>> Other examples are any scripts that now will not be able to ship
>> with a generic shebang line. It would be impossible to make a
>> generic python script executable to run on all platforms, even if
>> the code itself would work without problems.
> 
> PEP 394 already allows bin/python in a Python 3.x virtual
> environment to invoke python3. Distribution packaging similarly
> works around these issues by patching shebang lines to refer to an
> explicit major interpreter version. This does leave installing
> sdists/wheels into the global system context as a broken use case,
> but it's become increasingly popular within the Python packaging
> community to declare that particular case dangerous and unsupported:
> 
>     https://github.com/pypa/pip/issues/4805#issuecomment-381404626

pip still unfortunately allows you to install and remove dpkg/rpm based
installations, and with the pypi prominent suggestion to use pip install
everywhere, that probably doesn't get better.

Matthias

From ncoghlan at gmail.com  Wed Sep 26 06:16:55 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 Sep 2018 20:16:55 +1000
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <5167a891-8df4-d413-96fc-76abe4d7bc81@ubuntu.com>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
 <20180924143301.ejiy3ugesn3f4chv@yuggoth.org>
 <5167a891-8df4-d413-96fc-76abe4d7bc81@ubuntu.com>
Message-ID: <CADiSq7cRiSRkqk-FKJvW7vjzsT81wLL5oi0m2+4hbMsa1k6Xyg@mail.gmail.com>

On Wed, 26 Sep 2018 at 01:56, Matthias Klose <doko at ubuntu.com> wrote:
> On 24.09.2018 16:33, Jeremy Stanley wrote:
> > PEP 394 already allows bin/python in a Python 3.x virtual
> > environment to invoke python3. Distribution packaging similarly
> > works around these issues by patching shebang lines to refer to an
> > explicit major interpreter version. This does leave installing
> > sdists/wheels into the global system context as a broken use case,
> > but it's become increasingly popular within the Python packaging
> > community to declare that particular case dangerous and unsupported:
> >
> >     https://github.com/pypa/pip/issues/4805#issuecomment-381404626
>
> pip still unfortunately allows you to install and remove dpkg/rpm based
> installations, and with the pypi prominent suggestion to use pip install
> everywhere, that probably doesn't get better.

It should if system deb packages were to consistently populate the
INSTALLER file with a value other than pip:

* https://github.com/pypa/pip/issues/5346 (disables the pip upgrade message)
* https://github.com/pypa/pip/issues/5605 (figure out a way to tell
pip which INSTALLER files to leave alone)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Sep 26 06:19:50 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 Sep 2018 20:19:50 +1000
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <2F73FB2E-4D76-415D-80BF-F0B52E5A2709@python.org>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
 <2F73FB2E-4D76-415D-80BF-F0B52E5A2709@python.org>
Message-ID: <CADiSq7c0P1M98oQvOpu+7aioAW81BMopEMPmLh+i5TPx9jdkjA@mail.gmail.com>

On Wed, 26 Sep 2018 at 00:49, Barry Warsaw <barry at python.org> wrote:
>
> On Sep 24, 2018, at 05:48, Sorin Sbarnea <ssbarnea at redhat.com> wrote:
> >
> > I would like to propose a change on PEP 394 recommendations which were originally made almost 7 years ago when python3 was bleeding edge for most distributions.
>
> The last time this topic came up, we essentially deferred changing PEP 394 until after Python 2?s official EOL date.  I think that?s still appropriate.  Once Python 2 is for good and ever retired, then it?s worth considering changes to PEP 394?s language.

Yep, this is still where Fedora is at as well - step 1 is to eliminate
legacy references to the python symlink, and then step 2 will be to
figure out if/when it ever comes back as a way of referring to a
generic version of Python.

Personally, I suspect it's more likely that we''ll start shipping
https://github.com/brettcannon/python-launcher with CPython at some
point, similar to the way we ship the original py launcher for
Windows.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From encukou at gmail.com  Wed Sep 26 06:33:12 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 26 Sep 2018 12:33:12 +0200
Subject: [Linux-SIG] Updating PEP 394 to recommend soft-linking python
 to python3 on newer distributions
In-Reply-To: <CADiSq7c0P1M98oQvOpu+7aioAW81BMopEMPmLh+i5TPx9jdkjA@mail.gmail.com>
References: <CD6E6832-E1FE-430A-BDA9-9B33D083C1A1@redhat.com>
 <2F73FB2E-4D76-415D-80BF-F0B52E5A2709@python.org>
 <CADiSq7c0P1M98oQvOpu+7aioAW81BMopEMPmLh+i5TPx9jdkjA@mail.gmail.com>
Message-ID: <cf3362c4-d7c6-3d1a-1c5c-ccfe777d532a@gmail.com>

On 9/26/18 12:19 PM, Nick Coghlan wrote:
> On Wed, 26 Sep 2018 at 00:49, Barry Warsaw <barry at python.org> wrote:
>>
>> On Sep 24, 2018, at 05:48, Sorin Sbarnea <ssbarnea at redhat.com> wrote:
>>>
>>> I would like to propose a change on PEP 394 recommendations which were originally made almost 7 years ago when python3 was bleeding edge for most distributions.
>>
>> The last time this topic came up, we essentially deferred changing PEP 394 until after Python 2?s official EOL date.  I think that?s still appropriate.  Once Python 2 is for good and ever retired, then it?s worth considering changes to PEP 394?s language.
> 
> Yep, this is still where Fedora is at as well - step 1 is to eliminate
> legacy references to the python symlink, and then step 2 will be to
> figure out if/when it ever comes back as a way of referring to a
> generic version of Python.

Actually, step 1 is done, minus stragglers that are hard to find and 
won't switch until things break for them.
I would be fine switching "python" to Python 3 in Fedora 30 (which is 
being developed now, and will be supported into 2020), but only if we 
agree it's the thing to do -- and I consider PEP 394 to be the place for 
core dev consensus.

> Personally, I suspect it's more likely that we''ll start shipping
> https://github.com/brettcannon/python-launcher with CPython at some
> point, similar to the way we ship the original py launcher for
> Windows.

I have doubts that would help. Until `py` is everywhere, you'll still 
have different ways to start Python on different systems, and `py -3` 
vs. `python3` is, frankly, not much of a difference.