From barry at python.org  Tue Aug  1 18:54:55 2017
From: barry at python.org (Barry Warsaw)
Date: Tue, 1 Aug 2017 18:54:55 -0400
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
Message-ID: <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>

On Jul 31, 2017, at 19:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 1 August 2017 at 03:47, Barry Warsaw <barry at python.org> wrote:
>> On Jul 26, 2017, at 00:29, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> 
>>> At the PEP 394 level, what I'd personally like to see us do is to
>>> split the "Recommendations" section into two separate sections:
>>> "Python User Recommendations" and "Platform Publisher Recommendations?
>> 
>> I?d like to keep PEP 394 as short and proscriptive as possible.  I?ve always read PEP 394 as being aimed at ?platform publisher recommendations?, i.e. distributions who provide.  I think a user guide would make a good companion (i.e. separate) PEP.
> 
> Until yesterday I would have said the same thing regarding the current
> contents of PEP 394, but a surprisingly large number of the points in
> the new user recommendations section actually came from the current
> text of PEP 394 - they're down at the end of the recommendations
> section, and when they were all in one list, it was easy to miss that
> we'd crossed over from "Advice to platform publishers" into "Advice
> for platform users?.

After having reviewed the proposed PEP 548 PR, I still think we can keep 394 and target it specifically to platform vendors.

> - we declare "/usr/bin/env python" and other environment aware
> invocations as already being effectively addressed, thanks to venv,
> environment modules, conda, and other existing PATH based solutions
> that change the interactive meaning of "python"
> - by some suitable mechanism, we offer 3 different configurations for
> the absolute "/usr/bin/python" path: symlink to /usr/bin/python2,
> symlink to /usr/bin/python3, and "Not configured yet, here's how to
> select your default interpreter" (as a matter of UX, I'm beginning to
> think we'll need to make the last one an actual installable script in
> its own right, as the default interpreter not found error from the
> shebang handler isn't particularly informative)

Another option is to make /usr/bin/python not actually *the* Python interpreter, but instead a wrapper that does $magic to get the right version.  Several options have been bandied about over the years, and IIRC, Windows does something like this.  Maybe it?s switchable based on a config file (both system and user overridable), or maybe there?s some other way.  It won?t help start-up time <wink> but it could provide a better user experience.

> In that light, it may make sense to more explicitly focus the PEP 394
> recommendations on *absolute* invocations that *don't* rely on PATH.

+1

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20170801/1b051277/attachment.sig>

From doko at ubuntu.com  Wed Aug  2 02:00:40 2017
From: doko at ubuntu.com (Matthias Klose)
Date: Wed, 2 Aug 2017 08:00:40 +0200
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
Message-ID: <d800758a-e0e3-b3fd-7504-bc66935e0558@ubuntu.com>

On 01.08.2017 01:40, Nick Coghlan wrote:
> On 1 August 2017 at 03:47, Barry Warsaw <barry at python.org> wrote:
>> On Jul 26, 2017, at 00:29, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> At the PEP 394 level, what I'd personally like to see us do is to
>>> split the "Recommendations" section into two separate sections:
>>> "Python User Recommendations" and "Platform Publisher Recommendations?
>>
>> I?d like to keep PEP 394 as short and proscriptive as possible.  I?ve always read PEP 394 as being aimed at ?platform publisher recommendations?, i.e. distributions who provide.  I think a user guide would make a good companion (i.e. separate) PEP.
> 
> Until yesterday I would have said the same thing regarding the current
> contents of PEP 394, but a surprisingly large number of the points in
> the new user recommendations section actually came from the current
> text of PEP 394 - they're down at the end of the recommendations
> section, and when they were all in one list, it was easy to miss that
> we'd crossed over from "Advice to platform publishers" into "Advice
> for platform users".
> 
>>> For Fedora for example, we think that dropping "/usr/bin/python" from
>>> the repositories entirely would be too large of a compatibility break,
>>> whereas we're comfortable with the idea that by 2020, it will be
>>> reasonable to require that anything still running in "/usr/bin/python"
>>> be hybrid Python 2/3 code, such that finally changing the default
>>> won't cause too much immediate breakage (and for things that do still
>>> break, the recommended fix will be code modernisation to get into the
>>> hybrid subset prior to upgrading the OS).
>>
>> I like recommending /usr/bin/python be bilingual, but I?m wondering how you?ll enforce this, and what the implication is for running that at the command line.  Again, that gets to the UI/UX issue.
> 
> We're still working on the details (see
> https://fedoraproject.org/wiki/User:Ncoghlan/Default_python_module for
> my current notes), but what I'm currently leaning towards advocating
> at the Fedora level is:
> 
> - we declare "/usr/bin/env python" and other environment aware
> invocations as already being effectively addressed, thanks to venv,
> environment modules, conda, and other existing PATH based solutions
> that change the interactive meaning of "python"
> - by some suitable mechanism, we offer 3 different configurations for
> the absolute "/usr/bin/python" path: symlink to /usr/bin/python2,
> symlink to /usr/bin/python3, and "Not configured yet, here's how to
> select your default interpreter" (as a matter of UX, I'm beginning to
> think we'll need to make the last one an actual installable script in
> its own right, as the default interpreter not found error from the
> shebang handler isn't particularly informative)

that would only work, if the distribution has all uses of 'python' removed or
replaced by python2.  I don't see a re-use of the python command before it has
stopped shipping in a distro release.  Assuming that a next Debian release would
be in 2019, not shipping the python command in 2021, and reintroducing it in
2023 as python3 ... Is this really worth it?

Matthias

From ncoghlan at gmail.com  Wed Aug  2 09:57:15 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 2 Aug 2017 23:57:15 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
Message-ID: <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>

On 2 August 2017 at 08:54, Barry Warsaw <barry at python.org> wrote:
> On Jul 31, 2017, at 19:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> - we declare "/usr/bin/env python" and other environment aware
>> invocations as already being effectively addressed, thanks to venv,
>> environment modules, conda, and other existing PATH based solutions
>> that change the interactive meaning of "python"
>> - by some suitable mechanism, we offer 3 different configurations for
>> the absolute "/usr/bin/python" path: symlink to /usr/bin/python2,
>> symlink to /usr/bin/python3, and "Not configured yet, here's how to
>> select your default interpreter" (as a matter of UX, I'm beginning to
>> think we'll need to make the last one an actual installable script in
>> its own right, as the default interpreter not found error from the
>> shebang handler isn't particularly informative)
>
> Another option is to make /usr/bin/python not actually *the* Python interpreter, but instead a wrapper that does $magic to get the right version.  Several options have been bandied about over the years, and IIRC, Windows does something like this.  Maybe it?s switchable based on a config file (both system and user overridable), or maybe there?s some other way.  It won?t help start-up time <wink> but it could provide a better user experience.

The main case I'm interested in handling is systems that only have a
Python 3 stack and either don't have a Python 2 stack available at
all, or else the user doesn't actually want two parallel Python
installs (e.g. in a container image).

This already works OK-ish for "/usr/bin/env python" scripts - you can
just use venv or something like Software Collections to make "which
python" resolve to a Python 3 runtime, and you're done.

However, what I'm now considering is that instead of having the
default state be "/usr/bin/python is missing", I may be able to make
it "/usr/bin/python is present, but will be a Python script that tries
to compile the given script with the default Python runtime and
reports the results, rather than actually running it". That way:

- for scripts attempting to use Python 2 syntax on Python 3 only
systems, the script will still report the syntax error, but it will
*also* explain that Python 2.7 is not installed (and how to install
it)
- for scripts syntactically compatible with Python 3 on Python 3 only
systems, the script will be able to say "Either run this explicitly
with python3, or else reconfigure the system to do that implicitly"
(since there are still potential runtime compatibility problems that
mere compilation won't find, so just because it compiled, doesn't mean
it will work as intended)
- if both Python 2.7 and 3.x are installed, than the details of the
feedback would depend on how the respective compilations went

There would then be a distro supported way to switch that custom
script out for a plain symlink to the desired Python stack (in Fedora,
it's looking like we're going to have to use the new modularity
tooling in order to handle immutable Atomic Host images correctly, and
then work out how we want to handle the traditional monolithic repo
model based on that).

Another variant of that idea would be to have the default handler be
something like Geoffrey Thomas's pythonmux project
(https://github.com/geofft/pythonmux), and then patch
sys.excepthandler to report the Python version executed when
"sys.executable" isn't qualified as either Python 2 or Python 3 for
easier debugging of otherwise potentially cryptic error messages.

If we did do that, then I'd skip the "pyversions" comment idea, and
instead have the logic just be:

- if only /usr/bin/python2 is present, use it
- if only /usr/bin/python3 is present, use it
- if both are present, use /usr/bin/python2

As far I'm aware, that would match the logic currently used in the
Windows launcher, although PEP 397 is a bit ambiguous on that point.

Bringing up WIndows is also interesting in general, as for its shebang
line processing, Windows kinda follows PEP 394, where it supports all
of the following:

    python
    /usr/bin/python
    /usr/local/bin/python
    /usr/bin/env python

Plus their "python2" and "python3" equivalents and the specific
"pythonX.Y" versions.

In 3.6, the launcher changed such that just typing "py" will launch
the newest Python version it knows about, rather than defaulting to
Python 2 when both are present. However, the *shebang line* emulation
still defaults to Python 2 if it has the option, with a series of
environment variables and config files used to manage their
interpretation (that's mainly due to the challenges of using symlinks
on Windows, rather than for any other reason).

I'm thinking that if we update the guidance in PEP 394, it's also
likely to make sense to change the Windows launcher's shebang line
processing in 3.7+ to be such that it defaults to Python 3 even when
both 2 & 3 are present.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug  2 10:11:09 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 3 Aug 2017 00:11:09 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <d800758a-e0e3-b3fd-7504-bc66935e0558@ubuntu.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <d800758a-e0e3-b3fd-7504-bc66935e0558@ubuntu.com>
Message-ID: <CADiSq7fnLD7j7BXW2waWxLauc3zKrbDnG-3yZ8V7aiY9LrBmYQ@mail.gmail.com>

On 2 August 2017 at 16:00, Matthias Klose <doko at ubuntu.com> wrote:
> On 01.08.2017 01:40, Nick Coghlan wrote:
>> We're still working on the details (see
>> https://fedoraproject.org/wiki/User:Ncoghlan/Default_python_module for
>> my current notes), but what I'm currently leaning towards advocating
>> at the Fedora level is:
>>
>> - we declare "/usr/bin/env python" and other environment aware
>> invocations as already being effectively addressed, thanks to venv,
>> environment modules, conda, and other existing PATH based solutions
>> that change the interactive meaning of "python"
>> - by some suitable mechanism, we offer 3 different configurations for
>> the absolute "/usr/bin/python" path: symlink to /usr/bin/python2,
>> symlink to /usr/bin/python3, and "Not configured yet, here's how to
>> select your default interpreter" (as a matter of UX, I'm beginning to
>> think we'll need to make the last one an actual installable script in
>> its own right, as the default interpreter not found error from the
>> shebang handler isn't particularly informative)
>
> that would only work, if the distribution has all uses of 'python' removed or
> replaced by python2.  I don't see a re-use of the python command before it has
> stopped shipping in a distro release.  Assuming that a next Debian release would
> be in 2019, not shipping the python command in 2021, and reintroducing it in
> 2023 as python3 ... Is this really worth it?

For Debian potentially not, but Fedora's been pushing the
python->python2 shift hard enough that we think we can get there by
2019 (or potentially earlier, given that it looks like 100s of the
affected packages may actually be able to updated automatically). The
twice-yearly release cycle also makes a big difference to the
viability of the plan.

That said, that's also why my proposed updates are more about saying
to distributors "you know your specific audience better than we do,
and we're now comfortable with the idea of making /usr/bin/python
point to /usr/bin/python3 by default if that's what makes sense for
your user base" rather than "You should make /usr/bin/python point to
/usr/bin/python3 by default" (since the latter is still going to be
the *wrong* choice for many redistributors).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug  2 10:29:43 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 3 Aug 2017 00:29:43 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
Message-ID: <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>

On 2 August 2017 at 23:57, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Another variant of that idea would be to have the default handler be
> something like Geoffrey Thomas's pythonmux project
> (https://github.com/geofft/pythonmux), and then patch
> sys.excepthandler to report the Python version executed when
> "sys.executable" isn't qualified as either Python 2 or Python 3 for
> easier debugging of otherwise potentially cryptic error messages.
>
> If we did do that, then I'd skip the "pyversions" comment idea, and
> instead have the logic just be:
>
> - if only /usr/bin/python2 is present, use it
> - if only /usr/bin/python3 is present, use it
> - if both are present, use /usr/bin/python2

While I'm happy I at least brought it up, I still can't convince
myself to actually *like* this idea relative to the "compile it and
report on the results" approach combined with the ability to opt-in to
replacing that default script with a symlink to the desired version.

This is due to what happens in a situation like Fedora's with various
mux behaviours:

= Approach 1: impllcitly try Py3 =

All the same problems as a plain symlink. If we were to patch
sys.excepthandler() to report the version when sys.executable ==
'/usr/bin/python', then that UX tweak would apply just as well to the
symlink switching approach (maybe I should propose that as an
*upstream* change, and check for '/usr/local/bin/python' as well?).

= Approach 2: default to Py2, complain if it is missing =

Same effective consequence as defaulting to a custom script that
checks for Py3 syntax compatibility, just with a different more
implicit remedy ("Install Py2" vs "Install Py2 & make it the default
Python"). Also has the downside that you can't opt-in to making Py3
the default instead.

= Approach 3: add a magic script comment to opt-in to trying Py3 =

This whole problem just doesn't feel like a per-script setting to me -
it feels like a system administrator setting, where they say either
"Yes, I am confident in my ability to debug errors arising from myself
and others trying to run Python 2 scripts on Python 3", or "I want to
preserve compatibility with older Py2 only systems for now, even if
that means running parallel Python stacks on my newer systems".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From encukou at gmail.com  Wed Aug  2 10:41:24 2017
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 2 Aug 2017 16:41:24 +0200
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
Message-ID: <37e7287e-9ba6-f64f-9bba-3461e1b4e97b@gmail.com>

On 08/02/2017 04:29 PM, Nick Coghlan wrote:
> On 2 August 2017 at 23:57, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Another variant of that idea would be to have the default handler be
>> something like Geoffrey Thomas's pythonmux project
>> (https://github.com/geofft/pythonmux), and then patch
>> sys.excepthandler to report the Python version executed when
>> "sys.executable" isn't qualified as either Python 2 or Python 3 for
>> easier debugging of otherwise potentially cryptic error messages.
>>
>> If we did do that, then I'd skip the "pyversions" comment idea, and
>> instead have the logic just be:
>>
>> - if only /usr/bin/python2 is present, use it
>> - if only /usr/bin/python3 is present, use it
>> - if both are present, use /usr/bin/python2
> 
> While I'm happy I at least brought it up, I still can't convince
> myself to actually *like* this idea relative to the "compile it and
> report on the results" approach combined with the ability to opt-in to
> replacing that default script with a symlink to the desired version.


Just compiling won't work -- consider `import pathlib` for example.

This is UX, but for sysadmins (and other command-line users). Rather 
than trying to do what they mean, we should aim to synchronize their 
expectations and the reality.
I'd rather have /usr/bin/python failing hard with a good error message 
than trying non-obvious heuristics. Avoid the temptation to guess.


> This is due to what happens in a situation like Fedora's with various
> mux behaviours:
> 
> = Approach 1: impllcitly try Py3 =
> 
> All the same problems as a plain symlink. If we were to patch
> sys.excepthandler() to report the version when sys.executable ==
> '/usr/bin/python', then that UX tweak would apply just as well to the
> symlink switching approach (maybe I should propose that as an
> *upstream* change, and check for '/usr/local/bin/python' as well?).
> 
> = Approach 2: default to Py2, complain if it is missing =
> 
> Same effective consequence as defaulting to a custom script that
> checks for Py3 syntax compatibility, just with a different more
> implicit remedy ("Install Py2" vs "Install Py2 & make it the default
> Python"). Also has the downside that you can't opt-in to making Py3
> the default instead.
> 
> = Approach 3: add a magic script comment to opt-in to trying Py3 =
> 
> This whole problem just doesn't feel like a per-script setting to me -
> it feels like a system administrator setting, where they say either
> "Yes, I am confident in my ability to debug errors arising from myself
> and others trying to run Python 2 scripts on Python 3", or "I want to
> preserve compatibility with older Py2 only systems for now, even if
> that means running parallel Python stacks on my newer systems".

From barry at python.org  Wed Aug  2 11:42:44 2017
From: barry at python.org (Barry Warsaw)
Date: Wed, 02 Aug 2017 11:42:44 -0400
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
Message-ID: <olsrtf$hnb$1@blaine.gmane.org>

Nick Coghlan wrote:

> Another variant of that idea would be to have the default handler be
> something like Geoffrey Thomas's pythonmux project
> (https://github.com/geofft/pythonmux),

Yep, that's the project I was thinking of.  I still like this proposal 
best, although I might quibble about some details (I'll do that on 
Thomas's GH project).  It gives control to the right principles, i.e. 
the script author, or the person invoking the script (via environment 
variable), and doesn't require much from either the system administrator 
or the platform vendor, other than of course putting pythonmux on 
/usr/bin/python.

I think this also nicely supports the interactive use case, so it  kills 
two birds with one stone.

 > and then patch
> sys.excepthandler to report the Python version executed when
> "sys.executable" isn't qualified as either Python 2 or Python 3 for
> easier debugging of otherwise potentially cryptic error messages.

I do think it would be a good to have a diagnostic mode.  I'm not sure 
that should be the default or not.

> > If we did do that, then I'd skip the "pyversions" comment idea, and
> instead have the logic just be:
>
> - if only /usr/bin/python2 is present, use it
> - if only /usr/bin/python3 is present, use it
> - if both are present, use /usr/bin/python2

Why don't you like the pyversions comment?  I think he has the syntax 
wrong in his blog post, but I'd have to verify that.  I still think it's 
worthwhile to keep.

Cheers,
-Barry


From barry at python.org  Wed Aug  2 11:49:22 2017
From: barry at python.org (Barry Warsaw)
Date: Wed, 02 Aug 2017 11:49:22 -0400
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
Message-ID: <olss9t$u69$1@blaine.gmane.org>

Nick Coghlan wrote:

> = Approach 3: add a magic script comment to opt-in to trying Py3 =
>
> This whole problem just doesn't feel like a per-script setting to me -
> it feels like a system administrator setting, where they say either
> "Yes, I am confident in my ability to debug errors arising from myself
> and others trying to run Python 2 scripts on Python 3", or "I want to
> preserve compatibility with older Py2 only systems for now, even if
> that means running parallel Python stacks on my newer systems".

I think it *is* a per-script setting.  Remember that we're trying to fix 
the problem of what /usr/bin/python means to 1) third party scripts that 
the platform has no control over; 2) the CLI UX.

Now, a system administrator managing a suite of shebanged third party 
scripts for their organization should be able to change the mux'd 
default.  As much as I hate introducing a configuration file, I think 
that's going to be appropriate here.  That would allow:

* A system admin to select the system-wide default Python version, 
assuming the platform provides the option of different versions;

* Individual scripts can make their own selection, overriding the system 
defaults;

* Individual users can override the script selection by setting an 
environment variable.

* A user at the command line could also provide a switch to make the 
selection.

Cheers,
-Barry



From ncoghlan at gmail.com  Fri Aug  4 08:36:32 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 4 Aug 2017 22:36:32 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <37e7287e-9ba6-f64f-9bba-3461e1b4e97b@gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
 <37e7287e-9ba6-f64f-9bba-3461e1b4e97b@gmail.com>
Message-ID: <CADiSq7fq78+Avvs3M9R4uEkPJ9pSjmuVGLCyzrNFTB=Qek=aig@mail.gmail.com>

On 3 August 2017 at 00:41, Petr Viktorin <encukou at gmail.com> wrote:
> On 08/02/2017 04:29 PM, Nick Coghlan wrote:
>> While I'm happy I at least brought it up, I still can't convince
>> myself to actually *like* this idea relative to the "compile it and
>> report on the results" approach combined with the ability to opt-in to
>> replacing that default script with a symlink to the desired version.
>
> Just compiling won't work -- consider `import pathlib` for example.

I'm aiming to separate "definitely won't work" (i.e. won't even
complile) from "might work" (i.e. there are other reasons things can
fail at runtime)

> This is UX, but for sysadmins (and other command-line users). Rather than
> trying to do what they mean, we should aim to synchronize their expectations
> and the reality.
> I'd rather have /usr/bin/python failing hard with a good error message than
> trying non-obvious heuristics. Avoid the temptation to guess.

Exactly. However, *compiling* is something we can do quickly *before*
reporting that error message.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Aug  4 09:22:10 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 4 Aug 2017 23:22:10 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <olss9t$u69$1@blaine.gmane.org>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
 <olss9t$u69$1@blaine.gmane.org>
Message-ID: <CADiSq7dzo7jNSAxiG6He3cHhgiYZuVvPXon_pv6zZ1RAsAuGMA@mail.gmail.com>

On 3 August 2017 at 01:49, Barry Warsaw <barry at python.org> wrote:
> Now, a system administrator managing a suite of shebanged third party
> scripts for their organization should be able to change the mux'd default.
> As much as I hate introducing a configuration file, I think that's going to
> be appropriate here.  That would allow:
>
> * A system admin to select the system-wide default Python version, assuming
> the platform provides the option of different versions;
>
> * Individual scripts can make their own selection, overriding the system
> defaults;
>
> * Individual users can override the script selection by setting an
> environment variable.
>
> * A user at the command line could also provide a switch to make the
> selection.

The reason I don't think the "pyversion" comment will help in practice
is that there's only 3 compatibility states that we really care about
for this particular problem, and they're all already expressible
directly in the shebang line regardless of whether the target system
uses pythonmux or not:

* Python 2 only ("python2")
* Python 3 only ("python3")
* hybrid subset ("python")

As a result, the key remaining UX problem is how server admins want
scripts nominally in the hybrid subset to be handled, where there are
three main options that make sense (depending on the state of their
infrastructure, the amount of legacy Py2 scripts they still need to
cope with, and their level of confidence in debugging
"Py2-code-on-Py3" exceptions):

- always Py2 (status quo)
- always Py3 (desired future state)
- just let it fail and require that the Python version to use be
specified at runtime based on out-of-band compatibility info, instead
of relying on the shebang line (transition state)

If a particular script genuinely needs 2.x, then python2 is ubiquitous
enough now to just change the shebang line. If it genuinely needs Py3,
then it will typically *already* have "python3" in the shebang line,
due to "python" meaning Python 2.x by default on most current systems.

That means the required system level settings are still simple enough
to be handled as a symlink, rather than needing to be a config file.

Adding pythonmux and pyversion into the mix thus doesn't gain you much
except additional maintenance complexity, since people still can't
rely on it: there will still be too many systems where "python" will
either be with missing entirely or an unqualified reference to Py2 or
Py3. A new comment format unfortunately also doesn't even help
distinguish between "legacy Py2 only script" and "script that has been
updated to the hybrid subset of 2 & 3", since there are plenty of the
latter already in existence that don't have, and probably won't gain,
any new special comment syntax.

Where the idea of a "try to compile it as Py3 to see if that works,
but report an error anyway" script then comes in is as a potentially
better failure UX than the default "interpreter not found" from the
operating system's shebang processing.

Cheers,
Nick.

P.S. All that said, I'm not necessarily opposed to adding some form of
"Tested against" magic comment to Python scripts in order to help
distros provide a better UX when a only-tested-on-a-fast-moving distro
script fails on an older LTS distro (and the author of the script
remembered to provide the new embedded metadata). I just don't think
we should couple that discussion to the discussion of what we want to
do to provide a better UX when handling scripts that *lack* such a
marker *and* use an absolute path to an unqualified "python" command
in their shebang line.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Fri Aug  4 10:27:39 2017
From: barry at python.org (Barry Warsaw)
Date: Fri, 4 Aug 2017 10:27:39 -0400
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <CADiSq7dzo7jNSAxiG6He3cHhgiYZuVvPXon_pv6zZ1RAsAuGMA@mail.gmail.com>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
 <olss9t$u69$1@blaine.gmane.org>
 <CADiSq7dzo7jNSAxiG6He3cHhgiYZuVvPXon_pv6zZ1RAsAuGMA@mail.gmail.com>
Message-ID: <6097D839-9CB3-4FF4-8B57-5585CB063FD8@python.org>

On Aug 4, 2017, at 09:22, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Where the idea of a "try to compile it as Py3 to see if that works,
> but report an error anyway" script then comes in is as a potentially
> better failure UX than the default "interpreter not found" from the
> operating system's shebang processing.

This makes me uncomfortable, because I think there will be cases, especially with smaller, one-off scripts, where compilation succeeds but where the code still has subtle failure modes at run time.  And for larger applications, doing a pre-compile isn?t going to help start up time.  Plus, you have to disable pyc generation or ensure that if the compilation fails, all the artifacts of that are cleaned up.  It seems like a lot of complexity with plenty of opportunities for failure and false matches.

The Zen of Python just begs to be quoted. :)

Maybe muxing won?t be a useful technique in practice, but in that case I think a short term ?no /usr/bin/python? is a better transition step.  I agree that we should be actively recommending Python 2-only scripts be shebanged to /usr/bin/python2.  When Python 2.7 is EOL?d, one distro release without /usr/bin/python would be a pretty clear indication that you need to fix your scripts, but PEP 394 should give distros the leeway to choose their own timing on that.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20170804/a0a686bb/attachment.sig>

From ncoghlan at gmail.com  Sun Aug  6 22:33:27 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 7 Aug 2017 12:33:27 +1000
Subject: [Linux-SIG] Revisit of PEP 394 -- The "python" Command on
 Unix-Like Systems
In-Reply-To: <6097D839-9CB3-4FF4-8B57-5585CB063FD8@python.org>
References: <718257630.37973746.1500984074624.JavaMail.zimbra@redhat.com>
 <1481944193.37973848.1500984124214.JavaMail.zimbra@redhat.com>
 <CADiSq7eDytm=YFKC2kAWq9_7i6kmhbTPMPzgUOGKdc1LvwDepg@mail.gmail.com>
 <B2304DB9-6645-4D5F-829B-FF8BD03F12B4@python.org>
 <CADiSq7fc6YJWKrhMS5zD9gf4_WbxGw0vRNDS2WjPWYn7W0=kfQ@mail.gmail.com>
 <67D83649-EDCA-4BBD-B980-78A4642BCB23@python.org>
 <CADiSq7eayL0C+_FVjbB5hHfBKxKsqe_C2VQNg9q193zUkaSmsA@mail.gmail.com>
 <CADiSq7dPFjDFXDrp99obHp5PKaa6AnoWx_Vq7fQ0sKoQ3+5M0w@mail.gmail.com>
 <olss9t$u69$1@blaine.gmane.org>
 <CADiSq7dzo7jNSAxiG6He3cHhgiYZuVvPXon_pv6zZ1RAsAuGMA@mail.gmail.com>
 <6097D839-9CB3-4FF4-8B57-5585CB063FD8@python.org>
Message-ID: <CADiSq7d1vdZgBbnQ28Zm3xGkM9zv1pet1X=k+L+Z=VODPpTuMQ@mail.gmail.com>

On 5 August 2017 at 00:27, Barry Warsaw <barry at python.org> wrote:
> On Aug 4, 2017, at 09:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Where the idea of a "try to compile it as Py3 to see if that works,
>> but report an error anyway" script then comes in is as a potentially
>> better failure UX than the default "interpreter not found" from the
>> operating system's shebang processing.
>
> This makes me uncomfortable, because I think there will be cases, especially with smaller, one-off scripts, where compilation succeeds but where the code still has subtle failure modes at run time.  And for larger applications, doing a pre-compile isn?t going to help start up time.  Plus, you have to disable pyc generation or ensure that if the compilation fails, all the artifacts of that are cleaned up.  It seems like a lot of complexity with plenty of opportunities for failure and false matches.
>
> The Zen of Python just begs to be quoted. :)

A colleague also pointed out that it's much easier to *add* this kind
of heuristic based advice later than it is to take it away. So
consider me convinced that while a custom error message may be
appropriate, that message should be independent of the specific script
supplied (however, it may still take into account which Python stacks
are already installed on the system, and should *definitely* take into
account which stacks are available for installation on that particular
distro)

> Maybe muxing won?t be a useful technique in practice, but in that case I think a short term ?no /usr/bin/python? is a better transition step.

The only reason I moved away from that stance is because the shebang
handler doesn't give very good error messages when it doesn't find the
specified interpreter. However, it's the same idea in principle (i.e.
an unqualified shebang reference will fail by default, and then it's
up to the system administrator to decide whether it should be
reconfigured to do something else).

>  I agree that we should be actively recommending Python 2-only scripts be shebanged to /usr/bin/python2.  When Python 2.7 is EOL?d, one distro release without /usr/bin/python would be a pretty clear indication that you need to fix your scripts, but PEP 394 should give distros the leeway to choose their own timing on that.

Yeah, Fedora has effectively been in this state since F23: by default,
/usr/bin/python just fails, since the Python 2 stack is missing by
default.

The state I'm now aiming to get to is to have that failure offer a few
distinct options:

1. Change nothing, "/usr/bin/python" and "python" outside a venv of
some kind just fails
2. Change the shebang line to use the installed-by-default Py3 stack
3. Reconfigure the system to always use the Py3 stack by default
4. Install the Py2 stack, change the shebang line to use it explicitly
5. Install the Py2 stack, reconfigure the system to use it by default

Right now, the available options in F26 are the first two and the last
two, so the main thing I'm trying to figure out is how we can feasibly
offer the middle one as an opt-in admin-controlled feature.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia