From ncoghlan at gmail.com Wed Oct 7 05:28:47 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Oct 2015 13:28:47 +1000 Subject: [Linux-SIG] Introducing a Python launcher for *nix? Message-ID: The Python Launcher for Windows is a nice feature, but it's difficult for folks that use Linux or Mac OS X themselves to recommend it, as the "py" shorthand only works on Windows. What do folks think of the idea of introducing a "Python Launcher for *nix" using the following iterative development model: 1. We update the Python 3 *nix builds to provide a "py" symlink by default. This would be the baseline state for Python 3.6 if the next step isn't completed in time. 2. The symlink is replaced with a custom launcher that works like the Python 3.5 version of the Windows launcher: * accepts "-3", "-2", etc command line arguments like the Windows launcher * checks for an active virtualenv, if it finds one, uses the venv's default Python * checks for a user specific config file * checks for a global config file (This could potentially be achieved by making the existing launcher cross-platform, rather than having two independent launchers to maintain) Cheers, Nick. P.S. This list may not have enough subscribers yet to have a good discussion, so I'll post another thread about advertising, and then we can reboot this thread later if needed. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Oct 7 05:38:55 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Oct 2015 13:38:55 +1000 Subject: [Linux-SIG] Announcing/advertising the Python Linux SIG Message-ID: Hi folks, Where should we advertise for potential SIG members? My suggestions for the Python community side: * python-dev * python-ideas * distutils-sig * conda mailing list [1] And for the slice of downstream of most interest to me personally: * Fedora Python SIG [2] * Red Hat's internal Python list * softwarecollections.org mailing list [3] * Project Atomic mailing list [4] [1] https://groups.google.com/a/continuum.io/forum/#!forum/conda [2] https://lists.fedoraproject.org/mailman/listinfo/python-devel [3] https://www.redhat.com/mailman/listinfo/sclorg [4] https://lists.projectatomic.io/mailman/listinfo/atomic-devel -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Wed Oct 7 15:56:35 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 09:56:35 -0400 Subject: [Linux-SIG] Announcing/advertising the Python Linux SIG In-Reply-To: References: Message-ID: <20151007095635.4b710c30@limelight.wooz.org> On Oct 07, 2015, at 01:38 PM, Nick Coghlan wrote: >Where should we advertise for potential SIG members? I just posted a message to the debian-python@ mailing list. I will post something to ubuntu-devel@ momentarily. FWIW, I also just requested gmane.comp.python.linux-sig All of your other suggestions sound good too. Go for it! Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 17:52:02 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 11:52:02 -0400 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: References: Message-ID: <20151007115202.191d05ca@limelight.wooz.org> [Quoting out of order] >P.S. This list may not have enough subscribers yet to have a good >discussion, so I'll post another thread about advertising, and then we >can reboot this thread later if needed. Agreed. On Oct 07, 2015, at 01:28 PM, Nick Coghlan wrote: >What do folks think of the idea of introducing a "Python Launcher for >*nix" using the following iterative development model: > >1. We update the Python 3 *nix builds to provide a "py" symlink by >default. This would be the baseline state for Python 3.6 if the next >step isn't completed in time. > >2. The symlink is replaced with a custom launcher that works like the >Python 3.5 version of the Windows launcher: > >* accepts "-3", "-2", etc command line arguments like the Windows launcher >* checks for an active virtualenv, if it finds one, uses the venv's >default Python >* checks for a user specific config file >* checks for a global config file I like it, although I would also like to support major.minor specifications. For example, in Ubuntu 15.10 we'll have two supported Python 3 versions, 3.4 and 3.5. We do have /usr/bin/python3 pointing to the default version, i.e. /usr/bin/python3.4 and that changes as we roll over to the new default version (it'll be 3.5 only for the next LTS). So `py -3` is unambiguous but folks might want to use `py -3.4` which kind of looks ugly. Maybe the launcher should take an optional mutually exclusive argument such as -i/--interpreter which would either give a shorthand, like 3.4 or a full path to the interpreter to choose. That would let people install say 3.6 in /usr/local and then they could `py -i /usr/local/bin/python3.6` (Is that a win over just using /usr/local/bin/python3.6? Not really, but I can see the launcher being scriptable so you'd want to allow such scripts to substitute the exact interpreter they want to use. OTOH YAGNI.) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 18:23:04 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 12:23:04 -0400 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: References: Message-ID: <20151007122304.2a99865a@limelight.wooz.org> [Reposting for the Gmane newsgroup] [Quoting out of order] >P.S. This list may not have enough subscribers yet to have a good >discussion, so I'll post another thread about advertising, and then we >can reboot this thread later if needed. Agreed. On Oct 07, 2015, at 01:28 PM, Nick Coghlan wrote: >What do folks think of the idea of introducing a "Python Launcher for >*nix" using the following iterative development model: > >1. We update the Python 3 *nix builds to provide a "py" symlink by >default. This would be the baseline state for Python 3.6 if the next >step isn't completed in time. > >2. The symlink is replaced with a custom launcher that works like the >Python 3.5 version of the Windows launcher: > >* accepts "-3", "-2", etc command line arguments like the Windows launcher >* checks for an active virtualenv, if it finds one, uses the venv's >default Python >* checks for a user specific config file >* checks for a global config file I like it, although I would also like to support major.minor specifications. For example, in Ubuntu 15.10 we'll have two supported Python 3 versions, 3.4 and 3.5. We do have /usr/bin/python3 pointing to the default version, i.e. /usr/bin/python3.4 and that changes as we roll over to the new default version (it'll be 3.5 only for the next LTS). So `py -3` is unambiguous but folks might want to use `py -3.4` which kind of looks ugly. Maybe the launcher should take an optional mutually exclusive argument such as -i/--interpreter which would either give a shorthand, like 3.4 or a full path to the interpreter to choose. That would let people install say 3.6 in /usr/local and then they could `py -i /usr/local/bin/python3.6` (Is that a win over just using /usr/local/bin/python3.6? Not really, but I can see the launcher being scriptable so you'd want to allow such scripts to substitute the exact interpreter they want to use. OTOH YAGNI.) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ozarow at gmail.com Wed Oct 7 22:17:49 2015 From: ozarow at gmail.com (Piotr Ozarowski) Date: Wed, 7 Oct 2015 22:17:49 +0200 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: <20151007115202.191d05ca@limelight.wooz.org> References: <20151007115202.191d05ca@limelight.wooz.org> Message-ID: <20151007201749.GC3938@p1otr.com> I wanted to reply with some comments, but then I realized I don't know what's the purpose of this script. Why should one use it instead of `env python` or python or python3 or any pythonX.Y? If it's about letting users invoke any Python available (is pypy included here, BTW?), then why do you need -2 or -3 or worse: -X.Y? If it's meant to be used by developers, then they know much better than us which interpreter they want. If it's meant for administrators or shebangs, then HELL NO! -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From fred at fdrake.net Thu Oct 8 00:07:45 2015 From: fred at fdrake.net (Fred Drake) Date: Wed, 7 Oct 2015 18:07:45 -0400 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: <20151007201749.GC3938@p1otr.com> References: <20151007115202.191d05ca@limelight.wooz.org> <20151007201749.GC3938@p1otr.com> Message-ID: On Wed, Oct 7, 2015 at 4:17 PM, Piotr Ozarowski wrote: > If it's about letting users invoke any Python available (is pypy > included here, BTW?), then why do you need -2 or -3 or worse: -X.Y? I'm not sure what the original motivation for this proposal is; it appears to have been on a different list. The notion that a script is going to figure out which Python is the right one just seems pretty strange to me. When developing an application, I know exactly which Python I want to use, and build that application with that Python to begin with. Any user-facing scripts get the right sh-bang line to begin with. If I'm building a library, I'm going to do things with several versions (tests can be driven with tox, for example), but library users are going to know what version of Python their application calls for, and they'll use it. I may be missing something pretty obvious to many, though. My approaches to building an application don't appear to be commonly applied in Python environments. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From fred at fdrake.net Thu Oct 8 02:42:57 2015 From: fred at fdrake.net (Fred Drake) Date: Wed, 7 Oct 2015 20:42:57 -0400 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: References: <20151007115202.191d05ca@limelight.wooz.org> <20151007201749.GC3938@p1otr.com> Message-ID: On Wed, Oct 7, 2015 at 7:55 PM, Nick Coghlan wrote: > The first motive is consistent cross platform invocation - due to the way > Python invocation on Windows is handled, commands like "python3 -m pip" and > "python3.4 -m pip" don't work, and never will, and the differences in file > system layouts exclude the use of absolute paths. Thanks for explaining, Nick! I agree that's a very real group of use-cases, and this sounds like the right kind of approach for that. Since I'm not in those particular trenches (or the Windows miasma), I probably won't have a clue regarding the details of what these users need. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From ncoghlan at gmail.com Thu Oct 8 01:55:24 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Oct 2015 09:55:24 +1000 Subject: [Linux-SIG] Introducing a Python launcher for *nix? In-Reply-To: References: <20151007115202.191d05ca@limelight.wooz.org> <20151007201749.GC3938@p1otr.com> Message-ID: On 8 Oct 2015 08:09, "Fred Drake" wrote: > > On Wed, Oct 7, 2015 at 4:17 PM, Piotr Ozarowski wrote: > > If it's about letting users invoke any Python available (is pypy > > included here, BTW?), then why do you need -2 or -3 or worse: -X.Y? > > I'm not sure what the original motivation for this proposal is; it > appears to have been on a different list. The first motive is consistent cross platform invocation - due to the way Python invocation on Windows is handled, commands like "python3 -m pip" and "python3.4 -m pip" don't work, and never will, and the differences in file system layouts exclude the use of absolute paths. The Python launcher for Windows *does* exist, though, so introducing a "py" script on the *nix side of things offers the opportunity to provide a truly cross platform Python experience modelled on the way that works: https://docs.python.org/3/using/windows.html#python-launcher-for-windows The second motive is to make it easy for individual users to switch to an alternative implementation like PyPy as their default interactive shell without impacting shebang lines and other automated invocations. I also want to allow users to make that switch without needing admin rights on their machine. The third motive is simply to make -m invocations shorter ("py -m pip" rather than "python -m pip"). > > The notion that a script is going to figure out which Python is the > right one just seems pretty strange to me. When developing an > application, I know exactly which Python I want to use, and build that > application with that Python to begin with. You're already a professional developer, though, rather than someone learning about the command line and filesystems as part of learning to program in Python. > Any user-facing scripts > get the right sh-bang line to begin with. If I'm building a library, > I'm going to do things with several versions (tests can be driven with > tox, for example), but library users are going to know what version of > Python their application calls for, and they'll use it. No, they don't necessarily know this. If folks don't want to take my word for that, volunteering as a helper at a Software Carpentry bootcamp, a Python for Young Coders workshop or another community event that introduces children or adults to Python programming highlights just how radically wrong these kinds of assumptions are in the context of people learning to program, especially when instructors are needing to cope with a mixed audience of Windows, OS X, and Linux users. > I may be missing something pretty obvious to many, though. My > approaches to building an application don't appear to be commonly > applied in Python environments. Yeah, I failed to convey that the main intended beneficiaries are future newcomers to Python, and the folks that are themselves using Linux or OS X, but need to provide instructions that also work on Windows. Cheers, Nick. > > > -Fred > > -- > Fred L. Drake, Jr. > "A storm broke loose in my mind." --Albert Einstein > _______________________________________________ > Linux-sig mailing list > Linux-sig at python.org > https://mail.python.org/mailman/listinfo/linux-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Oct 8 18:22:32 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Oct 2015 12:22:32 -0400 Subject: [Linux-SIG] PEP 476 and backward incompatibilities for distros Message-ID: <20151008122232.59732eee@limelight.wooz.org> And now for something completely different... In Ubuntu, as probably with other distros, we have long-term support (LTS) releases. We have a process called stable release upgrades (SRU) to allow us to systematically update packages in existing, supported releases and we[*] try to keep LTS versions of Python current with upstream. The policy of breaking backward compatibility, even for security issues, can cause us all kinds of pain. I know these types of changes are controversial even upstream, but they are very problematic for distros. There's even talk about disallowing point updates for Python in LTS releases because of this. That does our users a disservice because they do want all the other great bug fixes in point releases. A current example is captured in this bug: https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768 which is related to PEP 476 https://www.python.org/dev/peps/pep-0476/ Sadly, I did not follow the discussion on PEP 476 at the time, and I won't argue in this mailing list whether it was a good idea or not. Here I want to talk about making such changes friendlier to downstream consumers such as Ubuntu. The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded. This backward incompatible change broke several other packages and user complaints started coming in. We eventually reverted the upload, but not until after lots of people probably upgraded to get it. So now their systems are broken. I know that urllib3/requests upstream was updated to work with this change, but that is problematic for SRUs because we don't want to just backport the latest versions --with new APIs-- of those into the trusty-updates channel. Those new versions may themselves cause cascading incompatibilities. We could try to cherry pick the fixes in those packages, but that's a lot of additional work, and it doesn't help any other breakages caused by PEP 476. Python need to do a better job of not breaking backward compatibility, or making it easy to identify and omit such compatibility breaking changes when pulled in by downstreams. For example, Misc/NEWS says only this in the What's new in 3.4.3. section: - Issue #22417: Verify certificates by default in httplib (PEP 476). That's way too easy to miss, doesn't include any of the backward compatibility break implications, and doesn't really provide much help in narrowing down the commit or commits related to this. The workarounds described in the PEP's Opting Out section isn't really useful either. If we applied this to the Ubuntu version, how would people know what to do to opt back in? These kinds of incompatible changes in Python stable releases give Python a bad reputation, and make folks very wary about just adopting such any maintenance release. That's not a good situation to be in when your users *also* want the latest updates to get other, non breaking fixes. It's a trade-off for sure. It's not necessarily a bad thing to want to improve your user's default security. It *might* be a bad thing to do that in stable releases. It's *definitely* a bad thing to do it in a way that breaks backward compatibility, or is not very clearly marked and easily ignored by distros which need to make different decisions. Cheers, -Barry [*] well Matthias does a considerable amount of the actual work. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Thu Oct 8 20:24:36 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Oct 2015 04:24:36 +1000 Subject: [Linux-SIG] PEP 476 and backward incompatibilities for distros In-Reply-To: <20151008122232.59732eee@limelight.wooz.org> References: <20151008122232.59732eee@limelight.wooz.org> Message-ID: On 9 Oct 2015 02:29, "Barry Warsaw" wrote: > > And now for something completely different... > > In Ubuntu, as probably with other distros, we have long-term support (LTS) > releases. We have a process called stable release upgrades (SRU) to allow us > to systematically update packages in existing, supported releases and we[*] > try to keep LTS versions of Python current with upstream. The policy of > breaking backward compatibility, even for security issues, can cause us all > kinds of pain. I know these types of changes are controversial even upstream, > but they are very problematic for distros. There's even talk about > disallowing point updates for Python in LTS releases because of this. This is the RHEL/CentOS backports-only model for long term support, and I think it's the only way to offer good commercial stability guarantees without placing an undue maintenance burden on community volunteers. Rebasing in long term support releases should be the exception rather than the rule. > That > does our users a disservice because they do want all the other great bug fixes > in point releases. This is a classic case of a conflict of interest between commercial redistributors and the upstream community. For upstream, working on PEP 466 persuaded me that the priority needs to be on providing safe defaults in order to minimise risks for new users, and for insufficiently paranoid existing users. The burden then falls on commercial redistributors to effectively manage backwards incompatible changes on behalf of our customers, rather than expecting upstream to prioritise our interests over those of the rest of the community. > A current example is captured in this bug: > > https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768 > > which is related to PEP 476 > > https://www.python.org/dev/peps/pep-0476/ > > Sadly, I did not follow the discussion on PEP 476 at the time, and I won't > argue in this mailing list whether it was a good idea or not. Here I want to > talk about making such changes friendlier to downstream consumers such as > Ubuntu. > > The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded. This > backward incompatible change broke several other packages and user complaints > started coming in. We eventually reverted the upload, but not until after > lots of people probably upgraded to get it. So now their systems are broken. The model we've adopted to deal with this on the Fedora/RHEL/CentOS side of things is: * Fedora will break things sometimes. If that's not acceptable to a given user, then Fedora probably isn't the distro for them * RHEL and (by extension) CentOS, are far more selective about the updates they bring in. This is more expensive to maintain on the redistributor side (since a lot of changes need to be backported to earlier upstream base releases), but means there's an additional layer of review and change management around potentially breaking changes, so we don't need to argue against improvements to default behaviour upstream. * Software Collections then make faster moving rebases readily available atop the stable RHEL/CentOS layer. They're more stable than Fedora, but also have major releases (with potential compatibility breaks) more often than RHEL. > I know that urllib3/requests upstream was updated to work with this change, > but that is problematic for SRUs because we don't want to just backport the > latest versions --with new APIs-- of those into the trusty-updates channel. > Those new versions may themselves cause cascading incompatibilities. We could > try to cherry pick the fixes in those packages, but that's a lot of additional > work, and it doesn't help any other breakages caused by PEP 476. Providing a better migration path for cases where commercial support obligations apply is what the proposed PEP 493 recommendations are about: https://www.python.org/dev/peps/pep-0493/ > Python need to do a better job of not breaking backward compatibility, or > making it easy to identify and omit such compatibility breaking changes when > pulled in by downstreams. s/Python/vendors commercially redistributing Python/ > For example, Misc/NEWS says only this in the What's > new in 3.4.3. section: > > - Issue #22417: Verify certificates by default in httplib (PEP 476). > > That's way too easy to miss, doesn't include any of the backward compatibility > break implications, and doesn't really provide much help in narrowing down the > commit or commits related to this. The workarounds described in the PEP's > Opting Out section isn't really useful either. If we applied this to the > Ubuntu version, how would people know what to do to opt back in? This is on redistributors to handle - if we want more or different documentation in cases like these, then we need to be paying people to spend more time on representing our interests (and those of our users) in CPython upstream development, rather than focusing our efforts solely on integration work for our respective operating systems. > These kinds of incompatible changes in Python stable releases give Python a > bad reputation, and make folks very wary about just adopting such any > maintenance release. That's not a good situation to be in when your users > *also* want the latest updates to get other, non breaking fixes. The perpetuation of bad default behaviours in the name of backwards compatibility is a large part of what turned our industry into the insecure user hostile mess that it is today, though. > It's a trade-off for sure. It's not necessarily a bad thing to want to > improve your user's default security. It *might* be a bad thing to do that in > stable releases. It's *definitely* a bad thing to do it in a way that breaks > backward compatibility, or is not very clearly marked and easily ignored by > distros which need to make different decisions. My own focus for these changes has been on Python 2.7, and the changes are far more clearly marked there: https://docs.python.org/2/whatsnew/2.7.html#py27-maintenance-enhancements I never looked at how the transition was handled for 3.4 myself (as I was mainly concerned about the impact on RHEL & CentOS, rather than the Python 3 stacks in Fedora & SCLs) Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Oct 9 00:27:02 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Oct 2015 18:27:02 -0400 Subject: [Linux-SIG] PEP 476 and backward incompatibilities for distros References: <20151008122232.59732eee@limelight.wooz.org> Message-ID: <20151008182702.6e503171@anarchist.wooz.org> On Oct 09, 2015, at 04:24 AM, Nick Coghlan wrote: >For upstream, working on PEP 466 persuaded me that the priority needs to be >on providing safe defaults in order to minimise risks for new users, and >for insufficiently paranoid existing users. 2.7 is a special case because there will never be a 2.8. I think it's a different calculus for Python 3. If we hadn't broken backward compatibility in 3.4.3 folks would have eventually gotten the change in 3.5. That doesn't eliminate the cost on developers for the compatibility break, but it does defer it until they do general 3.5 compatibility work. I'm also not convinced that the good PR we might gain from providing users with a more secure default offsets the PR hit we get for breaking compatibility in point releases. It also gives me a vague anti-we're-all-adults feeling. >The burden then falls on commercial redistributors to effectively manage >backwards incompatible changes on behalf of our customers, rather than >expecting upstream to prioritise our interests over those of the rest of >the community. "Effectively managing" could very well mean backing out the incompatible change. But as my larger point goes, that's may not actually be easy to do. If we're going to break backward compatibility in point releases, we need to advertise that loudly, make those changes auditable, and give people an easier way to return to the previous compatible if insecure default. Not all uses of the default are by naive users who need our help to keep all their bases off the intarwebs. It's also not just downstream distros that could be adversely affected by such changes. >Providing a better migration path for cases where commercial support >obligations apply is what the proposed PEP 493 recommendations are about: >https://www.python.org/dev/peps/pep-0493/ The other thing to keep in mind is that distros generally have a team dedicated to security issues. Was this change in response to a CVE? Certainly those teams are better able to judge the implications for the insecure compatible default for users and applications on their platform. I don't envy the work they do to balance those backward compatibility issues against increased security. A change here by upstream is an end-around of those team's responsibilities. Which might still mean it's the right decision for *Python* (and we can argue that on python-dev) but it behooves us to make that very obvious for downstream consumers who come out on the other side of that equation. >This is on redistributors to handle - if we want more or different >documentation in cases like these, then we need to be paying people to >spend more time on representing our interests (and those of our users) in >CPython upstream development, rather than focusing our efforts solely on >integration work for our respective operating systems. No, because as I said it doesn't only affect distros with paid employees. Maybe it affects distros that are purely volunteer run. Maybe it affects people installing Python from source for their own internal usage. Etc. Upstream is who knows that a compatibility break is coming in a point release. We owe it to our users to first, be very careful about that, and second to make sure that information is very well advertised. >The perpetuation of bad default behaviours in the name of backwards >compatibility is a large part of what turned our industry into the insecure >user hostile mess that it is today, though. So, we're going to have to break compatibility at some point to fix it. The question is where and when? Buried in a point release? Wait until the next major release? In documentation, a tech note, or announcement? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Fri Oct 9 09:22:27 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Oct 2015 17:22:27 +1000 Subject: [Linux-SIG] PEP 476 and backward incompatibilities for distros In-Reply-To: <20151008182702.6e503171@anarchist.wooz.org> References: <20151008122232.59732eee@limelight.wooz.org> <20151008182702.6e503171@anarchist.wooz.org> Message-ID: On 9 October 2015 at 08:27, Barry Warsaw wrote: > On Oct 09, 2015, at 04:24 AM, Nick Coghlan wrote: >>Providing a better migration path for cases where commercial support >>obligations apply is what the proposed PEP 493 recommendations are about: >>https://www.python.org/dev/peps/pep-0493/ > > The other thing to keep in mind is that distros generally have a team > dedicated to security issues. Was this change in response to a CVE? While we made the change first, it eventually became CVE-2014-9365: https://bugzilla.redhat.com/show_bug.cgi?id=1173041 Alex made the request: http://seclists.org/oss-sec/2014/q4/1022 The original escalation that led to my writing PEP 466 to upgrade 2.7's network security features and Alex writing PEP 476 came from Rackspace due to security problems encountered in OpenStack development. > Certainly those teams are better able to judge the implications for the > insecure compatible default for users and applications on their platform. I > don't envy the work they do to balance those backward compatibility issues > against increased security. Yeah, Red Hat's SRT have been some of the main reviewers for the proposals in PEP 493 as we've tried to figure out how best to handle PEP 476 in RHEL > A change here by upstream is an end-around of those team's responsibilities. > Which might still mean it's the right decision for *Python* (and we can argue > that on python-dev) but it behooves us to make that very obvious for > downstream consumers who come out on the other side of that equation. Alex wrote PEP 476 in August 2014, Guido approved it in October [1], Python 2.7.9 was released and the CVE was issued in December, and then Python 3.4.3 was released in February 2015. A new section was also added to the 3.4 What's New document: https://docs.python.org/dev/whatsnew/3.4.html#changed-in-3-4-3 [1] https://mail.python.org/pipermail/python-dev/2014-October/136676.html >>This is on redistributors to handle - if we want more or different >>documentation in cases like these, then we need to be paying people to >>spend more time on representing our interests (and those of our users) in >>CPython upstream development, rather than focusing our efforts solely on >>integration work for our respective operating systems. > > No, because as I said it doesn't only affect distros with paid employees. > Maybe it affects distros that are purely volunteer run. Maybe it affects > people installing Python from source for their own internal usage. Etc. It's my view that folks running open source community projects directly have no legitimate expectation of interface stability unless they're running the free variant of a commercially supported product with strong stability guarantees (e.g. Ubuntu LTS, CentOS/RHEL, openSUSE/SLES). I've become much, much harsher about this in recent years as I've come to understand the degree to which most organisations take open source developers for granted (my perspective had previously been skewed by Boeing's approach, where Supplier Management required that we first evaluate the project's sustaining engineering model before allowing us to incorporate a new open source project into our designs. I assumed every commercial operation consumed open source that way, but I was wrong - most orgs instead seem to go straight from the "No open source allowed" model to the "Magic Internet Pixies" approach, where they'll happily build business critical software atop dependencies where they have no idea who is maintaining them or why they're working on them) > Upstream is who knows that a compatibility break is coming in a point > release. We owe it to our users to first, be very careful about that, and > second to make sure that information is very well advertised. We did all that, though. If folks missed the PEP *and* the CVE *and* the What's New update *and* the NEWS entry *and* the voluminous mailing list threads... One place we did miss was the download pages for 3.4.3, but does anyone in the distros actually read those rather than the NEWS file? >>The perpetuation of bad default behaviours in the name of backwards >>compatibility is a large part of what turned our industry into the insecure >>user hostile mess that it is today, though. > > So, we're going to have to break compatibility at some point to fix it. The > question is where and when? Buried in a point release? Wait until the next > major release? In documentation, a tech note, or announcement? While we unfortunately didn't capture it in the PEP, the main benefit of updating 3.4.3 rather than waiting until 3.5.0 was released was to avoid creating a situation where folks suffered a security regression in upgrading from 2.7.9+ to 3.4.3+. Making the change in both versions kept that regression window to less than 3 months for folks consuming upstream directly, while leaving downstreams to figure out how to incorporate the change into their Python 2 & 3 stacks in a less disruptive way. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia