From ncoghlan at gmail.com Mon Sep 1 00:00:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 08:00:14 +1000 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 Message-ID: Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 maintenance release in addition to including it with Python 3.4. That part of the proposal proved to be controversial, so we dropped it from the original PEP in order to focus on meeting the Python 3.4 specific release deadlines. This also had the benefit of working out the kinks in the bootstrapping processing as part of the Python 3.4 release cycle. However, we still think we should start providing pip by default to Python 2.7 users as well, at least as part of the Windows and Mac OS X installers. One notable difference from PEP 453 is that because there is no venv module in 2.7, and hence no integration between venv and ensurepip, we can give redistributors the option of just disabling ensurepip entirely and redirecting users to platform specific installation tools. Regards, Nick. ====================== PEP: 477 Title: Backport ensurepip (PEP 453) to Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft Nick Coghlan Status: Active Type: Process Content-Type: text/x-rst Created: 26-Aug-2014 Post-History: 1-Sep-2014 Abstract ======== This PEP proposes that the ``ensurepip`` module, added to Python 3.4 by PEP 453, be backported to Python 2.7. It also proposes that automatic invocation of ``ensurepip`` be added to the Python 2.7 Windows and OSX installers. However it does **not** propose that automatic invocation be added to the ``Makefile``. It also proposes that the documentation changes for the package distribution and installation guides be updated to match that in 3.4, which references using the ``ensurepip`` module to bootstrap the installer. Rationale ========= Python 2.7 is effectively a LTS release of Python which represents the end of the 2.x series and there is still a very large contingent of users whom are still using Python 2.7 as their primary version. These users, in order to participate in the wider Python ecosystem, must manually attempt to go out and find the correct way to bootstrap the packaging tools. It is the opinion of this PEP that making it as easy as possible for end users to participate in the wider Python ecosystem is important for 3 primary reasons: 1. The Python 2.x to 3.x migration has a number of painpoints that are eased by a number of third party modules such as six [#six]_, modernize [#modernize]_, or future [#future]_. However relying on these tools requires that everyone who uses the project have a tool to install these packages. 2. In addition to tooling to aid in migration from Python 2.x to 3.x, there are also a number of modules that are *new* in Python 3 for which there are backports available on PyPI. This can also aid in the ability for people to write 2.x and 3.x compatible software as well as enable them to use some of the newer features of Python 3 on Python 2. 3. Users also will need a number of tools in order to create python packages that conform to the newer standards that are being proposed. Things like setuptools [#setuptools]_, Wheel [#wheel]_, and twine [#twine]_ are enabling a safer, faster, and more reliable packaging tool chain. These tools can be difficult for people to use if first they must be told how to go out and install the package manager. 4. One of Pythons biggest strengths is in the huge ecosystem of libraries and projects that have been built on top of it, most of which are distributed through PyPI. However in order to benefit from this wide ecosystem meaningfully requires end users, some of which are going to be new, to make a decision on which package manager they should get, how to get it, and then finally actually installing it first. Furthermore, alternative implementations of Python are recognizing the benefits of PEP 453 and both PyPy and Jython have plans to backport ensurepip to their 2.7 runtimes. Automatic Invocation ==================== PEP 453 has ``ensurepip`` automatically invoked by default in the ``Makefile`` and the Windows and OSX Installers. This allowed it to ensure that, by default, all users would get Python with pip already installed. This PEP however believes that while this is fine for the Python 2.7 Windows and Mac OS X installers it is *not* ok for the Python 2.7 ``Makefile`` in general. The primary consumers of the ``Makefile`` are downstream package managers which distribute Python themselves. These downstream distributors typically do not want pip to be installed via ``ensurepip`` and would prefer that end users install it with their own package manager. Not invoking ``ensurepip`` automatically from the ``Makefile`` would allow these distributors to simply ignore the fact that ``ensurepip`` has been backported and still not end up with pip installed via it. The primary consumers of the OSX and Windows installers are end users who are attempting to install Python on their own machine. There is not a package manager available where these users can install pip into their Python through a more supported mechanism. For this reason it is the belief of this PEP that installing by default on OSX and Windows is the best course of action. Documentation ============= As part of this PEP, the updated packaging distribution and installation guides for Python 3.4 would be backported to Python 2.7. Disabling ensurepip by Downstream Distributors ============================================== Due to its use in the ``venv`` module, downstream distributors cannot disable the ``ensurepip`` module in Python 3.4. However since Python 2.7 has no such module it is explicitly allowed for downstream distributors to patch the ``ensurepip`` module to prevent it from installing anything. If a downstream distributor wishes to disable ``ensurepip`` completely in Python 2.7, they should still at least provide the module and allow `python -m ensurepip` style invocation. However it should raise errors or otherwise exit with a non-zero exit code and print out an error on stderr directing users to what they can/should use instead of ``ensurepip``. References ========== .. [#six] `six.py `__ .. [#modernize] `modernize `__ .. [#future] `python-future `__ .. [#setuptools] `setuptools `__ .. [#wheel] `Wheel `__ .. [#twine] `twine `__ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko at pacujo.net Mon Sep 1 00:04:52 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 01:04:52 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: (Victor Stinner's message of "Sun, 31 Aug 2014 23:38:38 +0200") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> Message-ID: <87sikcqprv.fsf@elektro.pacujo.net> Victor Stinner : > But I don't get you point. How does this PEP make the situation worse? Did I say it would? I just wanted to make sure the system call resumption doesn't become mandatory. Haven't thought through what the exception raising technique would entail. It might be perfectly ok apart from being a change to the signal handler API. > I don't know issues of signals with select() (and without a file > descriptor used to wake up it). A signal handler often sets a flag, which is inspected when select() returns. The problem is when a signal arrives between testing the flag and calling select(). The pselect() system call allows you to block signals and have the system call unblock them correctly to avoid the race. Marko From ncoghlan at gmail.com Mon Sep 1 00:10:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 08:10:58 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <540396E9.2020800@python.org> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: On 1 Sep 2014 07:43, "Christian Heimes" wrote: > > On 31.08.2014 08:09, Nick Coghlan wrote: > > As Antoine says here, I'm also opposed to adding more Python specific > > configuration options. However, I think there may be something > > worthwhile we can do that's closer to the way browsers work, and has > > the significant benefit of being implementable as a PyPI module first > > (more on that in a separate reply). > > I'm on your and Antoine's side and strictly against any additional > environment variables or command line arguments. That would make the > whole validation process even more complex and harder to understand. > > There might be a better option to give people and companies the option > to tune the SSL module to their needs. Python already have a > customization hook for the site module called sitecustomize. How about > another module named sslcustomize? Such a module could be used to tune > the ssl module to the needs of users, e.g. configure a different default > context, add certificates to a default context etc. > > Companies could install them in a system global directory on their > servers. Users could put them in their own user site directory and even > each virtual env can have one sslcustomize of its own. It's fully > backward compatible, doesn't add any flags and developers have the full > power of Python for configuration and customization. And means a user specific store (if one became available) could be configured there. Yes, I think this would address my concerns, especially if combined with a clear recipe in the documentation on how to optionally disable cert validation at the application layer. Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. Cheers, Nick. > > Christian -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko at pacujo.net Mon Sep 1 00:15:12 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 01:15:12 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <540395BC.50701@stoneleaf.us> (Ethan Furman's message of "Sun, 31 Aug 2014 14:38:04 -0700") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <540395BC.50701@stoneleaf.us> Message-ID: <87oav0qpan.fsf@elektro.pacujo.net> Ethan Furman : > On 08/31/2014 02:19 PM, Marko Rauhamaa wrote: >> The application will often want the EINTR return (exception) instead >> of having the function resume on its own. > > Examples? > > As an ignorant person in this area, I do not know why I would ever > want to have EINTR raised instead just getting the results of, say, my > read() call. Say you are writing data into a file and it takes a long time (because there is a lot of data or the medium is very slow or there is a hardware problem). You might have designed in a signaling scheme to address just this possibility. Then, the system call had better come out right away without trying to complete the full extent of the call. If a signal is received when read() or write() has completed its task partially (> 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). Marko From donald at stufft.io Mon Sep 1 00:15:29 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 31 Aug 2014 18:15:29 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <540396E9.2020800@python.org> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: > On Aug 31, 2014, at 5:43 PM, Christian Heimes wrote: > > On 31.08.2014 08:09, Nick Coghlan wrote: >> As Antoine says here, I'm also opposed to adding more Python specific >> configuration options. However, I think there may be something >> worthwhile we can do that's closer to the way browsers work, and has >> the significant benefit of being implementable as a PyPI module first >> (more on that in a separate reply). > > I'm on your and Antoine's side and strictly against any additional > environment variables or command line arguments. That would make the > whole validation process even more complex and harder to understand. > > There might be a better option to give people and companies the option > to tune the SSL module to their needs. Python already have a > customization hook for the site module called sitecustomize. How about > another module named sslcustomize? Such a module could be used to tune > the ssl module to the needs of users, e.g. configure a different default > context, add certificates to a default context etc. > > Companies could install them in a system global directory on their > servers. Users could put them in their own user site directory and even > each virtual env can have one sslcustomize of its own. It's fully > backward compatible, doesn't add any flags and developers have the full > power of Python for configuration and customization. > > This may be a dumb question, but why can?t sitecustomize do this already? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Sep 1 00:36:36 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Sep 2014 00:36:36 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <87sikcqprv.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikcqprv.fsf@elektro.pacujo.net> Message-ID: Le 1 sept. 2014 00:04, "Marko Rauhamaa" a ?crit : > > Victor Stinner : > > > But I don't get you point. How does this PEP make the situation worse? > > Did I say it would? I just wanted to make sure the system call > resumption doesn't become mandatory. The syscall is only retried on EINTR if the signal handler didn't raise an exception. So it is not always retried: "Proposition If a system call fails with ``EINTR``, Python must call signalhandlers: call ``PyErr_CheckSignals()``. If a signal handler raisesan exception, the Python function fails with the exception.Otherwise, the system call is retried. If the system call takes atimeout parameter, the timeout is recomputed." > Haven't thought through what the exception raising technique would > entail. It might be perfectly ok apart from being a change to the signal > handler API. I don't think that it is safe to expect an InterruptedError if the signal handler doesn't raise an exception. Many Python module already retry the syscall on EINTR. So I'm not sure that the PEP is really a major change. It's just to make Python more homogeneous, always have the same reliable and portable behaviour. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Sep 1 00:37:14 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 00:37:14 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <540395BC.50701@stoneleaf.us> <87oav0qpan.fsf@elektro.pacujo.net> Message-ID: <20140901003714.69d034d3@fsol> On Mon, 01 Sep 2014 01:15:12 +0300 Marko Rauhamaa wrote: > > If a signal is received when read() or write() has completed its task > partially (> 0 bytes), no EINTR is returned but the partial count. > Obviously, Python should take that possibility into account so that > raising an exception in the signal handler (as mandated by the PEP) > doesn't cause the partial result to be lost on os.read() or os.write(). If the signal handler is called, the exception *will* be raised. There's no guarantee at which point in the Python code it will be raised (it's implementation-dependent), but it's near impossible to protect regular Python code against such asynchronous exceptions. Which is why you should switch to a wakeup fd scheme as mentioned by Victor, if you want to rely on signals at all. Regards Antoine. From tjreedy at udel.edu Mon Sep 1 00:56:22 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 31 Aug 2014 18:56:22 -0400 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: References: Message-ID: On 8/31/2014 6:00 PM, Nick Coghlan wrote: > Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 > maintenance release in addition to including it with Python 3.4. > > That part of the proposal proved to be controversial, so we dropped it > from the original PEP in order to focus on meeting the Python 3.4 > specific release deadlines. This also had the benefit of working out the > kinks in the bootstrapping processing as part of the Python 3.4 release > cycle. > > However, we still think we should start providing pip by default to > Python 2.7 users as well, at least as part of the Windows and Mac OS X > installers. Having used pip for 3.4 on two windows computers, the only reason I see to *not* install it with Python would be 'push' people to 3.4+ by keeping 2.7 harder to use. > One notable difference from PEP 453 is that because there is no venv > module in 2.7, and hence no integration between venv and ensurepip, > we can give redistributors the option of just disabling ensurepip > entirely and redirecting users to platform specific installation > tools. This also suggests that ensurepip does not have to be left on a Win/OS install either, after installation is done. Perhaps after ensuring that pip is installed, ensurepip could remove itself. This would remove any objection that its presence in /lib constitutes a new feature (which it is, even if not very useful). It would also keep /lib uniform across platforms. -- Terry Jan Reedy From victor.stinner at gmail.com Mon Sep 1 00:45:22 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Sep 2014 00:45:22 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <87oav0qpan.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <540395BC.50701@stoneleaf.us> <87oav0qpan.fsf@elektro.pacujo.net> Message-ID: Le 1 sept. 2014 00:17, "Marko Rauhamaa" a ?crit : > If a signal is received when read() or write() has completed its task > partially (> 0 bytes), no EINTR is returned but the partial count. > Obviously, Python should take that possibility into account so that > raising an exception in the signal handler (as mandated by the PEP) > doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. (When Python gets a signal, the C signal handler is immediatly called. The handler sets a flag which is cheched before executing an instruction. The Python signal handler can be called between two Python instructions. In some cases, it may be called earlier in functions checking manually the flag. IMO the exact behaviour is undefined. Python tries to call the Python signal handler as soon as possible, with a low performance overhead.) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Sep 1 01:17:29 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 01:17:29 +0200 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 References: Message-ID: <20140901011729.70cb7652@fsol> On Mon, 1 Sep 2014 08:00:14 +1000 Nick Coghlan wrote: > > That part of the proposal proved to be controversial, so we dropped it from > the original PEP in order to focus on meeting the Python 3.4 specific > release deadlines. This also had the benefit of working out the kinks in > the bootstrapping processing as part of the Python 3.4 release cycle. > > However, we still think we should start providing pip by default to Python > 2.7 users as well, at least as part of the Windows and Mac OS X installers. I don't agree with this. pip is simply not part of the 2.7 feature set. If you add pip to a bugfix version, then you have bugfix versions which are more featureful than others, which makes things more complicated to explain. Regards Antoine. From benjamin at python.org Mon Sep 1 01:20:38 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 31 Aug 2014 16:20:38 -0700 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: <20140901011729.70cb7652@fsol> References: <20140901011729.70cb7652@fsol> Message-ID: <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > On Mon, 1 Sep 2014 08:00:14 +1000 > Nick Coghlan wrote: > > > > That part of the proposal proved to be controversial, so we dropped it from > > the original PEP in order to focus on meeting the Python 3.4 specific > > release deadlines. This also had the benefit of working out the kinks in > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > However, we still think we should start providing pip by default to Python > > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > > I don't agree with this. pip is simply not part of the 2.7 feature set. > If you add pip to a bugfix version, then you have bugfix versions which > are more featureful than others, which makes things more complicated to > explain. 2.7.x has been and will be alive for so long that will already have to explain that sort thing; i.e. PEP 466 and why different bugfix releases support different versions of dependency libraries. From ncoghlan at gmail.com Mon Sep 1 01:51:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 09:51:56 +1000 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> References: <20140901011729.70cb7652@fsol> <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> Message-ID: On 1 Sep 2014 09:23, "Benjamin Peterson" wrote: > > On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > > On Mon, 1 Sep 2014 08:00:14 +1000 > > Nick Coghlan wrote: > > > > > > That part of the proposal proved to be controversial, so we dropped it from > > > the original PEP in order to focus on meeting the Python 3.4 specific > > > release deadlines. This also had the benefit of working out the kinks in > > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > > > However, we still think we should start providing pip by default to Python > > > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > > > > I don't agree with this. pip is simply not part of the 2.7 feature set. > > If you add pip to a bugfix version, then you have bugfix versions which > > are more featureful than others, which makes things more complicated to > > explain. > > 2.7.x has been and will be alive for so long that will already have to > explain that sort thing; i.e. PEP 466 and why different bugfix releases > support different versions of dependency libraries. Exactly. LTS is genuinely different from stopping maintenance after the next feature release - it requires considering the "stability risk" and "user experience improvement" questions separately. In this case, the problem is that the Python 2 platform *is* still evolving, but the centre of that evolution has moved to PyPI. For "standard library only" users, Python 2 stopped evolving back in 2010. For PyPI users, by contrast, it's still evolving at a rapid pace. For our Python 3 transition story to be coherent, we need to ensure tools like six, modernize and future are readily available, while still remaining free to evolve independently of the standard library. That means providing a package management utility as easily and as painlessly as possible. Embracing pip upstream for Python 2 as well as Python 3 also sends a powerful signal to redistributors that says "your users are going to need this" and makes them do additional work to *avoid* providing it. Some of them *will* choose that path, but that becomes a matter for discussion between them and their user base, rather than a problem we need to worry about upstream. Cheers, Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 1 02:06:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 10:06:59 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: On 1 Sep 2014 08:15, "Donald Stufft" wrote: > > >> On Aug 31, 2014, at 5:43 PM, Christian Heimes wrote: >> >> Companies could install them in a system global directory on their >> servers. Users could put them in their own user site directory and even >> each virtual env can have one sslcustomize of its own. It's fully >> backward compatible, doesn't add any flags and developers have the full >> power of Python for configuration and customization. > > This may be a dumb question, but why can?t sitecustomize do this already? It can. The advantage of a separate file is that it won't conflict with existing sitecustomize modules, so (for example) redistributors can add a default sslcustomize, and you can add one to your virtual environments that are integrated with the system Python environment without needing to worry about whether or not there's a global sitecustomize (you'd only have trouble if there was a global sslcustomize). Cheers, Nick. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Sep 1 00:28:04 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Sep 2014 10:28:04 +1200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> Message-ID: <5403A174.3020408@canterbury.ac.nz> Victor Stinner wrote: > As written in the PEP, if you want to be notified of the > signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. -- Greg From rdmurray at bitdance.com Mon Sep 1 03:10:44 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 31 Aug 2014 21:10:44 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: <20140901011044.B6458250E05@webabinitio.net> On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan wrote: > On 1 Sep 2014 07:43, "Christian Heimes" wrote: > > > > On 31.08.2014 08:09, Nick Coghlan wrote: > > > As Antoine says here, I'm also opposed to adding more Python specific > > > configuration options. However, I think there may be something > > > worthwhile we can do that's closer to the way browsers work, and has > > > the significant benefit of being implementable as a PyPI module first > > > (more on that in a separate reply). > > > > I'm on your and Antoine's side and strictly against any additional > > environment variables or command line arguments. That would make the > > whole validation process even more complex and harder to understand. > > > > There might be a better option to give people and companies the option > > to tune the SSL module to their needs. Python already have a > > customization hook for the site module called sitecustomize. How about > > another module named sslcustomize? Such a module could be used to tune > > the ssl module to the needs of users, e.g. configure a different default > > context, add certificates to a default context etc. > > > > Companies could install them in a system global directory on their > > servers. Users could put them in their own user site directory and even > > each virtual env can have one sslcustomize of its own. It's fully > > backward compatible, doesn't add any flags and developers have the full > > power of Python for configuration and customization. > > And means a user specific store (if one became available) could be > configured there. > > Yes, I think this would address my concerns, especially if combined with a > clear recipe in the documentation on how to optionally disable cert > validation at the application layer. > > Assuming sslcustomize was in site-packages rather than the standard library > directories, you would also be able to use virtual environments with an > appropriate sslcustomize module to disable cert checking even if the > application you were running didn't support direct configuration. It sounds like this would address my concerns as well (I don't really care *how* it is implemented as long as I don't have to touch the code of a third party application when I upgrade my python version to 3.5...remember, the context here is backward compatibility concerns). Does it address the issue of accepting an invalid cert, though? --David From drsalists at gmail.com Mon Sep 1 05:14:50 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 31 Aug 2014 20:14:50 -0700 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <5403A174.3020408@canterbury.ac.nz> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> Message-ID: On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing wrote: > Victor Stinner wrote: >> >> As written in the PEP, if you want to be notified of the signal, set a >> signal handler which raises an exception. > > I'm not convinced that this covers all possible use cases. > It might be all right if you have control over the signal > handler, but what if you don't? > > I think it's best if the functions in the os module remain > thin wrappers that expose the OS functionality as fully and > directly as possible. Anything else should be provided > by higher-level facilities. I'm inclined to agree about keeping the os module thin. If we were to recreate Python today, from scratch, it might make sense to hide this by default, but now there's almost certainly code out there that depends on the current behavior. But I also agree that it's hard to pin down which higher level Python library calls are going to be using system calls. My http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem with window resizing for a while (SIGWINCH), and although I use it pretty much daily now without problems, I'm still not sure I got 100% of the possibilities covered. Fortunately, wrapping a system call can be as simple as: def retry_on_eintr(function, *args, **kw): ''' Retry a system call until one of the following happens: 1) It succeeds 2) It errors with something other than EINTR ''' while True: try: return function(*args, **kw) except OSError: nothing, extra, nothing2 = sys.exc_info() dummy = nothing dummy = nothing2 if extra.errno == errno.EINTR: continue else: raise Note that os.read() and os.write() need different handling. From rdmurray at bitdance.com Mon Sep 1 06:51:25 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 01 Sep 2014 00:51:25 -0400 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> Message-ID: <20140901045126.5EC19250E05@webabinitio.net> On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg wrote: > On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing wrote: > > Victor Stinner wrote: > >> > >> As written in the PEP, if you want to be notified of the signal, set a > >> signal handler which raises an exception. > > > > I'm not convinced that this covers all possible use cases. > > It might be all right if you have control over the signal > > handler, but what if you don't? > > > > I think it's best if the functions in the os module remain > > thin wrappers that expose the OS functionality as fully and > > directly as possible. Anything else should be provided > > by higher-level facilities. > > I'm inclined to agree about keeping the os module thin. If we were to > recreate Python today, from scratch, it might make sense to hide this > by default, but now there's almost certainly code out there that > depends on the current behavior. > > But I also agree that it's hard to pin down which higher level Python > library calls are going to be using system calls. My > http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem > with window resizing for a while (SIGWINCH), and although I use it > pretty much daily now without problems, I'm still not sure I got 100% > of the possibilities covered. > > Fortunately, wrapping a system call can be as simple as: > > def retry_on_eintr(function, *args, **kw): > ''' > Retry a system call until one of the following happens: > 1) It succeeds > 2) It errors with something other than EINTR > ''' > > while True: > try: > return function(*args, **kw) > except OSError: > nothing, extra, nothing2 = sys.exc_info() > dummy = nothing > dummy = nothing2 > if extra.errno == errno.EINTR: > continue > else: > raise > > Note that os.read() and os.write() need different handling. Personally, I really want Python to handle EINTR for me. And indeed, that has been what we have been doing for a while now, piece by piece, bug by bug. Victor just wants to systematize and document that, and I think that's a good idea. We've been consistently treating lack of handling of EINTR as a bug. If there are *real* cases where that causes a backward compatibility problem, then we need to know. But so far, we have gotten zero complaints about the cases that we have fixed. --David PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket. From marko at pacujo.net Mon Sep 1 07:30:27 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 08:30:27 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <20140901045126.5EC19250E05@webabinitio.net> (R. David Murray's message of "Mon, 01 Sep 2014 00:51:25 -0400") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> <20140901045126.5EC19250E05@webabinitio.net> Message-ID: <8738cbrjpo.fsf@elektro.pacujo.net> "R. David Murray" : > PS: I recently switched from using selectors to using a timeout on a > socket because in that particular application I could, and because > reading a socket with a timeout handles EINTR (in recent python > versions), whereas reading a non-blocking socket doesn't. Under the > hood, a socket with a timeout is a non-blocking socket. Under what circumstances would a nonblocking socket generate an EINTR? I believe the biggest EINTR problem child is file I/O, which is always blocking in linux. Marko From ncoghlan at gmail.com Mon Sep 1 08:05:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 16:05:18 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140901011044.B6458250E05@webabinitio.net> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <20140901011044.B6458250E05@webabinitio.net> Message-ID: On 1 September 2014 11:10, R. David Murray wrote: > > It sounds like this would address my concerns as well (I don't really > care *how* it is implemented as long as I don't have to touch the > code of a third party application when I upgrade my python version to > 3.5...remember, the context here is backward compatibility concerns). > > Does it address the issue of accepting an invalid cert, though? That's actually an interesting question, as the PEP doesn't currently propose adding any new global configuration knobs to the ssl or httplib modules - it just proposes switching httplib from the legacy (but fully backwards compatible) ssl._create_stdlib_context() API to the newer (but potentially backwards incompatible in some environments) ssl.create_default_context() API. Having the ssl module import an sslcustomize module at the end wouldn't be enough unless the appropriate APIs were put in place to allow things to be configured at a process global level. One possible way to do that would be to provide a central context factory mapping that provide a module specific SSL context creator. We'd seed it appropriately for the stdlib modules where we wanted to use the legacy context definition, but it would default to using ssl.create_default_context. Under that kind of model, the first change we would actually make is to make ssl._create_stdlib_context() public under a suitable name, let's say ssl.create_legacy_context() Independenting of any other changes, exposing ssl.create_legacy_context() like that would also make it straightforward for folks to opt in to the old behaviour as an interim hack in a way that is easy to grep for and fix later (it's also something a linter can easily disallow). The second change would be to provide a mapping from arbitrary names to context factories in the ssl module that defaults to ssl.create_default_context: named_contexts = defaultdict((lambda name: create_default_context)) (A more accurate name would be "named_context_factory", but I think "named_contexts" reads better. Folks will learn quickly enough that it actually stores context factories rather than prebuilt context objects) The third change would be to replace all calls to "ssl._create_stdlib_context()" with calls to "ssl.named_contexts[__name__]()" instead. The final change would be to seed the context factory map appropriately for the standard library modules where we wanted to keep the *old* default: for modname in ("nntplib", "poplib", "imaplib", "ftplib", "smtplib", "asyncio.selector_events", "urllib.request", "http.client"): named_contexts[modname] = create_legacy_context The list I have above is for *all* current uses of "sss._create_stdlib_context". The backwards incompatible part of PEP 476 would then just be about removing names from that list (currently just "http.client", but I'd suggest "asyncio.selector_events" as another candidate, taking advantage of asyncio's provisional API status). The "revert to 3.4 behaviour" content for sslcustomize.py would then just be: import ssl ssl.named_contexts["http.client"] = ssl.create_legacy_context However, someone that wanted to also enforce SSL properly for other standard library modules could go the other way: import ssl for modname in ("nntplib", "poplib", "imaplib", "ftplib", "smtplib", "urllib.request"): ssl.named_contexts[modname] = ssl.create_default_context Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Sep 1 08:07:46 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Sep 2014 07:07:46 +0100 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: On 31 August 2014 23:10, Nick Coghlan wrote: > Assuming sslcustomize was in site-packages rather than the standard library > directories, you would also be able to use virtual environments with an > appropriate sslcustomize module to disable cert checking even if the > application you were running didn't support direct configuration. Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit. Paul From p.f.moore at gmail.com Mon Sep 1 08:12:06 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Sep 2014 07:12:06 +0100 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> Message-ID: On 31 August 2014 22:38, Victor Stinner wrote: > This case is described as the use case #2 in the PEP, so it is supported. As > written in the PEP, if you want to be notified of the signal, set a signal > handler which raises an exception. For example the default signal handler > for SIGINT raises KeyboardInterrupt. Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback... Paul From nad at acm.org Mon Sep 1 08:22:36 2014 From: nad at acm.org (Ned Deily) Date: Sun, 31 Aug 2014 23:22:36 -0700 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 References: <20140901011729.70cb7652@fsol> <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> Message-ID: In article , Nick Coghlan wrote: > On 1 Sep 2014 09:23, "Benjamin Peterson" wrote: > > On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > > > On Mon, 1 Sep 2014 08:00:14 +1000 > > > Nick Coghlan wrote: > > > > > > > > That part of the proposal proved to be controversial, so we dropped > it from > > > > the original PEP in order to focus on meeting the Python 3.4 specific > > > > release deadlines. This also had the benefit of working out the kinks > in > > > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > > > > > However, we still think we should start providing pip by default to > Python > > > > 2.7 users as well, at least as part of the Windows and Mac OS X > installers. A much bigger than average +1 > > > I don't agree with this. pip is simply not part of the 2.7 feature set. > > > If you add pip to a bugfix version, then you have bugfix versions which > > > are more featureful than others, which makes things more complicated to > > > explain. > > 2.7.x has been and will be alive for so long that will already have to > > explain that sort thing; i.e. PEP 466 and why different bugfix releases > > support different versions of dependency libraries. And that is a minor complication compared with the confusion and difficulty of trying to explain to users (stuck with 2.7 for the time being) of how to install third-party packages on each platform (especially Windows) versus the simplicity of the 3.4.x story, thanks to ensurepip. Having pip available as a documented, batteries-included tool for all current releases would be a huge usability improvement. > Exactly. LTS is genuinely different from stopping maintenance after the > next feature release - it requires considering the "stability risk" and > "user experience improvement" questions separately. > > In this case, the problem is that the Python 2 platform *is* still > evolving, but the centre of that evolution has moved to PyPI. For "standard > library only" users, Python 2 stopped evolving back in 2010. For PyPI > users, by contrast, it's still evolving at a rapid pace. > > For our Python 3 transition story to be coherent, we need to ensure tools > like six, modernize and future are readily available, while still remaining > free to evolve independently of the standard library. That means providing > a package management utility as easily and as painlessly as possible. > > Embracing pip upstream for Python 2 as well as Python 3 also sends a > powerful signal to redistributors that says "your users are going to need > this" and makes them do additional work to *avoid* providing it. Some of > them *will* choose that path, but that becomes a matter for discussion > between them and their user base, rather than a problem we need to worry > about upstream. FTR, I'm willing to backport the pieces I did for 3.4 and I could do the ensurepip backport, as well. I'll leave the Windows installer changes for someone else, though. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Mon Sep 1 08:44:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 16:44:43 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: On 1 September 2014 16:07, Paul Moore wrote: > On 31 August 2014 23:10, Nick Coghlan wrote: >> Assuming sslcustomize was in site-packages rather than the standard library >> directories, you would also be able to use virtual environments with an >> appropriate sslcustomize module to disable cert checking even if the >> application you were running didn't support direct configuration. > > Would this mean that a malicious package could install a custom > sslcustomize.py and so add unwanted certs to the system? I guess we > have to assume that installed packages are trusted, but I just wanted > to be explicit. Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The "-S" and "-I" switches would need to disable the implied "sslcustomize", just as they disable "import site". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Mon Sep 1 09:13:33 2014 From: christian at python.org (Christian Heimes) Date: Mon, 01 Sep 2014 09:13:33 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> Message-ID: <54041C9D.1080404@python.org> On 01.09.2014 08:44, Nick Coghlan wrote: > Yes, it would have exactly the same security failure modes as > sitecustomize, except it would only fire if the application > imported the ssl module. > > The "-S" and "-I" switches would need to disable the implied > "sslcustomize", just as they disable "import site". A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. Christian From greg.ewing at canterbury.ac.nz Mon Sep 1 09:15:33 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Sep 2014 19:15:33 +1200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <540395BC.50701@stoneleaf.us> <87oav0qpan.fsf@elektro.pacujo.net> Message-ID: <54041D15.60102@canterbury.ac.nz> Victor Stinner wrote: > > Le 1 sept. 2014 00:17, "Marko Rauhamaa" > a ?crit : > > If a signal is received when read() or write() has completed its task > > partially (> 0 bytes), no EINTR is returned but the partial count. > > Obviously, Python should take that possibility into account so that > > raising an exception in the signal handler (as mandated by the PEP) > > doesn't cause the partial result to be lost on os.read() or os.write(). > > This case is unrelated to the PEP, the PEP only changes the behaviour > when a syscall fails with EINTR. I think there's a problem here, though. As thing stand, a signal handler that doesn't raise an exception can set a flag, and code after the read() can test it. Under the proposed scheme, the signal handler has to be made to raise an exception so that the read will be broken out of in the EINTR case. But what happens if the read returns *without* an EINTR? The signal handler will still raise an exception, which is either going to clobber the partial return value or mess up the code that does something with it. -- Greg From victor.stinner at gmail.com Mon Sep 1 09:18:27 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Sep 2014 09:18:27 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> Message-ID: No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Victor Le 1 sept. 2014 08:12, "Paul Moore" a ?crit : > On 31 August 2014 22:38, Victor Stinner wrote: > > This case is described as the use case #2 in the PEP, so it is > supported. As > > written in the PEP, if you want to be notified of the signal, set a > signal > > handler which raises an exception. For example the default signal handler > > for SIGINT raises KeyboardInterrupt. > > Wait - sigint? Does this mean that (unless the application adds a > signal handler) keyboard interrupts will be in effect ignored while in > a system call? I'm not sure I like that - I'd rather Ctrl-C always > interrupted the program. Specifically, in one-off scripts that *don't* > take care to handle all errors appropriately and just show the > traceback... > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Sep 1 09:27:29 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Sep 2014 09:27:29 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <5403A174.3020408@canterbury.ac.nz> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> Message-ID: Le 1 sept. 2014 02:40, "Greg Ewing" a ?crit : > > Victor Stinner wrote: >> >> As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. > > > I'm not convinced that this covers all possible use cases. > It might be all right if you have control over the signal > handler, but what if you don't? Ok, let's say that a syscall is interrupted by a signal, but rhe signal doesn't raise an exception. So your program can only be interrupted if the signal is received during a syscall, right? I don't think that such program is reliable. Python should not promote such design. It should behave the same if the signal is received during a CPU-bound function. Extract of the PEP: Backward Compatibility: Applications relying on the fact that system calls are interruptedwith ``InterruptedError`` will hang. The authors of this PEP don'tthink that such application exist.If such applications exist, they are not portable and are subject torace conditions (deadlock if the signal comes before the system call).These applications must be fixed to handle signals differently, tohave a reliable behaviour on all platforms and all Python versions.For example, use a signal handler which raises an exception, or use awakeup file descriptor.For applications using event loops, ``signal.set_wakeup_fd()`` is therecommanded option to handle signals. The signal handler writes signalnumbers into the file descriptor and the event loop is awaken to readthem. The event loop can handle these signals without the restrictionof signal handlers. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Sep 1 09:29:22 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Sep 2014 03:29:22 -0400 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: References: <20140901011729.70cb7652@fsol> <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> Message-ID: <37AFC366-EAA2-4863-AF1D-9111B92BD7FE@stufft.io> > On Sep 1, 2014, at 2:22 AM, Ned Deily wrote: > > In article > >, > Nick Coghlan > wrote: >> On 1 Sep 2014 09:23, "Benjamin Peterson" wrote: >>> On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: >>>> On Mon, 1 Sep 2014 08:00:14 +1000 >>>> Nick Coghlan wrote: >>>>> >>>>> That part of the proposal proved to be controversial, so we dropped >> it from >>>>> the original PEP in order to focus on meeting the Python 3.4 specific >>>>> release deadlines. This also had the benefit of working out the kinks >> in >>>>> the bootstrapping processing as part of the Python 3.4 release cycle. >>>>> >>>>> However, we still think we should start providing pip by default to >> Python >>>>> 2.7 users as well, at least as part of the Windows and Mac OS X >> installers. > > A much bigger than average +1 > >>>> I don't agree with this. pip is simply not part of the 2.7 feature set. >>>> If you add pip to a bugfix version, then you have bugfix versions which >>>> are more featureful than others, which makes things more complicated to >>>> explain. >>> 2.7.x has been and will be alive for so long that will already have to >>> explain that sort thing; i.e. PEP 466 and why different bugfix releases >>> support different versions of dependency libraries. > > And that is a minor complication compared with the confusion and > difficulty of trying to explain to users (stuck with 2.7 for the time > being) of how to install third-party packages on each platform > (especially Windows) versus the simplicity of the 3.4.x story, thanks to > ensurepip. Having pip available as a documented, batteries-included > tool for all current releases would be a huge usability improvement. Yes this is a major driver. I mean I think I probably have an above average knowledge of how to bootstrap pip, and if you dump me on a Windows box I struggle to actually do it the first time around without stumbling around and doing things in the wrong order and the like. (Getting a compiler toolchain is worse, but yay for Wheels). > >> Exactly. LTS is genuinely different from stopping maintenance after the >> next feature release - it requires considering the "stability risk" and >> "user experience improvement" questions separately. >> >> In this case, the problem is that the Python 2 platform *is* still >> evolving, but the centre of that evolution has moved to PyPI. For "standard >> library only" users, Python 2 stopped evolving back in 2010. For PyPI >> users, by contrast, it's still evolving at a rapid pace. >> >> For our Python 3 transition story to be coherent, we need to ensure tools >> like six, modernize and future are readily available, while still remaining >> free to evolve independently of the standard library. That means providing >> a package management utility as easily and as painlessly as possible. >> >> Embracing pip upstream for Python 2 as well as Python 3 also sends a >> powerful signal to redistributors that says "your users are going to need >> this" and makes them do additional work to *avoid* providing it. Some of >> them *will* choose that path, but that becomes a matter for discussion >> between them and their user base, rather than a problem we need to worry >> about upstream. > > FTR, I'm willing to backport the pieces I did for 3.4 and I could do the > ensurepip backport, as well. I'll leave the Windows installer changes > for someone else, though. Awesome, I?m of course willing to back port ensure pip itself as well. Truthfully it shouldn?t be a very difficult backport. It?s only ~200 SLOC or so and the only real things would be removing a Python3ism here or there. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko at pacujo.net Mon Sep 1 09:38:01 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 10:38:01 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: (Victor Stinner's message of "Mon, 1 Sep 2014 09:18:27 +0200") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> Message-ID: <87sikbeqp2.fsf@elektro.pacujo.net> Victor Stinner : > No, it's the opposite. The PEP doesn't change the default behaviour of > SIGINT: CTRL+C always interrupt the program. Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? Marko From ncoghlan at gmail.com Mon Sep 1 10:09:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 18:09:57 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <54041C9D.1080404@python.org> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> Message-ID: On 1 September 2014 17:13, Christian Heimes wrote: > On 01.09.2014 08:44, Nick Coghlan wrote: >> Yes, it would have exactly the same security failure modes as >> sitecustomize, except it would only fire if the application >> imported the ssl module. >> >> The "-S" and "-I" switches would need to disable the implied >> "sslcustomize", just as they disable "import site". > > A malicious package can already play havoc with your installation with > a custom ssl module. If somebody is able to sneak in a ssl.py then you > are screwed anyway. sslcustomize is not going to make the situation worse. That's not quite true - we're fairly careful about putting the standard library before userspace directories, so aside from the "current directory" problem, shadowing "ssl" itself can be tricky to arrange. "sslcustomize" would be more like "sitecustomize" - since it wouldn't normally be in the standard library, it can appear anywhere on sys.path, rather than having to be injected ahead of the standard library. I think that's OK though - compared to the security nightmare that is downloading modules from PyPI and running "./setup.py install" (or, even worse, "sudo ./setup.py install"), this would be a rather esoteric attack vector, and the existing -S and -I mechanisms could be used to defend against it :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mcepl at cepl.eu Mon Sep 1 10:12:36 2014 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Mon, 1 Sep 2014 10:12:36 +0200 Subject: [Python-Dev] Bug 19494 ... urllib2.HTTPBasicAuthHandler for GitHub et al. Message-ID: Hi, now when the vacations even in Europe are over could I ask for some movement on http://bugs.python.org/issue19494? Demanding a half-megabyte amount of packages from PIP ('just use requests' mentioned by some comments in the thread) or for that matter any package from PIP (including mine https://pypi.python.org/pypi/urllib2_prior_auth/) for something which is essentially a ten-lines patch seems to me rather crazy. What happened to all those batteries which were supposed to be included? Best, Mat?j From ncoghlan at gmail.com Mon Sep 1 11:15:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Sep 2014 19:15:52 +1000 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: <37AFC366-EAA2-4863-AF1D-9111B92BD7FE@stufft.io> References: <20140901011729.70cb7652@fsol> <1409527238.1660146.158837425.1B75F535@webmail.messagingengine.com> <37AFC366-EAA2-4863-AF1D-9111B92BD7FE@stufft.io> Message-ID: On 1 Sep 2014 17:31, "Donald Stufft" wrote: > > >> On Sep 1, 2014, at 2:22 AM, Ned Deily wrote: >> >> >> And that is a minor complication compared with the confusion and >> difficulty of trying to explain to users (stuck with 2.7 for the time >> being) of how to install third-party packages on each platform >> (especially Windows) versus the simplicity of the 3.4.x story, thanks to >> ensurepip. Having pip available as a documented, batteries-included >> tool for all current releases would be a huge usability improvement. > > > Yes this is a major driver. I mean I think I probably have an above average > knowledge of how to bootstrap pip, and if you dump me on a Windows box > I struggle to actually do it the first time around without stumbling around and > doing things in the wrong order and the like. (Getting a compiler toolchain is > worse, but yay for Wheels). Yeah. I've mentioned it before, but I think it bears repeating that trying to install pip on Windows with both Python 2 & 3 installed was one of the key things that convinced me to write PEP 453 in the first place. The default settings in both Internet and Windows explorer make it tricky regardless, but parallel installs make it even worse. >> FTR, I'm willing to backport the pieces I did for 3.4 and I could do the >> ensurepip backport, as well. I'll leave the Windows installer changes >> for someone else, though. > > > Awesome, I?m of course willing to back port ensure pip itself as well. Truthfully > it shouldn?t be a very difficult backport. It?s only ~200 SLOC or so and the only > real things would be removing a Python3ism here or there. Backporting meaningful tests will actually be the annoying part: the current unit tests use unittest.mock, while the current functional tests use pyvenv :) Both of those can be dealt with, but the tests will be a bit of an ugly hack by comparison with their Py3 counterparts :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Mon Sep 1 12:40:29 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Mon, 1 Sep 2014 11:40:29 +0100 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <87sikbeqp2.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> Message-ID: There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. As for the general behavior: all programming languages/platforms handle EINTR transparently. It's high time for Python to have a sensible behavior in this regard. 2014-09-01 8:38 GMT+01:00 Marko Rauhamaa : > Victor Stinner : > >> No, it's the opposite. The PEP doesn't change the default behaviour of >> SIGINT: CTRL+C always interrupt the program. > > Which raises an interesting question: what happens to the os.read() > return value if SIGINT is received? > > > Marko > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/cf.natali%40gmail.com From marko at pacujo.net Mon Sep 1 13:15:52 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 14:15:52 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: (=?utf-8?Q?=22Charles-Fran=C3=A7ois?= Natali"'s message of "Mon, 1 Sep 2014 11:40:29 +0100") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> Message-ID: <87egvveglz.fsf@elektro.pacujo.net> Charles-Fran?ois Natali : >> Which raises an interesting question: what happens to the os.read() >> return value if SIGINT is received? > > There's no return value, a KeywordInterrupt exception is raised. > The PEP wouldn't change this behavior. Slightly disconcerting... but I'm sure overriding SIGINT would cure that. You don't want to lose data if you want to continue running. > As for the general behavior: all programming languages/platforms > handle EINTR transparently. C doesn't. EINTR is there for a purpose. I sure hope Python won't bury it under opaque APIs. The two requirements are: * Allow the application to react to signals immediately in the main flow. * Don't lose information. Marko From cf.natali at gmail.com Mon Sep 1 13:25:53 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Mon, 1 Sep 2014 12:25:53 +0100 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <87egvveglz.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> <87egvveglz.fsf@elektro.pacujo.net> Message-ID: 2014-09-01 12:15 GMT+01:00 Marko Rauhamaa : > Charles-Fran?ois Natali : > >>> Which raises an interesting question: what happens to the os.read() >>> return value if SIGINT is received? >> >> There's no return value, a KeywordInterrupt exception is raised. >> The PEP wouldn't change this behavior. > > Slightly disconcerting... but I'm sure overriding SIGINT would cure > that. You don't want to lose data if you want to continue running. > >> As for the general behavior: all programming languages/platforms >> handle EINTR transparently. > > C doesn't. EINTR is there for a purpose. Python is slightly higher level than C, right? I was referring to Java, go, Haskell... Furthermore, that's not true: many operating systems actually restart syscalls by default (including Linux, man 7 signal): """ Interruption of system calls and library functions by signal handlers If a signal handler is invoked while a system call or library function call is blocked, then either: * the call is automatically restarted after the signal handler returns; or * the call fails with the error EINTR. Which of these two behaviors occurs depends on the interface and whether or not the signal handler was established using the SA_RESTART flag (see sigaction(2)). The details vary across UNIX systems; below, the details for Linux. """ The reason the interpreter is subject to so many EINTR is that we *explicitly* clear SA_RESTART because the C-level signal handler must be handled by the interpreter to have a chance to run the Python-level handlers from the main loop. There are many aspects of signal handling in Python that make it different from C: if you want C semantics, stick to C. I do not want to have to put all blocking syscalls within a try/except loop: have a look at the stdlib code, you'll see it's really a pain and ugly. And look at the number of EINTR-related syscalls we've had. From antoine at python.org Mon Sep 1 14:41:56 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 01 Sep 2014 14:41:56 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> Message-ID: Le 01/09/2014 10:09, Nick Coghlan a ?crit : > On 1 September 2014 17:13, Christian Heimes wrote: >> On 01.09.2014 08:44, Nick Coghlan wrote: >>> Yes, it would have exactly the same security failure modes as >>> sitecustomize, except it would only fire if the application >>> imported the ssl module. >>> >>> The "-S" and "-I" switches would need to disable the implied >>> "sslcustomize", just as they disable "import site". >> >> A malicious package can already play havoc with your installation with >> a custom ssl module. If somebody is able to sneak in a ssl.py then you >> are screwed anyway. sslcustomize is not going to make the situation worse. > > That's not quite true - we're fairly careful about putting the > standard library before userspace directories, so aside from the > "current directory" problem, shadowing "ssl" itself can be tricky to > arrange. Not sure why. Just put another module named "ssl" in sys.modules directly. You can also monkeypatch the genuine ssl module. Regards Antoine. From solipsis at pitrou.net Mon Sep 1 14:46:27 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 14:46:27 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <540395BC.50701@stoneleaf.us> <87oav0qpan.fsf@elektro.pacujo.net> <54041D15.60102@canterbury.ac.nz> Message-ID: <20140901144627.314eae66@fsol> On Mon, 01 Sep 2014 19:15:33 +1200 Greg Ewing wrote: > Victor Stinner wrote: > > > > Le 1 sept. 2014 00:17, "Marko Rauhamaa" > > a ?crit : > > > If a signal is received when read() or write() has completed its task > > > partially (> 0 bytes), no EINTR is returned but the partial count. > > > Obviously, Python should take that possibility into account so that > > > raising an exception in the signal handler (as mandated by the PEP) > > > doesn't cause the partial result to be lost on os.read() or os.write(). > > > > This case is unrelated to the PEP, the PEP only changes the behaviour > > when a syscall fails with EINTR. > > I think there's a problem here, though. As thing stand, a > signal handler that doesn't raise an exception can set a flag, > and code after the read() can test it. > > Under the proposed scheme, the signal handler has to > be made to raise an exception so that the read will be > broken out of in the EINTR case. The PEP already addresses this remark: ""Applications relying on the fact that system calls are interrupted with ``InterruptedError`` will hang. The authors of this PEP don't think that such application exist. If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call).""" Regards Antoine. From solipsis at pitrou.net Mon Sep 1 14:58:18 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 14:58:18 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: Message-ID: <20140901145818.1e5af142@fsol> Hi, I'm +1 on the whole PEP. > Writing a signal handler is difficult, only "async-signal safe" > functions can be called. You mean a C signal handler? Python signal handlers are not restricted. > Some signals are not interesting and should not interrupt the the > application. There are two options to only interrupt an application > on some signals: > > * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for > ``SIGINT`` > * Use a I/O multiplexing function like ``select()`` with the Python > signal "wakeup" file descriptor: see the function > ``signal.set_wakeup_fd()``. This section looks a bit incomplete. Some calls such as os.read() or os.write() will (should) return a partial result when interrupted and they already handled >0 bytes. Perhaps other functions have a similar behaviour? > On Unix, the ``asyncio`` module uses the wakeup file descriptor to > wake up its event loop. How about Windows? Regards Antoine. From rosuav at gmail.com Mon Sep 1 15:24:39 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Sep 2014 23:24:39 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> Message-ID: On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: > Not sure why. Just put another module named "ssl" in sys.modules directly. > You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? ChrisA From solipsis at pitrou.net Mon Sep 1 15:34:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 15:34:30 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> Message-ID: <20140901153430.6e11bbe3@fsol> On Mon, 1 Sep 2014 23:24:39 +1000 Chris Angelico wrote: > On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: > > Not sure why. Just put another module named "ssl" in sys.modules directly. > > You can also monkeypatch the genuine ssl module. > > That has to be done inside the same process. But imagine this > scenario: You have a program that gets invoked as root (or some other > user than yourself), and you're trying to fiddle with what it sees. > You don't have root access, but you can manipulate the file system, to > the extent that your userid has access. What can you do to affect this > other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Regards Antoine. From rosuav at gmail.com Mon Sep 1 15:42:10 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Sep 2014 23:42:10 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140901153430.6e11bbe3@fsol> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> Message-ID: On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou wrote: > On Mon, 1 Sep 2014 23:24:39 +1000 > Chris Angelico wrote: >> On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: >> > Not sure why. Just put another module named "ssl" in sys.modules directly. >> > You can also monkeypatch the genuine ssl module. >> >> That has to be done inside the same process. But imagine this >> scenario: You have a program that gets invoked as root (or some other >> user than yourself), and you're trying to fiddle with what it sees. >> You don't have root access, but you can manipulate the file system, to >> the extent that your userid has access. What can you do to affect this >> other program? > > If you're root you shouldn't run untrusted code. See > https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. ChrisA From solipsis at pitrou.net Mon Sep 1 15:59:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 15:59:35 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> Message-ID: <20140901155935.578ca8ef@fsol> On Mon, 1 Sep 2014 23:42:10 +1000 Chris Angelico wrote: > On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou wrote: > > On Mon, 1 Sep 2014 23:24:39 +1000 > > Chris Angelico wrote: > >> On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: > >> > Not sure why. Just put another module named "ssl" in sys.modules directly. > >> > You can also monkeypatch the genuine ssl module. > >> > >> That has to be done inside the same process. But imagine this > >> scenario: You have a program that gets invoked as root (or some other > >> user than yourself), and you're trying to fiddle with what it sees. > >> You don't have root access, but you can manipulate the file system, to > >> the extent that your userid has access. What can you do to affect this > >> other program? > > > > If you're root you shouldn't run untrusted code. See > > https://docs.python.org/3/using/cmdline.html#cmdoption-I > > Right, which is why sslcustomize has to be controlled by that, but the > possibility of patching (or monkeypatching) ssl.py isn't as big a > deal. To be frank I don't understand what you're arguing about. From ncoghlan at gmail.com Mon Sep 1 16:53:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Sep 2014 00:53:11 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140901155935.578ca8ef@fsol> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> Message-ID: On 2 Sep 2014 00:08, "Antoine Pitrou" wrote: > > On Mon, 1 Sep 2014 23:42:10 +1000 > Chris Angelico wrote: > > >> > > >> That has to be done inside the same process. But imagine this > > >> scenario: You have a program that gets invoked as root (or some other > > >> user than yourself), and you're trying to fiddle with what it sees. > > >> You don't have root access, but you can manipulate the file system, to > > >> the extent that your userid has access. What can you do to affect this > > >> other program? > > > > > > If you're root you shouldn't run untrusted code. See > > > https://docs.python.org/3/using/cmdline.html#cmdoption-I > > > > Right, which is why sslcustomize has to be controlled by that, but the > > possibility of patching (or monkeypatching) ssl.py isn't as big a > > deal. > > To be frank I don't understand what you're arguing about. When I said "shadowing ssl can be tricky to arrange", Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. I don't consider the latter cases to be interesting attack scenarios, as they imply the attacker is *already* running arbitrary Python code inside your CPython process, so you've already lost. Cheers, Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Sep 1 16:57:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 16:57:35 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> Message-ID: <20140901165735.59dae44b@fsol> On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan wrote: > On 2 Sep 2014 00:08, "Antoine Pitrou" wrote: > > > > On Mon, 1 Sep 2014 23:42:10 +1000 > > Chris Angelico wrote: > > > >> > > > >> That has to be done inside the same process. But imagine this > > > >> scenario: You have a program that gets invoked as root (or some other > > > >> user than yourself), and you're trying to fiddle with what it sees. > > > >> You don't have root access, but you can manipulate the file system, > to > > > >> the extent that your userid has access. What can you do to affect > this > > > >> other program? > > > > > > > > If you're root you shouldn't run untrusted code. See > > > > https://docs.python.org/3/using/cmdline.html#cmdoption-I > > > > > > Right, which is why sslcustomize has to be controlled by that, but the > > > possibility of patching (or monkeypatching) ssl.py isn't as big a > > > deal. > > > > To be frank I don't understand what you're arguing about. > > When I said "shadowing ssl can be tricky to arrange", Chris correctly > interpreted it as referring to the filesystem based privilege escalation > scenario that isolated mode handles, not to normal in-process > monkeypatching or module injection. There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything "tricky" about that. Regards Antoine. From ncoghlan at gmail.com Mon Sep 1 17:35:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Sep 2014 01:35:05 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140901165735.59dae44b@fsol> References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> <20140901165735.59dae44b@fsol> Message-ID: On 2 Sep 2014 00:59, "Antoine Pitrou" wrote: > > On Tue, 2 Sep 2014 00:53:11 +1000 > Nick Coghlan wrote: > > > > > > To be frank I don't understand what you're arguing about. > > > > When I said "shadowing ssl can be tricky to arrange", Chris correctly > > interpreted it as referring to the filesystem based privilege escalation > > scenario that isolated mode handles, not to normal in-process > > monkeypatching or module injection. > > There's no actual difference. You can have a sitecustomize.py that does > the monkeypatching or the shadowing. There doesn't seem to be anything > "tricky" about that. Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Sep 1 17:42:03 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 01 Sep 2014 11:42:03 -0400 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <8738cbrjpo.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> <20140901045126.5EC19250E05@webabinitio.net> <8738cbrjpo.fsf@elektro.pacujo.net> Message-ID: <20140901154204.87569250E11@webabinitio.net> On Mon, 01 Sep 2014 08:30:27 +0300, Marko Rauhamaa wrote: > "R. David Murray" : > > > PS: I recently switched from using selectors to using a timeout on a > > socket because in that particular application I could, and because > > reading a socket with a timeout handles EINTR (in recent python > > versions), whereas reading a non-blocking socket doesn't. Under the > > hood, a socket with a timeout is a non-blocking socket. > > Under what circumstances would a nonblocking socket generate an EINTR? Windows. Enough said? The exact error message was: BlockingIOError on my non-blocking socket: 'a non-blocking socket operation could not be completed immediately" Needless to say, I was not expecting this, and was about to tear my remaining hair out about having to completely restructure the code in order to be able to handle an EINTR on a read on an FD that I got back from select as ready, until I realized that the way the code had evolved the only thing I still needed the select for was the timeout, and that the EINTR bug in sockets with a timeout had already been fixed (thank goodness I'm able to use python3.4 for this project). I got lucky, but this is clearly a serious problem for writing selectors based code on Windows. This should tell you just about everything you need to know about why we want to fix this problem so that things work cross platform. --David From rdmurray at bitdance.com Mon Sep 1 17:47:07 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 01 Sep 2014 11:47:07 -0400 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <87egvveglz.fsf@elektro.pacujo.net> References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> <87egvveglz.fsf@elektro.pacujo.net> Message-ID: <20140901154708.78FA3250E11@webabinitio.net> On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa wrote: > Charles-Fran??ois Natali : > > >> Which raises an interesting question: what happens to the os.read() > >> return value if SIGINT is received? > > > > There's no return value, a KeywordInterrupt exception is raised. > > The PEP wouldn't change this behavior. > > Slightly disconcerting... but I'm sure overriding SIGINT would cure > that. You don't want to lose data if you want to continue running. > > > As for the general behavior: all programming languages/platforms > > handle EINTR transparently. > > C doesn't. EINTR is there for a purpose. I sure hope Python won't bury > it under opaque APIs. > > The two requirements are: > > * Allow the application to react to signals immediately in the main > flow. You don't want to be writing your code in Python then. In Python you *never* get to react immediately to signals. The interpreter sets a flag and calls the python signal handler later. Yes, the call is ASAP, but ASAP is *not* "immediately". > * Don't lose information. > > > Marko > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From solipsis at pitrou.net Mon Sep 1 17:51:16 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Sep 2014 17:51:16 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> <87egvveglz.fsf@elektro.pacujo.net> <20140901154708.78FA3250E11@webabinitio.net> Message-ID: <20140901175116.2c95d0a0@fsol> On Mon, 01 Sep 2014 11:47:07 -0400 "R. David Murray" wrote: > > > > The two requirements are: > > > > * Allow the application to react to signals immediately in the main > > flow. > > You don't want to be writing your code in Python then. In Python > you *never* get to react immediately to signals. The interpreter > sets a flag and calls the python signal handler later. Yes, the > call is ASAP, but ASAP is *not* "immediately". Especially if the signal is delivered to another thread (which is OS-dependent), and the main thread is blocked in *another* system call ;-) Regards Antoine. From marko at pacujo.net Mon Sep 1 17:59:08 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 18:59:08 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <20140901154204.87569250E11@webabinitio.net> (R. David Murray's message of "Mon, 01 Sep 2014 11:42:03 -0400") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <5403A174.3020408@canterbury.ac.nz> <20140901045126.5EC19250E05@webabinitio.net> <8738cbrjpo.fsf@elektro.pacujo.net> <20140901154204.87569250E11@webabinitio.net> Message-ID: <87vbp7pc1f.fsf@elektro.pacujo.net> "R. David Murray" : > Windows. Enough said? > [...] > This should tell you just about everything you need to know about why > we want to fix this problem so that things work cross platform. I feel your pain. Well, not really; I just don't want my linux bliss to be taken away. Marko From marko at pacujo.net Mon Sep 1 18:00:58 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Mon, 01 Sep 2014 19:00:58 +0300 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: <20140901154708.78FA3250E11@webabinitio.net> (R. David Murray's message of "Mon, 01 Sep 2014 11:47:07 -0400") References: <87lhq4sn3v.fsf@elektro.pacujo.net> <874mwss6gc.fsf@elektro.pacujo.net> <87sikbeqp2.fsf@elektro.pacujo.net> <87egvveglz.fsf@elektro.pacujo.net> <20140901154708.78FA3250E11@webabinitio.net> Message-ID: <87r3zvpbyd.fsf@elektro.pacujo.net> "R. David Murray" : > On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa wrote: >> * Allow the application to react to signals immediately in the main >> flow. > > You don't want to be writing your code in Python then. In Python you > *never* get to react immediately to signals. The interpreter sets a > flag and calls the python signal handler later. Yes, the call is ASAP, > but ASAP is *not* "immediately". You don't have to get that philosophical. "Immediately" means, "without delay", "without further I/O". Marko From jonas.wagner at epfl.ch Mon Sep 1 11:51:31 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Mon, 1 Sep 2014 11:51:31 +0200 Subject: [Python-Dev] cpython and parallel make Message-ID: Hi, what?s people?s experience with compiling cpython using multiple jobs (e.g., make -j 8)? In my case, I sometimes experience build errors that happen when using -j, whereas the single-job build always works. I haven?t tracked this down in detail, though? here?s an extract from a sample log: 17:42:56 make[1]: Entering directory `/path/to/cpython/build' 17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c 17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/ast.o ../../cpython/Python/ast.c 17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/ast.o ../../cpython/Python/ast.c 17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/ceval.o ../../cpython/Python/ceval.c 17:42:58 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/compile.o ../../cpython/Python/compile.c 17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c 17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/ceval.o ../../cpython/Python/ceval.c 17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/compile.o ../../cpython/Python/compile.c 17:43:08 make Include/graminit.h 17:43:08 make[1]: Entering directory `/path/to/cpython/build' 17:43:08 make[1]: `Include/graminit.h' is up to date. 17:43:08 make[1]: Leaving directory `/path/to/cpython/build' 17:43:24 clang -fsanitize=address -o Modules/_freeze_importlib Modules/_freeze_importlib.o Modules/getbuildinfo.o Parser/acceler.o [...] Objects/abstract.o [...] Objects/weakrefobject.o Python/_warnings.o [...] Python/ceval.o Python/compile.o Python/codecs.o [...] Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o -ldl -lutil -lm 17:43:24 /usr/bin/ld: error: Python/ceval.o: file is empty 17:43:24 /usr/bin/ld: error: Python/compile.o: file is empty For some reason, some files like Python/ceval.o seem to be built multiple times. I reckon this causes race conditions that result in some empty object files, which causes the linker to fail. Any ideas about where this comes from, and how to fix it? Cheers, Jonas ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Sep 1 18:48:23 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Sep 2014 12:48:23 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> <20140901165735.59dae44b@fsol> Message-ID: <947DE077-E0A1-467F-BB7B-388074854E85@stufft.io> > On Sep 1, 2014, at 11:35 AM, Nick Coghlan wrote: > > > On 2 Sep 2014 00:59, "Antoine Pitrou" > wrote: > > > > On Tue, 2 Sep 2014 00:53:11 +1000 > > Nick Coghlan > wrote: > > > > > > > > To be frank I don't understand what you're arguing about. > > > > > > When I said "shadowing ssl can be tricky to arrange", Chris correctly > > > interpreted it as referring to the filesystem based privilege escalation > > > scenario that isolated mode handles, not to normal in-process > > > monkeypatching or module injection. > > > > There's no actual difference. You can have a sitecustomize.py that does > > the monkeypatching or the shadowing. There doesn't seem to be anything > > "tricky" about that. > > Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). > > I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). > > Cheers, > Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Or you can just install something with easy_install, or you can drop a .pth file and monkey patch there. You can?t stop people from overriding modules, it?s trivial to do. The sys.path ordering just makes it slightly less trivial. ? Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Mon Sep 1 19:01:55 2014 From: christian at python.org (Christian Heimes) Date: Mon, 01 Sep 2014 19:01:55 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> <20140901165735.59dae44b@fsol> Message-ID: <5404A683.5020105@python.org> On 01.09.2014 17:35, Nick Coghlan wrote: > Oh, now I get what you mean - yes, sitecustomize already poses the same > kind of problem as the proposed sslcustomize (hence the existence of the > related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. From donald at stufft.io Mon Sep 1 19:08:11 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Sep 2014 13:08:11 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <5404A683.5020105@python.org> References: <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> <20140901165735.59dae44b@fsol> <5404A683.5020105@python.org> Message-ID: <5CC24097-5D56-4059-97FC-7ED9B6D622E1@stufft.io> > On Sep 1, 2014, at 1:01 PM, Christian Heimes wrote: > > On 01.09.2014 17:35, Nick Coghlan wrote: >> Oh, now I get what you mean - yes, sitecustomize already poses the same >> kind of problem as the proposed sslcustomize (hence the existence of the >> related command line options). > > If an attacker is able to place a module like sitecustomize.py in an > import directory or any .pth file in a site-packages directory than this > Python installation is compromised. .pth files are insidious because > they are always loaded and their code is always executed. I don't see > how sslcustomize is going to make a difference here. > Right, this is the point I was trying to make. If you?ve installed a malicious package it?s game over. There?s nothing Python can do to help you. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Sep 1 21:16:30 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Sep 2014 21:16:30 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: References: Message-ID: Hi, My bashrc sets MAKEFLAGS to -j9 and Python compilation works fine on Fedora 20 with GNU make and GCC. My computer has 8 cores (4 physical with hyper threading). It looks like your compiler is Clang. What is your OS and OS version? Can you try to run make in verbose mode and attach the full log to your email? Ex: try make SHELL="bash -x" to see executed shell commands. (Run "make clean" before) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Sep 1 22:27:48 2014 From: brett at python.org (Brett Cannon) Date: Mon, 01 Sep 2014 20:27:48 +0000 Subject: [Python-Dev] cpython and parallel make References: Message-ID: On Mon, Sep 1, 2014, 15:16 Victor Stinner wrote: Hi, My bashrc sets MAKEFLAGS to -j9 and Python compilation works fine on Fedora 20 with GNU make and GCC. My computer has 8 cores (4 physical with hyper threading). It looks like your compiler is Clang. What is your OS and OS version? I compile with -j8 with Clang on OS X and never have issues. -Brett Can you try to run make in verbose mode and attach the full log to your email? Ex: try make SHELL="bash -x" to see executed shell commands. (Run "make clean" before) Victor _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at woodcraft.me.uk Mon Sep 1 23:17:33 2014 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Mon, 1 Sep 2014 21:17:33 +0000 (UTC) Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: Message-ID: Victor Stinner wrote: > HTML version: > http://legacy.python.org/dev/peps/pep-0475/ > PEP: 475 > Title: Retry system calls failing with EINTR I think the proposed design for how Python should behave is a good one. But I think this proposal needs to be treated in the same way as any other backwards-incompatible change. > Applications relying on the fact that system calls are interrupted > with ``InterruptedError`` will hang. The authors of this PEP don't > think that such application exist. The authors are mistaken here. I have a program still running which was designed around this behaviour. My company won't be inconvenienced by this change because I can't imagine the elderly program ever being ported to Python 3. But I think it's very likely there are other such programs out there. > If such applications exist, they are not portable and are subject to > race conditions (deadlock if the signal comes before the system call). The program is certainly not portable (which is not any kind of a problem), and as pselect is unavailable there is indeed the usual theoretical race (which has not been a problem in practice in the ten years it's been running). (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) -M- From ncoghlan at gmail.com Mon Sep 1 23:31:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Sep 2014 07:31:44 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <5CC24097-5D56-4059-97FC-7ED9B6D622E1@stufft.io> References: <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> <20140901153430.6e11bbe3@fsol> <20140901155935.578ca8ef@fsol> <20140901165735.59dae44b@fsol> <5404A683.5020105@python.org> <5CC24097-5D56-4059-97FC-7ED9B6D622E1@stufft.io> Message-ID: On 2 Sep 2014 03:08, "Donald Stufft" wrote: > > >> On Sep 1, 2014, at 1:01 PM, Christian Heimes wrote: >> >> On 01.09.2014 17:35, Nick Coghlan wrote: >>> >>> Oh, now I get what you mean - yes, sitecustomize already poses the same >>> kind of problem as the proposed sslcustomize (hence the existence of the >>> related command line options). >> >> >> If an attacker is able to place a module like sitecustomize.py in an >> import directory or any .pth file in a site-packages directory than this >> Python installation is compromised. .pth files are insidious because >> they are always loaded and their code is always executed. I don't see >> how sslcustomize is going to make a difference here. >> > > Right, this is the point I was trying to make. If you?ve installed a malicious > package it?s game over. There?s nothing Python can do to help you. Yes, that's what I said originally when pointing out that isolated mode and the switch to disable site module processing would need to disable sslcustomize processing as well. Antoine was replying to a side comment about it being tricky to shadow stdlib modules. I left out the qualifier "directly" in my original comment, and he left out "indirectly through sitecustomize" in his initial reply, so we were talking past each for a while. Cheers, Nick. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Sep 2 01:16:11 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Sep 2014 01:16:11 +0200 Subject: [Python-Dev] Daily reference leaks (640c575ab3e1): sum=151940 References: Message-ID: <20140902011611.388af13e@fsol> Is anyone working on those? On Mon, 01 Sep 2014 10:41:45 +0200 solipsis at pitrou.net wrote: > results for 640c575ab3e1 on branch "default" > -------------------------------------------- > > test_codecs leaked [5825, 5825, 5825] references, sum=17475 > test_codecs leaked [1172, 1174, 1174] memory blocks, sum=3520 > test_collections leaked [0, 2, 0] references, sum=2 > test_distutils leaked [37735, 37735, 37735] references, sum=113205 > test_distutils leaked [5909, 5911, 5911] memory blocks, sum=17731 > test_functools leaked [0, 0, 3] memory blocks, sum=3 > test_site leaked [0, 0, 2] references, sum=2 > test_site leaked [0, 0, 2] memory blocks, sum=2 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogHPiXJp', '-x'] From benjamin at python.org Tue Sep 2 01:24:29 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 01 Sep 2014 16:24:29 -0700 Subject: [Python-Dev] Daily reference leaks (640c575ab3e1): sum=151940 In-Reply-To: <20140902011611.388af13e@fsol> References: <20140902011611.388af13e@fsol> Message-ID: <1409613869.690175.162452961.29C7510F@webmail.messagingengine.com> The codecs one is https://bugs.python.org/issue22166 On Mon, Sep 1, 2014, at 16:16, Antoine Pitrou wrote: > > Is anyone working on those? > > > > On Mon, 01 Sep 2014 10:41:45 +0200 > solipsis at pitrou.net wrote: > > results for 640c575ab3e1 on branch "default" > > -------------------------------------------- > > > > test_codecs leaked [5825, 5825, 5825] references, sum=17475 > > test_codecs leaked [1172, 1174, 1174] memory blocks, sum=3520 > > test_collections leaked [0, 2, 0] references, sum=2 > > test_distutils leaked [37735, 37735, 37735] references, sum=113205 > > test_distutils leaked [5909, 5911, 5911] memory blocks, sum=17731 > > test_functools leaked [0, 0, 3] memory blocks, sum=3 > > test_site leaked [0, 0, 2] references, sum=2 > > test_site leaked [0, 0, 2] memory blocks, sum=2 > > > > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogHPiXJp', '-x'] > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org From gokoproject at gmail.com Tue Sep 2 02:34:51 2014 From: gokoproject at gmail.com (John Wong) Date: Mon, 1 Sep 2014 20:34:51 -0400 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: References: <1408393321.2083664.154095037.3A4EB862@webmail.messagingengine.com> <20140818203043.GC1782@phdru.name> <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> Message-ID: As of today I still am getting untrusted cert thought I would re-ping to see if there is an ETA. On Thu, Aug 21, 2014 at 10:32 PM, Terry Reedy wrote: > On 8/21/2014 7:25 PM, Nick Coghlan wrote: > >> >> On 22 Aug 2014 04:45, "Benjamin Peterson" > > wrote: >> > >> > Perhaps some board members could comment, but I hope the PSF could just >> > pay a few hundred a year for a proper certificate. >> >> That's exactly what we're doing - MAL reminded me we reached the same >> conclusion last time this came up, we'll just track it better this time >> to make sure it doesn't slip through the cracks again. >> >> (And yes, switching to forced HTTPS once this is addressed would also be >> a good idea - we'll add it to the list) >> > > I just switched from a 'low variety' short password of the sort almost > crackable with brute force (today, though not several years ago) to a > higher variety longer password. People with admin privileges on the tracker > might be reminded to recheck. What was adequate 10 years ago is not so now. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/gokoproject%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Tue Sep 2 03:32:27 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 1 Sep 2014 20:32:27 -0500 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: References: <1408393321.2083664.154095037.3A4EB862@webmail.messagingengine.com> <20140818203043.GC1782@phdru.name> <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> Message-ID: I got the same in Chrome on my Mac. Skip On Sep 1, 2014 8:00 PM, "John Wong" wrote: > As of today I still am getting untrusted cert thought I would re-ping to > see if there is an ETA. > > On Thu, Aug 21, 2014 at 10:32 PM, Terry Reedy wrote: > >> On 8/21/2014 7:25 PM, Nick Coghlan wrote: >> >>> >>> On 22 Aug 2014 04:45, "Benjamin Peterson" >> > wrote: >>> > >>> > Perhaps some board members could comment, but I hope the PSF could >>> just >>> > pay a few hundred a year for a proper certificate. >>> >>> That's exactly what we're doing - MAL reminded me we reached the same >>> conclusion last time this came up, we'll just track it better this time >>> to make sure it doesn't slip through the cracks again. >>> >>> (And yes, switching to forced HTTPS once this is addressed would also be >>> a good idea - we'll add it to the list) >>> >> >> I just switched from a 'low variety' short password of the sort almost >> crackable with brute force (today, though not several years ago) to a >> higher variety longer password. People with admin privileges on the tracker >> might be reminded to recheck. What was adequate 10 years ago is not so now. >> >> -- >> Terry Jan Reedy >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/gokoproject%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/skip%40pobox.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Sep 2 07:49:01 2014 From: phd at phdru.name (Oleg Broytman) Date: Tue, 2 Sep 2014 07:49:01 +0200 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: References: <20140818203043.GC1782@phdru.name> <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> Message-ID: <20140902054901.GA18170@phdru.name> Hi! On Mon, Sep 01, 2014 at 08:32:27PM -0500, Skip Montanaro wrote: > I got the same in Chrome on my Mac. > > Skip > On Sep 1, 2014 8:00 PM, "John Wong" wrote: > > > As of today I still am getting untrusted cert thought I would re-ping to > > see if there is an ETA. The signing certificate is still CAcert. One can install their root certificate from http://www.cacert.org/index.php?id=3 Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ncoghlan at gmail.com Tue Sep 2 14:23:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Sep 2014 22:23:57 +1000 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: Message-ID: On 2 September 2014 07:17, Matthew Woodcraft wrote: > > (The program handles SIGTERM so that it can do a bit of cleanup before > exiting, and it uses the signal-handler-sets-a-flag technique. The call > that might be interrupted is sleep(), so the program doesn't strictly > _rely_ on the existing behaviour; it would just become very slow to > exit.) Making an exception for sleep() (i.e. still letting it throw EINTR) sounds like a reasonable idea to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 2 15:16:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 2 Sep 2014 23:16:27 +1000 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: References: <1408393321.2083664.154095037.3A4EB862@webmail.messagingengine.com> <20140818203043.GC1782@phdru.name> <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> Message-ID: On 2 September 2014 10:34, John Wong wrote: > As of today I still am getting untrusted cert thought I would re-ping to see > if there is an ETA. Thanks for the ping - I got sidetracked by other things, and didn't follow up on this one. I've kicked things into motion again, and it will hopefully be resolved before too long. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Sep 2 16:48:00 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Sep 2014 16:48:00 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: Message-ID: <20140902164800.1bd76c63@fsol> On Mon, 1 Sep 2014 21:17:33 +0000 (UTC) Matthew Woodcraft wrote: > > > If such applications exist, they are not portable and are subject to > > race conditions (deadlock if the signal comes before the system call). > > The program is certainly not portable (which is not any kind of a > problem), and as pselect is unavailable there is indeed the usual > theoretical race (which has not been a problem in practice in the ten > years it's been running). > > (The program handles SIGTERM so that it can do a bit of cleanup before > exiting, and it uses the signal-handler-sets-a-flag technique. The call > that might be interrupted is sleep(), so the program doesn't strictly > _rely_ on the existing behaviour; it would just become very slow to > exit.) So, if it's just for process exit, just let the signal handler raise an exception and catch the exception at the top-level. Regards Antoine. From tjreedy at udel.edu Tue Sep 2 22:14:25 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 02 Sep 2014 16:14:25 -0400 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: <20140902054901.GA18170@phdru.name> References: <20140818203043.GC1782@phdru.name> <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> <20140902054901.GA18170@phdru.name> Message-ID: On 9/2/2014 1:49 AM, Oleg Broytman wrote: > Hi! > > On Mon, Sep 01, 2014 at 08:32:27PM -0500, Skip Montanaro wrote: >> I got the same in Chrome on my Mac. >> >> Skip >> On Sep 1, 2014 8:00 PM, "John Wong" wrote: >> >>> As of today I still am getting untrusted cert thought I would re-ping to >>> see if there is an ETA. > > The signing certificate is still CAcert. One can install their root > certificate from http://www.cacert.org/index.php?id=3 This seems not to work for Firefox. "Windows installer package for browsers that use the Windows certificate store (for example Internet Explorer, Chrome on Windows and Safari on Windows)" I installed it anyway, closed and reopened Firefox (but not rebooted) and https://bugs.python.org still gives Untrusted message. -- Terry Jan Reedy From glyph at twistedmatrix.com Tue Sep 2 23:00:02 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 2 Sep 2014 14:00:02 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: Message-ID: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> On Aug 29, 2014, at 7:44 PM, Alex Gaynor wrote: > Disabling verification entirely externally to the program, through a CLI flag > or environment variable. I'm pretty down on this idea, the problem you hit is > that it's a pretty blunt instrument to swing, and it's almost impossible to > imagine it not hitting things it shouldn't; it's far too likely to be used in > applications that make two sets of outbound connections: 1) to some internal > service which you want to disable verification on, and 2) some external > service which needs strong validation. A global flag causes the latter to > fail silently when subjected to a MITM attack, and that's exactly what we're > trying to avoid. It also makes things much harder for library authors: I > write an API client for some API, and make TLS connections to it. I want > those to be verified by default. I can't even rely on the httplib defaults, > because someone might disable them from the outside. I would strongly recommend against such a mechanism. For what it's worth, Twisted simply unconditionally started verifying certificates in 14.0 with no "disable" switch, and (to my knowledge) literally no users have complained. Twisted has a very, very strict backwards compatibility policy. For example, I once refused to accept the deletion of a class that raised an exception upon construction, on the grounds that someone might have been inadvertently importing that class, and they shouldn't see an exception until they've seen a deprecation for one release. Despite that, we classified failing to verify certificates as a security bug, and fixed it with no deprecation period. When users type the 's' after the 'p' and before the ':' in a URL, they implicitly expect browser-like certificate verification. The lack of complaints is despite the fact that 14.0 has been out for several months now, and, thanks to the aforementioned strict policy, users tend to upgrade fairly often (since they know they can almost always do so without fear of application-breaking consequences). According to PyPI metadata, 14.0.0 has had 273283 downloads so far. Furthermore, "disable verification" is a nonsensical thing to do with TLS. "select a trust root" is a valid configuration option, and OpenSSL already provides it via the SSL_CERT_DIR environment variable, so there's no need for Python to provide anything beyond that. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at woodcraft.me.uk Tue Sep 2 23:02:15 2014 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Tue, 2 Sep 2014 21:02:15 +0000 (UTC) Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: Message-ID: Nick Coghlan wrote: >On 2 September 2014 07:17, Matthew Woodcraft wrote: >> (The program handles SIGTERM so that it can do a bit of cleanup before >> exiting, and it uses the signal-handler-sets-a-flag technique. The call >> that might be interrupted is sleep(), so the program doesn't strictly >> _rely_ on the existing behaviour; it would just become very slow to >> exit.) > Making an exception for sleep() (i.e. still letting it throw EINTR) > sounds like a reasonable idea to me. I think people who use sleep() in their programs could benefit from not having to worry about EINTR as much as anyone else. And I don't think there's any reason to suppose that programs using sleep() are more likely than others to want to see EINTR. I think blocking network operations are more likely to be involved. -M- From matthew at woodcraft.me.uk Tue Sep 2 23:03:36 2014 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Tue, 2 Sep 2014 21:03:36 +0000 (UTC) Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: <20140902164800.1bd76c63@fsol> Message-ID: Antoine Pitrou wrote: >Matthew Woodcraft wrote: >> (The program handles SIGTERM so that it can do a bit of cleanup before >> exiting, and it uses the signal-handler-sets-a-flag technique. The call >> that might be interrupted is sleep(), so the program doesn't strictly >> _rely_ on the existing behaviour; it would just become very slow to >> exit.) > So, if it's just for process exit, just let the signal handler raise an > exception and catch the exception at the top-level. I wouldn't recommend that anyone take this advice. The signal might come at some time other than the sleep(), and introducing an exception which can mysteriously appear at abitrary points is not a way to make life easier. Nonetheless I'm sure my program would be easy enough to fix to use set_wakeup_fd() or signalfd() if necessary. I'm not saying we shouldn't make this backwards-incompatible change because it will be hard to update existing programs to cope; I'm saying we shouldn't pretend it doesn't count as a backwards-incompatible change. In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? (I think it would be helpful if we could tell people "if you want the old EINTR behaviour, just do this simple thing". And I suppose siginterrupt flag=True is a candidate for that.) -M- From phd at phdru.name Tue Sep 2 23:08:02 2014 From: phd at phdru.name (Oleg Broytman) Date: Tue, 2 Sep 2014 23:08:02 +0200 Subject: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox) In-Reply-To: References: <53F61E83.6060605@v.loewis.de> <1408646706.3182921.155332569.3F0E8F5E@webmail.messagingengine.com> <20140902054901.GA18170@phdru.name> Message-ID: <20140902210802.GA1936@phdru.name> On Tue, Sep 02, 2014 at 04:14:25PM -0400, Terry Reedy wrote: > On 9/2/2014 1:49 AM, Oleg Broytman wrote: > >On Mon, Sep 01, 2014 at 08:32:27PM -0500, Skip Montanaro wrote: > >>I got the same in Chrome on my Mac. > >> > >>Skip > >>On Sep 1, 2014 8:00 PM, "John Wong" wrote: > >> > >>>As of today I still am getting untrusted cert thought I would re-ping to > >>>see if there is an ETA. > > > > The signing certificate is still CAcert. One can install their root > >certificate from http://www.cacert.org/index.php?id=3 > > This seems not to work for Firefox. "Windows installer package for > browsers that use the Windows certificate store (for example > Internet Explorer, Chrome on Windows and Safari on Windows)" > > I installed it anyway, closed and reopened Firefox (but not > rebooted) and https://bugs.python.org still gives Untrusted message. Did you install it in the Firefox own certificate manager? http://wiki.cacert.org/FAQ/BrowserClients#Mozilla_Firefox "Firefox uses it's own Certificate Manager. So even if your Windows (and other Microsoft) applications already use a root certificate Firefox still might not." Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Tue Sep 2 23:32:08 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 2 Sep 2014 23:32:08 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> Message-ID: <20140902233208.4e580ef4@fsol> On Tue, 2 Sep 2014 14:00:02 -0700 Glyph Lefkowitz wrote: > > I would strongly recommend against such a mechanism. > > For what it's worth, Twisted simply unconditionally started verifying certificates in 14.0 with no "disable" switch, and (to my knowledge) literally no users have complained. And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?) > Furthermore, "disable verification" is a nonsensical thing to do with TLS. It's not. For example, if you have an expired cert, all you can do AFAIK is to disable verification. Regards Antoine. From victor.stinner at gmail.com Tue Sep 2 23:35:28 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 2 Sep 2014 23:35:28 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: Message-ID: 2014-09-02 23:02 GMT+02:00 Matthew Woodcraft : > I think people who use sleep() in their programs could benefit from not > having to worry about EINTR as much as anyone else. The behaviour of time.sleep() is worse than what I expected. On UNIX, if select() fails with EINTR, time.sleep() calls PyErr_CheckSignals(). If the signal handler doesn't raise an exception, time.sleep() returns None and just simply ignores the error. But on Windows, it's the opposite. If time.sleep() is interrupt by CTRL+c, time.sleep() raises an InterruptedError... Good luck to write portable code :-p With the PEP 475, time.sleep(secs) now has a well defined behaviour. It sleeps at least "secs" seconds, retry the syscall on EINTR, and raises an exception if the signal handler raises an exception. Victor From victor.stinner at gmail.com Tue Sep 2 23:40:45 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 2 Sep 2014 23:40:45 +0200 Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR In-Reply-To: References: <20140902164800.1bd76c63@fsol> Message-ID: 2014-09-02 23:03 GMT+02:00 Matthew Woodcraft : > In any case I think PEP 475 should be explaining what is going to happen > to signal.siginterrupt(). Will setting flag=True be supported? I first proposed to deprecate the function, but Charles-Fran?ois thinks that it's unrelated to the PEP (it can be addressed later). The function will still be available and work. > If so, will doing so change the behaviour of those parts of the stdlib which > have already been modified to retry after EINTR? I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code. > (I think it would be helpful if we could tell people "if you want the > old EINTR behaviour, just do this simple thing". And I suppose > siginterrupt flag=True is a candidate for that.) Why do you want the old behaviour? Victor From ncoghlan at gmail.com Wed Sep 3 00:12:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 08:12:16 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <20140901011044.B6458250E05@webabinitio.net> Message-ID: On 1 Sep 2014 16:05, "Nick Coghlan" wrote: > > The final change would be to seed the context factory map > appropriately for the standard library modules where we wanted to keep > the *old* default: > > for modname in ("nntplib", "poplib", "imaplib", "ftplib", > "smtplib", "asyncio.selector_events", "urllib.request", > "http.client"): > named_contexts[modname] = create_legacy_context > > The list I have above is for *all* current uses of > "sss._create_stdlib_context". The backwards incompatible part of PEP > 476 would then just be about removing names from that list (currently > just "http.client", but I'd suggest "asyncio.selector_events" as > another candidate, taking advantage of asyncio's provisional API > status). Update on this: Christian & I both like the SSL named context and customisation idea independently of the proposal to change the default behaviour, so we're going to write it up as a separate PEP. Changing the default behaviour would then be reduced to a proposal to take a couple of module names off the "uses legacy SSL settings" list. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Sep 3 00:13:22 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 3 Sep 2014 00:13:22 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots Message-ID: Hi, I'm using Python buildbots to ensure that my changes don't fail on some platform. It's important for changes close to the operation system. The problem is that many buildbots are ill. Before, only a few buildbots had sporadic failures. Now most buildbots are always fail (are red). Here is an incomplete list of failures seen on buildbots. Before I opened an issue for each bug, but usually nobody takes care of them. So today I'm trying the mailing list. How can we fix all these issues to have more "green" buildbots? Can anyone help on this task? AMD64 OpenIndiana 3.x: a lot of tests fail with OSError(12, "Not enough space") or MemoryError. It's probably on issue on the host. AMD64 Snow Leop 3.x: many tests are not reliable (stable) on this platform. For example, test_logging.test_race() sometimes fail with PermissionError(1, "Operation not permitted: '/tmp/test_logging-3-bjulw8iz.log'"). Another example, test_asyncio.test_stdin_broken_pipe() sometimes fail with an assertion error because BrokenPipeError was not raised. Do we still support this old version of Mac OS X? Released in 2009, it is 5 years old. Is upgrading to Maverick (10.9) free and possible on old Mac computers? I don't have access to this old OS. x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it. x86 Windows Server 2003 [SB] 3.x: test_build_ext() of test_distutils hangs during 1 hang before being killed, it hangs something in the C compiler. x86 Windows7 3.x: test_nntplib fails with OSError("cannot read from timed out object"). x86 XP-4 3.x: test_distutils fails because Visual Studio linker (link.exe) fails with the error 1181 (cannot open input file). test_capi.test_forced_io_encoding() fails with an assertion error because sys.__stdin__ is None. AMD64 Solaris 11 [SB] 3.x: sometimes, tests fail with MemoryError. test_uuid.test_uuid4() fails with an assertion error. ctypes has issue with the sign of integer numbers (bug in libffi?). ARMv7 3.x: "Read-only file system", Mercurial fails... x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs. x86 FreeBSD 6.4 3.x: test_urllib2net.test_urlwithfrag() fails with urllib.error.URLError: . test_tcl has many issues. test_multiprocessing_spawn fails whereas the test should be skipped. x86 OpenBSD 5.5 3.x: many failing tests. x86 OpenIndiana 3.x: MemoryError. TestReadline.test_init() also fails. x86 Tiger 3.x: test_nntplib fails with OSError("cannot read from timed out object". I skipped "SPARC Solaris 10 OpenCSW 3.x" and "PPC64 AIX 3.x" in my list, I'm not really interested by these platforms. Victor From alex.gaynor at gmail.com Wed Sep 3 00:16:18 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 2 Sep 2014 22:16:18 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_476=3A_Enabling_certificate_validation?= =?utf-8?q?_by=09default!?= References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: Antoine Pitrou pitrou.net> writes: > > And how many people are using Twisted as an HTTPS client? > (compared to e.g. Python's httplib, and all the third-party libraries > building on it?) > I don't think anyone could give an honest estimate of these counts, however there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2. > > Furthermore, "disable verification" is a nonsensical thing to do with TLS. > > It's not. For example, if you have an expired cert, all you can do > AFAIK is to disable verification. > It really is a nonsensical operation, accepting any random TLS certificate without pinning or use of a certificate authorities makes a valid connection completely indistinguishable from a MITM attack. If I were the emperor of the universe (or even just Python ;-)) I wouldn't allow this operation at all, however, I'm not and you can still disable any and all verification. It just requires you to pass a different argument, which doesn't seem overly burdensome. This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version. Alex From solipsis at pitrou.net Wed Sep 3 00:25:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 00:25:38 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: <20140903002538.3b507988@fsol> On Tue, 2 Sep 2014 22:16:18 +0000 (UTC) Alex Gaynor wrote: > > > Furthermore, "disable verification" is a nonsensical thing to do with TLS. > > > > It's not. For example, if you have an expired cert, all you can do > > AFAIK is to disable verification. > > It really is a nonsensical operation, accepting any random TLS certificate > without pinning or use of a certificate authorities makes a valid connection > completely indistinguishable from a MITM attack. It's still distinguishable from a passively-eavesdroppable clear-text connection, though. > This whole scenario seems to be predicated on a siutation where: You have a > peer whose certificate you can't change, and you have a piece of code you can't > change, and you're going to upgrade your Python installation, and you want to > talk to this peer, and you need to use an encrypted channel, but don't really > care if it's being MITM'd. It doesn't seem to me that this is reasonably > Python's responsibility to deal with the fact that you have no ability to > upgrade any of your infrastructure, except your Python version. Oh, I agree that whoever upgrades their Python version should be able to fix any of their applications should they start failing. That's why I don't want to let new command-line options and environment variables proliferate in the name of damage control. Regards Antoine. From christian at python.org Wed Sep 3 00:41:46 2014 From: christian at python.org (Christian Heimes) Date: Wed, 03 Sep 2014 00:41:46 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140902233208.4e580ef4@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: <540647AA.4060509@python.org> On 02.09.2014 23:32, Antoine Pitrou wrote: >> Furthermore, "disable verification" is a nonsensical thing to do with TLS. > > It's not. For example, if you have an expired cert, all you can do > AFAIK is to disable verification. It's possible to ignore or just warn about expired certs with simple verify callback. The callback looks like this: int verify_callback(int ok, X509_STORE_CTX *ctx) { if (X509_STORE_CTX_get_error(ctx) == X509_V_ERR_CERT_HAS_EXPIRED) return 1; return ok; } It's installed like this: PySSLContext *self; X509_STORE *store = SSL_CTX_get_cert_store(self->ctx); X509_STORE_set_verify_cb(store, verify_callback); The X509_STORE_CTX struct is created when a certificate chain is verified. It holds all sorts of states like chain, leaf cert, current cert that is tested, validation depth, error flags and more. In order to write useful verify callbacks me or somebody else has to write a X509_STORE_CTX type and X509 cert type. It's something I want to do for more than a year but I don't find any spare time. :( Christian From ncoghlan at gmail.com Wed Sep 3 00:53:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 08:53:51 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: On 3 Sep 2014 08:15, "Victor Stinner" wrote: > > x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't > have to this platform, I don't know how to fix it. Sorry, I haven't been a very good maintainer for that buildbot (the main reason it never graduated to the "stable" list). If you send me your public SSH key, I can add it (I think - if not, I can ask Luke to do it). Alternatively, CentOS 6 may exhibit the same problem. >From a completely different perspective, does anyone have experience with using BuildBot with OpenStack hosted clients? We may be able to take advantage of the PSF's new(ish) Rackspace infrastructure to provide more stable test VMs. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Wed Sep 3 01:00:08 2014 From: brian at python.org (Brian Curtin) Date: Tue, 2 Sep 2014 18:00:08 -0500 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: On Tue, Sep 2, 2014 at 5:53 PM, Nick Coghlan wrote: > > On 3 Sep 2014 08:15, "Victor Stinner" wrote: > > > > x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't > > have to this platform, I don't know how to fix it. > > Sorry, I haven't been a very good maintainer for that buildbot (the main > reason it never graduated to the "stable" list). If you send me your public > SSH key, I can add it (I think - if not, I can ask Luke to do it). > Alternatively, CentOS 6 may exhibit the same problem. > > From a completely different perspective, does anyone have experience with > using BuildBot with OpenStack hosted clients? We may be able to take > advantage of the PSF's new(ish) Rackspace infrastructure to provide more > stable test VMs. > Is this a Buildbot feature (as in Buildbot master spins up VMs fresh for a test run, or something), or do you just want to spin up a bunch of VMs, give access, and we configure them the same as we do today? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 3 01:01:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 09:01:55 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: On 3 Sep 2014 08:18, "Alex Gaynor" wrote: > > Antoine Pitrou pitrou.net> writes: > > > > > And how many people are using Twisted as an HTTPS client? > > (compared to e.g. Python's httplib, and all the third-party libraries > > building on it?) > > > > I don't think anyone could give an honest estimate of these counts, however > there's two factors to bare in mind: a) It's extremely strongly recommended to > use requests to make any HTTP requests precisely because httplib is negligent > in certificate and hostname checking by default, b) We're talking about > Python3, which has fewer users than Python2. Creating *new* incompatibilities between Python 2 & Python 3 is a major point of concern. One key focus of 3.5 is *reducing* barriers to migration, and this PEP would be raising a new one. It's a change worth making, but we have time to ensure there are easy ways to do things like skipping cert validation, or tolerate expired certificates. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 3 01:05:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 09:05:18 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: On 3 Sep 2014 09:00, "Brian Curtin" wrote: > > On Tue, Sep 2, 2014 at 5:53 PM, Nick Coghlan wrote: >> >> From a completely different perspective, does anyone have experience with using BuildBot with OpenStack hosted clients? We may be able to take advantage of the PSF's new(ish) Rackspace infrastructure to provide more stable test VMs. > > Is this a Buildbot feature (as in Buildbot master spins up VMs fresh for a test run, or something), or do you just want to spin up a bunch of VMs, give access, and we configure them the same as we do today? The former would be better, but I have no idea if BuildBot supports it natively. The latter may still help if it replaces resource constrained VMs (and would also mean the PSF infra team could grant access for investigations). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dreid at dreid.org Wed Sep 3 01:06:53 2014 From: dreid at dreid.org (David Reid) Date: Tue, 2 Sep 2014 23:06:53 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_476=3A_Enabling_certificate_validation?= =?utf-8?q?_by=09default!?= References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: Nick Coghlan gmail.com> writes: > Creating *new* incompatibilities between Python 2 & Python 3 is a major point > of concern. Clearly this change should be backported to Python2. -David From rdmurray at bitdance.com Wed Sep 3 01:20:15 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 02 Sep 2014 19:20:15 -0400 Subject: [Python-Dev] =?utf-8?q?PEP_476=3A_Enabling_certificate_validation?= =?utf-8?q?_by=09default!?= In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: <20140902232015.88590B1408D@webabinitio.net> On Tue, 02 Sep 2014 22:16:18 -0000, Alex Gaynor wrote: > This whole scenario seems to be predicated on a siutation where: You have a > peer whose certificate you can't change, and you have a piece of code you can't > change, and you're going to upgrade your Python installation, and you want to > talk to this peer, and you need to use an encrypted channel, but don't really > care if it's being MITM'd. It doesn't seem to me that this is reasonably > Python's responsibility to deal with the fact that you have no ability to > upgrade any of your infrastructure, except your Python version. I would say that is an excellent summary, except that you are sometimes *forced* to use an encrypted channel (the device only opens the https port). That is, in the real messy world of network devices, accepting an invalid cert is not a nonsensical operation. (Of course, what I really want is to be able to say "I know this cert has invalid fields, accept it anyway, but warn me if it changes when I connect again".) It's a good point that the actual number of people who will be hit by this and can't hack the code in question will be (possibly vanishingly) small, especially if you only consider python3. But we've already had one call for a backport :) --David From glyph at twistedmatrix.com Wed Sep 3 01:21:07 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 2 Sep 2014 16:21:07 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: On Sep 2, 2014, at 4:01 PM, Nick Coghlan wrote: > > On 3 Sep 2014 08:18, "Alex Gaynor" wrote: > > > > Antoine Pitrou pitrou.net> writes: > > > > > > > > And how many people are using Twisted as an HTTPS client? > > > (compared to e.g. Python's httplib, and all the third-party libraries > > > building on it?) > > > > > > > I don't think anyone could give an honest estimate of these counts, however > > there's two factors to bare in mind: a) It's extremely strongly recommended to > > use requests to make any HTTP requests precisely because httplib is negligent > > in certificate and hostname checking by default, b) We're talking about > > Python3, which has fewer users than Python2. > > Creating *new* incompatibilities between Python 2 & Python 3 is a major point of concern. One key focus of 3.5 is *reducing* barriers to migration, and this PEP would be raising a new one. > No. Providing the security that the user originally asked for is not a "backwards incompatible change". It is a bug fix. And believe me: I care a _LOT_ about reducing barriers to migration. This would not be on my list of the top 1000 things that make migration difficult. > It's a change worth making, but we have time to ensure there are easy ways to do things like skipping cert validation, or tolerate expired certificates. > The API already supports both of these things. What I believe you're implicitly saying is that there needs to be a way to do this without editing code, and... no, there really doesn't. Not to mention the fact that you could already craft a horrific monkeypatch to allow operators to cause the ssl module to malfunction by 'pip install'ing a separate package, which is about as supported as this should be. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 3 01:28:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 09:28:08 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: On 3 Sep 2014 09:08, "David Reid" wrote: > > Nick Coghlan gmail.com> writes: > > > Creating *new* incompatibilities between Python 2 & Python 3 is a major point > > of concern. > > Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release would require an *even better* migration story for handling legacy infrastructure. There's a reason the Py3 compatibility break was accompanied by a long term support commitment for Python 2 (and why that commitment was extended another 5 years when it became clear the implications of the binary and text handling were more significant than we originally realised). It's important to remember that the tech industry in general, and open source focused companies in particular, tend to be amongst the best of the best when it comes to infrastructure management (and we're still pretty terrible at it). The vast majority of infrastructure in the world *isn't* in the hands of those companies though - it's in the hands of companies where having any kind of computing infrastructure at all is merely a cost of doing business. Alex's "ridiculous" scenario doesn't sound ridiculous to me - it sounds like a typical corporate computing environment. Regards, Nick. > > -David > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Sep 3 01:47:35 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 2 Sep 2014 16:47:35 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> Message-ID: <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> On Sep 2, 2014, at 4:28 PM, Nick Coghlan wrote: > On 3 Sep 2014 09:08, "David Reid" wrote: > > > > Nick Coghlan gmail.com> writes: > > > > > Creating *new* incompatibilities between Python 2 & Python 3 is a major point > > > of concern. > > > > Clearly this change should be backported to Python2. > > Proposing to break backwards compatibility in a maintenance release (...) > As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a "working" application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a "compatibility break" that the system starts correctly rejecting invalid connections. By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.) Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do "3 + 4" in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code! When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it "certificate verification" because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.) For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken. I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Sep 3 02:06:58 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 2 Sep 2014 20:06:58 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> Message-ID: <9DF8635E-1CD3-421C-88A9-9E0F5A657B6F@stufft.io> > On Sep 2, 2014, at 7:47 PM, Glyph Lefkowitz wrote: > > > On Sep 2, 2014, at 4:28 PM, Nick Coghlan > wrote: > >> On 3 Sep 2014 09:08, "David Reid" > wrote: >> > >> > Nick Coghlan gmail.com > writes: >> > >> > > Creating *new* incompatibilities between Python 2 & Python 3 is a major point >> > > of concern. >> > >> > Clearly this change should be backported to Python2. >> >> Proposing to break backwards compatibility in a maintenance release (...) >> > > As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a "working" application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a "compatibility break" that the system starts correctly rejecting invalid connections. > > By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.) > > Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do "3 + 4" in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code! > > When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it "certificate verification" because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.) > > For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken. > > I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security. > I completely agree with everything Glyph has said in this post. (To the shock of everyone involved I?m sure!). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Wed Sep 3 02:19:53 2014 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 3 Sep 2014 10:19:53 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: <20140903001953.GA87109@cskk.homeip.net> On 03Sep2014 00:13, Victor Stinner wrote: >AMD64 Snow Leop 3.x: many tests are not reliable (stable) on this >platform. For example, test_logging.test_race() sometimes fail with >PermissionError(1, "Operation not permitted: >'/tmp/test_logging-3-bjulw8iz.log'"). Another example, >test_asyncio.test_stdin_broken_pipe() sometimes fail with an assertion >error because BrokenPipeError was not raised. Do we still support this >old version of Mac OS X? Released in 2009, it is 5 years old. Is >upgrading to Maverick (10.9) free and possible on old Mac computers? I >don't have access to this old OS. As a negative data point, my GF runs Snow Leopard by choice and will not upgrade that machine; we've both got Mavericks laptops and there are major regressions in the UI and OS behaviour (Apple UI and apps, not Python). I would imagine she's not alone in resisting change. Cheers, Cameron Simpson Life IS pain, highness... anyone who tries to tell you different is trying to sell you something. - Wesley, The_Princess_Bride From solipsis at pitrou.net Wed Sep 3 02:19:46 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 02:19:46 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> Message-ID: <20140903021946.1bbcdd0c@fsol> On Tue, 2 Sep 2014 16:47:35 -0700 Glyph Lefkowitz wrote: > > On Sep 2, 2014, at 4:28 PM, Nick Coghlan wrote: > > > On 3 Sep 2014 09:08, "David Reid" wrote: > > > > > > Nick Coghlan gmail.com> writes: > > > > > > > Creating *new* incompatibilities between Python 2 & Python 3 is a major point > > > > of concern. > > > > > > Clearly this change should be backported to Python2. > > > > Proposing to break backwards compatibility in a maintenance release (...) > > > > As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Keeping saying it doesn't make it magically true. Besides, it can perfectly well be a bug fix *as well as* a break in backwards compatibility. Which is why we sometimes choose to fix bugs only in the feature development branch. Regards Antoine. From solipsis at pitrou.net Wed Sep 3 02:33:47 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 02:33:47 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots References: Message-ID: <20140903023347.3d2ff46a@fsol> On Wed, 3 Sep 2014 00:13:22 +0200 Victor Stinner wrote: > > AMD64 Snow Leop 3.x: many tests are not reliable (stable) on this > platform. For example, test_logging.test_race() sometimes fail with > PermissionError(1, "Operation not permitted: > '/tmp/test_logging-3-bjulw8iz.log'"). Another example, > test_asyncio.test_stdin_broken_pipe() sometimes fail with an assertion > error because BrokenPipeError was not raised. Do we still support this > old version of Mac OS X? Released in 2009, it is 5 years old. 5 years old is not that old. I don't know what the upgrade expectations are on OS X, but I'm sure Python runs well on 5-yeard old Windows, Linuces and perhaps even FreeBSDs. Regards Antoine. From solipsis at pitrou.net Wed Sep 3 02:37:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 02:37:38 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots References: Message-ID: <20140903023738.1ddc8aac@fsol> On Wed, 3 Sep 2014 08:53:51 +1000 Nick Coghlan wrote: > On 3 Sep 2014 08:15, "Victor Stinner" wrote: > > > > x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't > > have to this platform, I don't know how to fix it. > > Sorry, I haven't been a very good maintainer for that buildbot (the main > reason it never graduated to the "stable" list). If you send me your public > SSH key, I can add it (I think - if not, I can ask Luke to do it). > Alternatively, CentOS 6 may exhibit the same problem. > > From a completely different perspective, does anyone have experience with > using BuildBot with OpenStack hosted clients? We may be able to take > advantage of the PSF's new(ish) Rackspace infrastructure to provide more > stable test VMs. I'm not sure that's an answer to the problem. What we need is not more machines, but dedicated buildbot maintainers. That also means being willing to diagnose issues themselves rather than kindly offer SSH access, by the way ;-) (I can't speak for Victor, but being offered SSH access when I point out an issue on a remote machine is really a depressing answer to get, as far as I'm concerned) And, we also need some people to look after the master configuration and status - yes, that would have been me, but I've not been very motivated recently :-) Regards Antoine. From tjreedy at udel.edu Wed Sep 3 02:59:54 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 02 Sep 2014 20:59:54 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> Message-ID: On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote: > > On Sep 2, 2014, at 4:28 PM, Nick Coghlan > wrote: > >> On 3 Sep 2014 09:08, "David Reid" > > wrote: >> > Clearly this change should be backported to Python2. >> >> Proposing to break backwards compatibility in a maintenance release (...) For code that depends on a bug, fixing the bug breaks compatibility between releases without and with the fix. Nothing new. We occasionally don't backpart disruptive bugfixes because of that. We also backport some security fixes to otherwise unmaintained versions. > As we keep saying, this is not a break in backwards compatibility, it's > a bug fix. Disagreement on whether changing the default for certificate validation is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker sense) the seems like the crux of the disagreement here. The case for 'bug', at least in the colloquial meaning, is at least plausible. On the other hand, the tracker meaning of 'behavior' issue (contrary to intention and doc) is more strict in including implementation bugs but not necessarily design bugs. -- Terry Jan Reedy From rdmurray at bitdance.com Wed Sep 3 03:29:16 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 02 Sep 2014 21:29:16 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> Message-ID: <20140903012917.1E8D3B1408D@webabinitio.net> On Tue, 02 Sep 2014 20:59:54 -0400, Terry Reedy wrote: > On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote: > > > > On Sep 2, 2014, at 4:28 PM, Nick Coghlan > > wrote: > > > >> On 3 Sep 2014 09:08, "David Reid" >> > wrote: > > >> > Clearly this change should be backported to Python2. > >> > >> Proposing to break backwards compatibility in a maintenance release (...) > > For code that depends on a bug, fixing the bug breaks compatibility > between releases without and with the fix. Nothing new. We > occasionally don't backpart disruptive bugfixes because of that. We also > backport some security fixes to otherwise unmaintained versions. > > > As we keep saying, this is not a break in backwards compatibility, it's > > a bug fix. > > Disagreement on whether changing the default for certificate validation > is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker > sense) the seems like the crux of the disagreement here. The case for > 'bug', at least in the colloquial meaning, is at least plausible. On the > other hand, the tracker meaning of 'behavior' issue (contrary to > intention and doc) is more strict in including implementation bugs but > not necessarily design bugs. Up to this point the suggestion has been to change the default only in 3.5, and what we are arguing about is what kind of and how much aid to give to those for whom 3.5 breaks backward compatibility. So arguing about feature versus bug is irrelevant. The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for "this cert" or "this CLI session". It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.) The more I think about it the less I like sslcustomize. I think I'd rather do without the ability to turn off checking without modifying code rather than introduce an intentional vector for a system-wide decrease in security. --David From stephen at xemacs.org Wed Sep 3 04:43:25 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 03 Sep 2014 11:43:25 +0900 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903021946.1bbcdd0c@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903021946.1bbcdd0c@fsol> Message-ID: <87ppfd8lv6.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > On Tue, 2 Sep 2014 16:47:35 -0700 > Glyph Lefkowitz wrote: > > As we keep saying, this is not a break in backwards > > compatibility, it's a bug fix. > > Keeping saying it doesn't make it magically true. It's not "magically" true, it is "just" true. What the hardliners fail to acknowledge is that this is *not a bug in Python, it's a bug in the whole system*, and *mostly* in the environment. Changing Python will not change the environment, and applications will fail, with unknown consequences. Saying they "should" fail *right* now is bogus when you don't even know what those applications are, or what other security measures may be in place: Now is better than never. Although never is often better than *right* now. On the other hand, I commend the Twisted developers for putting their values into their code with their reputation on the line. I hope they win big with this move! Shouldn't we all hope for that? Steve From stephen at xemacs.org Wed Sep 3 04:47:38 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 03 Sep 2014 11:47:38 +0900 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > Sorry, I haven't been a very good maintainer for that buildbot (the main > reason it never graduated to the "stable" list). If you send me your public > SSH key, I can add it (I think - if not, I can ask Luke to do it). > Alternatively, CentOS 6 may exhibit the same problem. I wonder how many of these buildbots could be maintained by the kind of folks who show up on core-mentorship asking "how can I help?" Just a thought -- I wouldn't be surprised if the reaction is universal horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" And of course most want to write code, not sysadm. From rosuav at gmail.com Wed Sep 3 04:52:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Sep 2014 12:52:47 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > Sorry, I haven't been a very good maintainer for that buildbot (the main > > reason it never graduated to the "stable" list). If you send me your public > > SSH key, I can add it (I think - if not, I can ask Luke to do it). > > Alternatively, CentOS 6 may exhibit the same problem. > > I wonder how many of these buildbots could be maintained by the kind > of folks who show up on core-mentorship asking "how can I help?" > > Just a thought -- I wouldn't be surprised if the reaction is universal > horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" > > And of course most want to write code, not sysadm. Maintaining a buildbot isn't hard. (Although one thing I'm not sure of: If my bot goes down for an extended period of time, is any sort of automated email sent to me? I don't often check their status.) But it does mean a measure of trust in some external entity, or else some very careful rules (mainly firewall), which not every coder will know about. ChrisA From rajshorya at gmail.com Wed Sep 3 05:09:12 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Wed, 3 Sep 2014 15:09:12 +1200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hello all I don't mind helping out with maintaining buildbots / other build machines. Although I don't have a lot of experience with this sort of thing (I usually just ran a Jenkins for my own work), I think it would be a useful way to contribute. Let me know what I should do if you are all fine with this. Thanks Shorya Raj On Wed, Sep 3, 2014 at 2:52 PM, Chris Angelico wrote: > On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull > wrote: > > Nick Coghlan writes: > > > > > Sorry, I haven't been a very good maintainer for that buildbot (the > main > > > reason it never graduated to the "stable" list). If you send me your > public > > > SSH key, I can add it (I think - if not, I can ask Luke to do it). > > > Alternatively, CentOS 6 may exhibit the same problem. > > > > I wonder how many of these buildbots could be maintained by the kind > > of folks who show up on core-mentorship asking "how can I help?" > > > > Just a thought -- I wouldn't be surprised if the reaction is universal > > horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" > > > > And of course most want to write code, not sysadm. > > Maintaining a buildbot isn't hard. (Although one thing I'm not sure > of: If my bot goes down for an extended period of time, is any sort of > automated email sent to me? I don't often check their status.) But it > does mean a measure of trust in some external entity, or else some > very careful rules (mainly firewall), which not every coder will know > about. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Wed Sep 3 06:58:51 2014 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 3 Sep 2014 14:58:51 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140903045851.GA93101@cskk.homeip.net> On 03Sep2014 11:47, Stephen J. Turnbull wrote: >Nick Coghlan writes: > > Sorry, I haven't been a very good maintainer for that buildbot (the main > > reason it never graduated to the "stable" list). If you send me your public > > SSH key, I can add it (I think - if not, I can ask Luke to do it). > > Alternatively, CentOS 6 may exhibit the same problem. > >I wonder how many of these buildbots could be maintained by the kind >of folks who show up on core-mentorship asking "how can I help?" > >Just a thought -- I wouldn't be surprised if the reaction is universal >horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" > >And of course most want to write code, not sysadm. I do both. Happy to help in a small way if wanted. Cheers, Cameron Simpson Maintainer's Motto: If we can't fix it, it ain't broke. From cory at lukasa.co.uk Wed Sep 3 10:26:44 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 3 Sep 2014 09:26:44 +0100 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903021946.1bbcdd0c@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903021946.1bbcdd0c@fsol> Message-ID: On 3 September 2014 01:19, Antoine Pitrou wrote: > Keeping saying it doesn't make it magically true. Sure, but it *is* true, at the very least for HTTP. RFC 2818 (HTTP over TLS) has the following language in section 3.1: > If the hostname is available, the client MUST check it against the > server's identity as presented in the server's Certificate message, > in order to prevent man-in-the-middle attacks. > > If the client has external information as to the expected identity of > the server, the hostname check MAY be omitted. The default behaviour of httplib is in contravention of the normative language of this specification and is therefore bugged. (For those unclear about the relevance of RFC 2818, it's normatively referenced by RFC 7230, which is the HTTP/1.1 spec.) This should silence the debate about whether or not httplib's behaviour is a bug or not. > Besides, it can perfectly well be a bug fix *as well as* a break in > backwards compatibility. This is definitely true, and this change is both. The only question that matters is whether we believe we're doing users a service by breaking their code. I'd argue, along with Glyph, Alex and Donald, that we are. I've been on the losing side of this debate a number of times though, and I expect I will be again. From christian.staffa at gmx.de Wed Sep 3 10:32:47 2014 From: christian.staffa at gmx.de (Christian Staffa) Date: Wed, 03 Sep 2014 10:32:47 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: Hi all I am using buildbot now for some time and i would be willing to contribute on that. I had small work on openstack buildbot slave but had not the proper infrastructure to get more value out of it. I like that project and automation. Anyway, if i could be of help let me know (; Short to my person: i am christian staffa and actually living in germany. Chris Sent from my Sony Xperia? smartphone ---- Nick Coghlan wrote ---- >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: https://mail.python.org/mailman/options/python-dev/christian.staffa%40gmx.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 3 12:34:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Sep 2014 20:34:32 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903021946.1bbcdd0c@fsol> Message-ID: On 3 Sep 2014 18:28, "Cory Benfield" wrote: > This is definitely true, and this change is both. The only question > that matters is whether we believe we're doing users a service by > breaking their code. I'd argue, along with Glyph, Alex and Donald, > that we are. I've been on the losing side of this debate a number of > times though, and I expect I will be again. The default stdlib behaviour will change in 3.5, I don't think anyone is disputing that. While I earlier said that should depend on the sslcustomize PEP, I now think they should be made orthogonal so the SSL customisation PEP can focus on its potential for *increasing* security in properly configured environments rather than deliberately decreasing it after upgrading to Python 3.5 in improperly configured ones. The backwards compatibility argument only applies to Python 2 maintenance releases (where dreid indicated an intention to request backporting the change), and there I'm quite happy to take the position of "use requests, Twisted or Python 3.5+ to get HTTPS done right". There are a variety of reasons not to use the Python 2 stdlib for modern networking, and making better tools more readily accessible to Python 2 users by backporting ensurepip is my preferred answer. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Sep 3 16:29:15 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 16:29:15 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903021946.1bbcdd0c@fsol> Message-ID: <20140903162915.5321428d@fsol> On Wed, 3 Sep 2014 20:34:32 +1000 Nick Coghlan wrote: > > The backwards compatibility argument only applies to Python 2 maintenance > releases (where dreid indicated an intention to request backporting the > change), and there I'm quite happy to take the position of "use requests, > Twisted or Python 3.5+ to get HTTPS done right". or asyncio. Regards Antoine. From solipsis at pitrou.net Wed Sep 3 16:31:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 16:31:13 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> Message-ID: <20140903163113.2165f744@fsol> On Tue, 02 Sep 2014 21:29:16 -0400 "R. David Murray" wrote: > > The top proposal so far is an sslcustomize.py file that could be used to > either decrease or increase the default security. This is a much less > handy solution than application options (eg, curl, wget) that allow > disabling security for "this cert" or "this CLI session". It also is > more prone to unthinking abuse since it is persistent. So perhaps > it is indeed not worth it. (That's why I suggested an environment > variable...something you could specify on the command line for a one-off.) I'll be fine with not adding any hooks at all, and letting people configure their application code correctly :-) Regards Antoine. From rdmurray at bitdance.com Wed Sep 3 17:58:07 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 03 Sep 2014 11:58:07 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903163113.2165f744@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> Message-ID: <20140903155808.08F50250E04@webabinitio.net> On Wed, 03 Sep 2014 16:31:13 +0200, Antoine Pitrou wrote: > On Tue, 02 Sep 2014 21:29:16 -0400 > "R. David Murray" wrote: > > > > The top proposal so far is an sslcustomize.py file that could be used to > > either decrease or increase the default security. This is a much less > > handy solution than application options (eg, curl, wget) that allow > > disabling security for "this cert" or "this CLI session". It also is > > more prone to unthinking abuse since it is persistent. So perhaps > > it is indeed not worth it. (That's why I suggested an environment > > variable...something you could specify on the command line for a one-off.) > > I'll be fine with not adding any hooks at all, and letting people > configure their application code correctly :-) Again, the problem arises when it is not *their* application code, but a third party tool that hasn't been ported to 3.5. I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. --David From ethan at stoneleaf.us Wed Sep 3 19:09:36 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Sep 2014 10:09:36 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903155808.08F50250E04@webabinitio.net> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: <54074B50.1070208@stoneleaf.us> On 09/03/2014 08:58 AM, R. David Murray wrote: > > I'm OK with letting go of this invalid-cert issue myself, given the lack > of negative feedback Twisted got. I'll just keep my fingers crossed. I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? -- ~Ethan~ From alex.gaynor at gmail.com Wed Sep 3 19:15:13 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 3 Sep 2014 17:15:13 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_476=3A_Enabling_certificate_validation?= =?utf-8?q?_by=09default!?= References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54074B50.1070208@stoneleaf.us> Message-ID: Ethan Furman stoneleaf.us> writes: > > I apologize if I missed this point, but if we have the source code then it is > possible to go in and directly modify the application/utility to be able to > talk over https to a router with an invalid certificate? This is an option > when creating the ssl_context? > > -- > ~Ethan~ > Yes, it's totally possible to create (and pass to ``http.client``) an ``SSLContext`` which doesn't verify various things. My proposal is only about changing what happens when you don't explicitly pass a context. Cheers, Alex From rajshorya at gmail.com Wed Sep 3 07:57:51 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Wed, 3 Sep 2014 17:57:51 +1200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <20140903045851.GA93101@cskk.homeip.net> References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> <20140903045851.GA93101@cskk.homeip.net> Message-ID: As mentioned, I don't mind sysadmining a bit, if required. My primary joy would be helping code python, but can't seem to figure out the ideal place to start doing so. Therefore, helping out as sysadmin would be a good start. On Wed, Sep 3, 2014 at 4:58 PM, Cameron Simpson wrote: > On 03Sep2014 11:47, Stephen J. Turnbull wrote: > >> Nick Coghlan writes: >> > Sorry, I haven't been a very good maintainer for that buildbot (the main >> > reason it never graduated to the "stable" list). If you send me your >> public >> > SSH key, I can add it (I think - if not, I can ask Luke to do it). >> > Alternatively, CentOS 6 may exhibit the same problem. >> >> I wonder how many of these buildbots could be maintained by the kind >> of folks who show up on core-mentorship asking "how can I help?" >> >> Just a thought -- I wouldn't be surprised if the reaction is universal >> horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" >> >> And of course most want to write code, not sysadm. >> > > I do both. Happy to help in a small way if wanted. > > Cheers, > Cameron Simpson > > Maintainer's Motto: If we can't fix it, it ain't broke. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > rajshorya%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 3 17:35:29 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 3 Sep 2014 08:35:29 -0700 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: References: Message-ID: On Sun, Aug 31, 2014 at 3:00 PM, Nick Coghlan wrote: > > However, we still think we should start providing pip by default to Python > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > serious +1 here. Just last night I was writing up notes for an intro to python class on getting started. (py2, :-( ) The "go download get-pip.py and run it with python" step(s) is a serious hiccup. Better than it's been for years, but sure would be nice to just "install python" then "use pip to install packages" As for the feature freeze of py2.7 -- this is really an addition to teh installers, not to python itself (or it can be seen that way, anyway) Thanks Ned for offering to back port for OS-X ! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Sep 3 19:29:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Sep 2014 10:29:42 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54074B50.1070208@stoneleaf.us> Message-ID: <54075006.1060702@stoneleaf.us> On 09/03/2014 10:15 AM, Alex Gaynor wrote: > Ethan Furman writes: >> >> I apologize if I missed this point, but if we have the source code then it is >> possible to go in and directly modify the application/utility to be able to >> talk over https to a router with an invalid certificate? This is an option >> when creating the ssl_context? > > Yes, it's totally possible to create (and pass to ``http.client``) an > ``SSLContext`` which doesn't verify various things. My proposal is only about > changing what happens when you don't explicitly pass a context. Excellent. Last question (I hope): it is possible to (easily) create an SSLContext that will verify against a self-signed certificate? -- ~Ethan~ From francismb at email.de Wed Sep 3 19:32:22 2014 From: francismb at email.de (francis) Date: Wed, 03 Sep 2014 19:32:22 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <540750A6.9020406@email.de> [...] But it > does mean a measure of trust in some external entity, or else some > very careful rules (mainly firewall), which not every coder will know > about. Just curious, is there a way to mount the infrastructure the oder way around? One sets a system polling for sources changes, if so it starts a build an sends per e-mail the results to some address. Thanks in advance! francis From guido at python.org Wed Sep 3 19:54:55 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Sep 2014 10:54:55 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903155808.08F50250E04@webabinitio.net> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray wrote: > I'm OK with letting go of this invalid-cert issue myself, given the lack > of negative feedback Twisted got. I'll just keep my fingers crossed. > I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion. Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue. Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org -- sooner or later some forgotten part of our infrastructure *will* come under attack.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Sep 3 20:37:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 3 Sep 2014 20:37:38 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: <20140903203738.516f34cd@fsol> On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum wrote: > > Let's take the plunge on this issue for the next 2.7 release (3.5 being a > done deal). I'm entirely against this. > Yes, some people will find that they have an old script > accessing an old service which breaks. Surely some of the other changes in > the same 2.7 bugfix release will also break some other scripts. People deal > with it. Probably 90% of the time it's an annoyance (but no worse than any > other minor-release upgrade -- you should test upgrades before committing > to them, and if all else fails, roll it back). Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. > Today (working at Dropbox, a much smaller company!) I don't > even remember the last time I had to deal with such a browser > complaint -- internal services here all redirect to SSL, and not a > browser that can find fault with their certs. Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them). You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously... > But at least some of the > time it will be a wake-up call and an expired certificate will be replaced, > resulting in more security for all. Only if you are actually the one managing that certificate and the machine it's installed one... Regards Antoine. From donald at stufft.io Wed Sep 3 20:39:32 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 3 Sep 2014 14:39:32 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: <6F40DA4F-17C8-4418-BA3C-ACB1330C92DE@stufft.io> > On Sep 3, 2014, at 1:54 PM, Guido van Rossum wrote: > > On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray > wrote: > I'm OK with letting go of this invalid-cert issue myself, given the lack > of negative feedback Twisted got. I'll just keep my fingers crossed. > > I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion. > > Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue. > > Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. +1, this makes me unreasonably happy. > > I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org -- sooner or later some forgotten part of our infrastructure *will* come under attack.) This is an ongoing effort amongst the Infra team, part of the process is moving infrastructure away from hand crafted servers towards servers managed by config management as well as making sure all our services are behind TLS as well. > > -- > --Guido van Rossum (python.org/~guido ) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed Sep 3 21:06:39 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 03 Sep 2014 15:06:39 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903203738.516f34cd@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> Message-ID: <20140903190640.2B11E250E0C@webabinitio.net> On Wed, 03 Sep 2014 20:37:38 +0200, Antoine Pitrou wrote: > On Wed, 3 Sep 2014 10:54:55 -0700 > Guido van Rossum wrote: > > Today (working at Dropbox, a much smaller company!) I don't > > even remember the last time I had to deal with such a browser > > complaint -- internal services here all redirect to SSL, and not a > > browser that can find fault with their certs. > > Good for you. I still sometimes get warnings about expired certificates > - and sometimes ones that don't exactly match the domain being > fetched (for example, the certificate wouldn't be valid for that > specific subdomain - note that CAs often charge a premium for multiple > subdomains, which why small or non-profit Web sites sometimes skimp on > them). > > You shouldn't assume that the experience of well-connected people in > the Silicon Valley is representative of what people over the world > encounter. Yes, where there's a lot of money and a lot of accumulated > domain competence, security procedures are updated and followed more > scrupulously... Heck, yesterday I got invalid certs from...I think it was roku.com, but in any case not some obscure little company...the actual cert was an akamai cert, which means something is configured wrong somewhere. --David From christian at python.org Wed Sep 3 21:07:18 2014 From: christian at python.org (Christian Heimes) Date: Wed, 03 Sep 2014 21:07:18 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <54075006.1060702@stoneleaf.us> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54074B50.1070208@stoneleaf.us> <54075006.1060702@stoneleaf.us> Message-ID: On 03.09.2014 19:29, Ethan Furman wrote: > Excellent. Last question (I hope): it is possible to (easily) create an > SSLContext that will verify against a self-signed certificate? Yes: context = ssl.create_default_context(cafile="/path/to/selfsigned.pem") That works iff the certificate is valid, not expired and its CN or SAN matches the hostname of the service. When the hostname doesn't match then you have to set context.check_hostname = False Christian From rdmurray at bitdance.com Wed Sep 3 21:10:07 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 03 Sep 2014 15:10:07 -0400 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <54074B50.1070208@stoneleaf.us> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54074B50.1070208@stoneleaf.us> Message-ID: <20140903191008.48D13250E0C@webabinitio.net> On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman wrote: > On 09/03/2014 08:58 AM, R. David Murray wrote: > > > > I'm OK with letting go of this invalid-cert issue myself, given the lack > > of negative feedback Twisted got. I'll just keep my fingers crossed. > > I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the > application/utility to be able to talk over https to a router with an invalid certificate? This is an option when > creating the ssl_context? The immediately preceding paragraph that you didn't quote said that the context was 3rd party applications, not source code under your control. Yes, you can (usually) still hack the source, but there are good reasons to prefer to not do that, unfamiliarity with the codebase being just one of them. --David From guido at python.org Wed Sep 3 21:11:49 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Sep 2014 12:11:49 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903203738.516f34cd@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> Message-ID: Antoine, I think we are well past the point where arguments can sway positions. There clearly is no agreement on this issue. So please treat my post as a BDFL tie-breaker. I will just give you one thing to ponder -- those small/non-profit websites that can't afford proper certs are exactly the ones that will be hosting malware soon. Sorry for them, and the certificate vendors certainly aren't in it for charity, but they must fix their certificate issues (and probably improve many other sysadmin practices). --Guido On Wed, Sep 3, 2014 at 11:37 AM, Antoine Pitrou wrote: > On Wed, 3 Sep 2014 10:54:55 -0700 > Guido van Rossum wrote: > > > > Let's take the plunge on this issue for the next 2.7 release (3.5 being a > > done deal). > > I'm entirely against this. > > > Yes, some people will find that they have an old script > > accessing an old service which breaks. Surely some of the other changes > in > > the same 2.7 bugfix release will also break some other scripts. People > deal > > with it. Probably 90% of the time it's an annoyance (but no worse than > any > > other minor-release upgrade -- you should test upgrades before committing > > to them, and if all else fails, roll it back). > > Python is routinely updated to bugfix releases by Linux distributions > and other distribution channels, you usually have no say over what's > shipped in those updates. This is not like changing the major version > used for executing the script, which is normally a manual change. > > > Today (working at Dropbox, a much smaller company!) I don't > > even remember the last time I had to deal with such a browser > > complaint -- internal services here all redirect to SSL, and not a > > browser that can find fault with their certs. > > Good for you. I still sometimes get warnings about expired certificates > - and sometimes ones that don't exactly match the domain being > fetched (for example, the certificate wouldn't be valid for that > specific subdomain - note that CAs often charge a premium for multiple > subdomains, which why small or non-profit Web sites sometimes skimp on > them). > > You shouldn't assume that the experience of well-connected people in > the Silicon Valley is representative of what people over the world > encounter. Yes, where there's a lot of money and a lot of accumulated > domain competence, security procedures are updated and followed more > scrupulously... > > > But at least some of the > > time it will be a wake-up call and an expired certificate will be > replaced, > > resulting in more security for all. > > Only if you are actually the one managing that certificate and the > machine it's installed one... > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Wed Sep 3 21:26:32 2014 From: christian at python.org (Christian Heimes) Date: Wed, 03 Sep 2014 21:26:32 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: <54076B68.6030705@python.org> On 03.09.2014 19:54, Guido van Rossum wrote: > Let's take the plunge on this issue for the next 2.7 release (3.5 being > a done deal). Yes, some people will find that they have an old script > accessing an old service which breaks. Surely some of the other changes > in the same 2.7 bugfix release will also break some other scripts. > People deal with it. Probably 90% of the time it's an annoyance (but no > worse than any other minor-release upgrade -- you should test upgrades > before committing to them, and if all else fails, roll it back). But at > least some of the time it will be a wake-up call and an expired > certificate will be replaced, resulting in more security for all. I'm +1 for Python 3.5 but -1 for Python 2.7. The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly. The default settings must stay until we decide to backport the context argument and have a way to configure the default behavior. Nick and me are planing a PEP. Christian From guido at python.org Wed Sep 3 21:37:24 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Sep 2014 12:37:24 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <54076B68.6030705@python.org> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido On Wed, Sep 3, 2014 at 12:26 PM, Christian Heimes wrote: > On 03.09.2014 19:54, Guido van Rossum wrote: > > Let's take the plunge on this issue for the next 2.7 release (3.5 being > > a done deal). Yes, some people will find that they have an old script > > accessing an old service which breaks. Surely some of the other changes > > in the same 2.7 bugfix release will also break some other scripts. > > People deal with it. Probably 90% of the time it's an annoyance (but no > > worse than any other minor-release upgrade -- you should test upgrades > > before committing to them, and if all else fails, roll it back). But at > > least some of the time it will be a wake-up call and an expired > > certificate will be replaced, resulting in more security for all. > > I'm +1 for Python 3.5 but -1 for Python 2.7. > > The SSLContext backport will landed in Python 2.7.9 (to be released). No > Python 2 user is familiar with the feature yet. But more importantly: > None of the stdlib modules support the new feature, too. httplib, > imaplib ... they all don't take a SSLContext object as an argument. > PEP-466 does not include the backport for the network modules. Without > the context argument there is simply no clean way to configure the SSL > handshake properly. > > The default settings must stay until we decide to backport the context > argument and have a way to configure the default behavior. Nick and me > are planing a PEP. > > Christian > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Sep 3 21:37:40 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 3 Sep 2014 21:37:40 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <54076B68.6030705@python.org> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: 2014-09-03 21:26 GMT+02:00 Christian Heimes : > On 03.09.2014 19:54, Guido van Rossum wrote: > I'm +1 for Python 3.5 but -1 for Python 2.7. > > The SSLContext backport will landed in Python 2.7.9 (to be released). No > Python 2 user is familiar with the feature yet. But more importantly: > None of the stdlib modules support the new feature, too. httplib, > imaplib ... they all don't take a SSLContext object as an argument. > PEP-466 does not include the backport for the network modules. Without > the context argument there is simply no clean way to configure the SSL > handshake properly. Thanks, you replied before I asked the question :-) (If certificates are validated by default, how do you disable the checks?) Sorry, I didn't follow the whole discussion and Python 2.7 changes related to security. Does Python 2.7 support loading (automatically) system certificate authorities? Like the Windows certificate store: https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Victor From ethan at stoneleaf.us Wed Sep 3 21:42:57 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Sep 2014 12:42:57 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903191008.48D13250E0C@webabinitio.net> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54074B50.1070208@stoneleaf.us> <20140903191008.48D13250E0C@webabinitio.net> Message-ID: <54076F41.7020500@stoneleaf.us> On 09/03/2014 12:10 PM, R. David Murray wrote: > On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman wrote: >> On 09/03/2014 08:58 AM, R. David Murray wrote: >>> >>> I'm OK with letting go of this invalid-cert issue myself, given the lack >>> of negative feedback Twisted got. I'll just keep my fingers crossed. >> >> I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the >> application/utility to be able to talk over https to a router with an invalid certificate? This is an option when >> creating the ssl_context? > > The immediately preceding paragraph that you didn't quote said that the > context was 3rd party applications, not source code under your control. > Yes, you can (usually) still hack the source, but there are good reasons to > prefer to not do that, unfamiliarity with the codebase being just one of > them. I appreciate that there is a distinction, yet in most cases we have the source code available (it is the nature of Python) and if push comes to shove (and a bunch of other colloquialisms) then modifying that source code can get you up and running again. -- ~Ethan~ From christian at python.org Wed Sep 3 21:44:14 2014 From: christian at python.org (Christian Heimes) Date: Wed, 03 Sep 2014 21:44:14 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: <54076F8E.8000703@python.org> On 03.09.2014 21:37, Victor Stinner wrote: > Thanks, you replied before I asked the question :-) (If > certificates are validated by default, how do you disable the > checks?) > > Sorry, I didn't follow the whole discussion and Python 2.7 changes > related to security. Does Python 2.7 support loading > (automatically) system certificate authorities? Like the Windows > certificate store: > https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Yes! Alex and others have ported all SSL features back to Python 2.7. The SSL module in Python 2.7.9 will have the same feature set as Python 3.4. Christian From christian at python.org Wed Sep 3 21:50:27 2014 From: christian at python.org (Christian Heimes) Date: Wed, 03 Sep 2014 21:50:27 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: <54077103.1040408@python.org> On 03.09.2014 21:37, Guido van Rossum wrote: > OK, that changes my position for 2.7 (but not for 3.5). I had > assumed there was a way to disable the cert check by changing one > parameter to the urlopen() call. (And I had wanted to add that > there should be a clear FAQ about the subject.) If this isn't > possible that changes the situation. (But I still think that once > we do have that simple change option we should do it, in a later > 2.7 upgrade.) > > I apologize for speaking before I had read all facts, and I'll > await what you and Nick come up with. You are welcome! :) I like to see cert checks in Python 2.7, too. Eventually Python 2.7 should have them enabled by default or at least have one very simple way to enable them globally. Right now we aren't there yet. Perhaps Python 2.7.10 or 2.7.11 will have the necessary bits and backports. Christian From matthew at woodcraft.me.uk Wed Sep 3 22:26:08 2014 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Wed, 3 Sep 2014 20:26:08 +0000 (UTC) Subject: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR References: <20140902164800.1bd76c63@fsol> Message-ID: In article , Victor Stinner wrote: >2014-09-02 23:03 GMT+02:00 Matthew Woodcraft : >> In any case I think PEP 475 should be explaining what is going to happen >> to signal.siginterrupt(). Will setting flag=True be supported? > I first proposed to deprecate the function, but Charles-Fran??ois > thinks that it's unrelated to the PEP (it can be addressed later). > The function will still be available and work. >> If so, will doing so change the behaviour of those parts of the stdlib which >> have already been modified to retry after EINTR? > I think that the stdlib should not handle InterruptedError exception > anymore in the Python code, to simplify the code. That seems good to me. I think it's worth writing in the PEP. But if Python is going to provide its own higher-level model of signal handling, then I think signal.siginterrupt() will need to be documented in terms of that model; it should be saying something along the lines of "if a signal arrives while Python is blocking in a system call then InterruptedError will be raised", and I suppose it will need to say what happens if the signal handler also raised an exception. >> (I think it would be helpful if we could tell people "if you want the >> old EINTR behaviour, just do this simple thing". And I suppose >> siginterrupt flag=True is a candidate for that.) > Why do you want the old behaviour? Purely to avoid breaking existing programs, particularly in ways which will require effort to fix. I think Python's backward-compatibility rules are basically "the behavior of an API must not change without going through the deprecation process". If we're going to be making an exception to that here, then it would be much better to say "here's a simple change to make to keep your old program working", rather than "please rewrite your fiddly signal-handling code to use more modern techniques", however much nicer those modern techniques are. Or perhaps it would be better to not make an exception at all, and instead add an 'interrupt_syscalls' boolean keyword argument to signal.signal(), while strongly recommending that people set it False. -M- From alex.gaynor at gmail.com Wed Sep 3 22:37:48 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 3 Sep 2014 20:37:48 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_476=3A_Enabling_certificate_validation?= =?utf-8?q?_by=09default!?= References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: Guido van Rossum python.org> writes: > OK, that changes my position for 2.7 (but not for 3.5). I had assumed there > was a way to disable the cert check by changing one parameter to the > urlopen() call. (And I had wanted to add that there should be a clear FAQ > about the subject.) If this isn't possible that changes the situation. (But I > still think that once we do have that simple change option we should do it, > in a later 2.7 upgrade.) I apologize for speaking before I had read all > facts, and I'll await what you and Nick come up with. > --Guido This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476? Alex From victor.stinner at gmail.com Wed Sep 3 23:52:55 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 3 Sep 2014 23:52:55 +0200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: 2014-09-03 0:13 GMT+02:00 Victor Stinner : > AMD64 OpenIndiana 3.x: a lot of tests fail with OSError(12, "Not > enough space") or MemoryError. It's probably on issue on the host. > > x86 OpenIndiana 3.x: MemoryError. TestReadline.test_init() also fails. I sent an email to Jesus Cea (owner of these buildbots). > x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't > have to this platform, I don't know how to fix it. I sent my SSH key to Nick (owner). > x86 Windows Server 2003 [SB] 3.x: test_build_ext() of test_distutils > hangs during 1 hang before being killed, it hangs something in the C > compiler. I sent an email to the mailing list of the snakebite network. > x86 XP-4 3.x: test_distutils fails because Visual Studio linker > (link.exe) fails with the error 1181 (cannot open input file). > test_capi.test_forced_io_encoding() fails with an assertion error > because sys.__stdin__ is None. I sent an email to David Bolen (owner). > ARMv7 3.x: "Read-only file system", Mercurial fails... I sent an email to Gregory P. Smith (owner). > x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs. I skipped the test on FreeBSD 7.2 (even if the test pass on FreeBSD 6.4): http://bugs.python.org/issue22331 > x86 FreeBSD 6.4 3.x: (...) test_multiprocessing_spawn fails whereas the > test should be skipped. I skipped the test on FreeBSD 6.4: http://bugs.python.org/issue22332 Victor From stephen at xemacs.org Thu Sep 4 00:48:05 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 04 Sep 2014 07:48:05 +0900 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> Message-ID: <87d2bc8gnu.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > lot: five years ago (when I worked at Google!) it was common to find > internal services that required SSL but had a misconfigured certificate, > and the only way to access those services was to override the browser > complaints. Today (working at Dropbox, a much smaller company!) I don't > even remember the last time I had to deal with such a browser complaint -- I would tend to discount your recent experience, then. Smaller (and possibly even more important in this fast-developing area, younger) organizations are a lot more nimble about things like this. That is not intended to express an opinion about a backport, though. From benjamin at python.org Thu Sep 4 01:10:31 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 03 Sep 2014 16:10:31 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: <1409785831.3031060.163368065.551E64E8@webmail.messagingengine.com> On Wed, Sep 3, 2014, at 13:37, Alex Gaynor wrote: > Guido van Rossum python.org> writes: > > > OK, that changes my position for 2.7 (but not for 3.5). I had assumed there > > was a way to disable the cert check by changing one parameter to the > > urlopen() call. (And I had wanted to add that there should be a clear FAQ > > about the subject.) If this isn't possible that changes the situation. (But I > > still think that once we do have that simple change option we should do it, > > in a later 2.7 upgrade.) I apologize for speaking before I had read all > > facts, and I'll await what you and Nick come up with. > > --Guido > > This probably doesn't surprise anyone, but I'm more than happy to do the > back- > porting work for httplib, and any other modules which need SSLContext > support; > does this require an additional PEP, or does it fit under PEP466 or > PEP476? I imagine you could put it into 476. From ncoghlan at gmail.com Thu Sep 4 01:19:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 09:19:56 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140903203738.516f34cd@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> Message-ID: On 4 Sep 2014 04:39, "Antoine Pitrou" wrote: > > On Wed, 3 Sep 2014 10:54:55 -0700 > Guido van Rossum wrote: > > > > Let's take the plunge on this issue for the next 2.7 release (3.5 being a > > done deal). > > I'm entirely against this. > > > Yes, some people will find that they have an old script > > accessing an old service which breaks. Surely some of the other changes in > > the same 2.7 bugfix release will also break some other scripts. People deal > > with it. Probably 90% of the time it's an annoyance (but no worse than any > > other minor-release upgrade -- you should test upgrades before committing > > to them, and if all else fails, roll it back). > > Python is routinely updated to bugfix releases by Linux distributions > and other distribution channels, you usually have no say over what's > shipped in those updates. This is not like changing the major version > used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. That would make this a case of upstream asking us to do the kind of risk assessment that people pay us for, which is a kind of request I'm generally happy to get :) That way, if downstream users get upset, we can point them at their vendor support department, rather than having them take out their ire on upstream volunteers. Also, after thinking through the implications a bit more, my guess is that Fedora & Software Collections will accept the change without any fuss, but the CentOS/RHEL side could be a more involved discussion. On the other hand, orgs with these kinds of problems aren't likely to have rolled out RHEL 7 or CentOS 7 yet either, so they're probably still back on Python 2.6 (RHEL 6) or even 2.4 (RHEL 5). 2.7.9 is going to be a somewhat "interesting" release that requires careful attention anyway (due to the completion of the PEP 466 backports), so if Guido's OK with it, sure, let's kill the "HTTPS isn't" problem for Python 2 as well. One additional wrinkle in that case: we will likely need to backport the SSLContext related changes to httplib as well. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 4 01:29:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 09:29:00 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <54076B68.6030705@python.org> Message-ID: On 4 Sep 2014 06:39, "Alex Gaynor" wrote: > > Guido van Rossum python.org> writes: > > > OK, that changes my position for 2.7 (but not for 3.5). I had assumed there > > was a way to disable the cert check by changing one parameter to the > > urlopen() call. (And I had wanted to add that there should be a clear FAQ > > about the subject.) If this isn't possible that changes the situation. (But I > > still think that once we do have that simple change option we should do it, > > in a later 2.7 upgrade.) I apologize for speaking before I had read all > > facts, and I'll await what you and Nick come up with. > > --Guido > > This probably doesn't surprise anyone, but I'm more than happy to do the back- > porting work for httplib, and any other modules which need SSLContext support; > does this require an additional PEP, or does it fit under PEP466 or PEP476? I suggest writing up a separate PEP for 2.7 that covers exactly what would need to be backported in order to make the same HTTPS handling change in Python 2. For 476, I suggest taking my list of modules that call "_create_stdlib_cert" and being completely explicit as to which ones are *not* changing (as that will help clarify the scope of the backport proposal). I learned that lesson with PEP 453 - it's well worth making the Python 3 PEP easier to accept by making it independent of the inevitably more controversial Python 2 backport PEP :) Cheers, Nick. > > Alex > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Sep 4 01:36:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Sep 2014 01:36:40 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> Message-ID: <20140904013640.10fdaf3e@fsol> On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan wrote: > > > > Python is routinely updated to bugfix releases by Linux distributions > > and other distribution channels, you usually have no say over what's > > shipped in those updates. This is not like changing the major version > > used for executing the script, which is normally a manual change. > > We can potentially deal with the more conservative part of the user base on > the redistributor side - so long as the PEP says it's OK for us to not > apply this particular change if we deem it appropriate to do so. So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... Of course, people could read distribution-specific package changelogs, but nobody does that. > 2.7.9 is going to be a somewhat "interesting" release that requires careful > attention anyway (due to the completion of the PEP 466 backports), so if > Guido's OK with it, sure, let's kill the "HTTPS isn't" problem for Python 2 > as well. Possible unvoluntary breakage due to a large backport is one thing. Deliberate breakage is another. Regards Antoine. From ethan at stoneleaf.us Thu Sep 4 02:00:39 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Sep 2014 17:00:39 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140904013640.10fdaf3e@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> <20140904013640.10fdaf3e@fsol> Message-ID: <5407ABA7.1080900@stoneleaf.us> On 09/03/2014 04:36 PM, Antoine Pitrou wrote: > On Thu, 4 Sep 2014 09:19:56 +1000 > Nick Coghlan wrote: >>> >>> Python is routinely updated to bugfix releases by Linux distributions >>> and other distribution channels, you usually have no say over what's >>> shipped in those updates. This is not like changing the major version >>> used for executing the script, which is normally a manual change. >> >> We can potentially deal with the more conservative part of the user base on >> the redistributor side - so long as the PEP says it's OK for us to not >> apply this particular change if we deem it appropriate to do so. > > So people would believe python.org that they would get HTTPS cert > validation by default, but their upstream distributor would have > disabled it for them? That's even worse... I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8. -- ~Ethan~ From ncoghlan at gmail.com Thu Sep 4 02:03:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 10:03:09 +1000 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: References: Message-ID: On 1 September 2014 08:00, Nick Coghlan wrote: > Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 > maintenance release in addition to including it with Python 3.4. > > That part of the proposal proved to be controversial, so we dropped it from > the original PEP in order to focus on meeting the Python 3.4 specific > release deadlines. This also had the benefit of working out the kinks in the > bootstrapping processing as part of the Python 3.4 release cycle. > > However, we still think we should start providing pip by default to Python > 2.7 users as well, at least as part of the Windows and Mac OS X installers. I suggested off-list that, as Python 2.7 release manager, Benjamin would be a good candidate for BDFL-Delegate for this proposal, and both Guido and Benjamin agreed. The PEP has been updated accordingly: http://hg.python.org/peps/rev/909954e4ba43 (I also fixed a copy-and-paste error in the PEP status!) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Thu Sep 4 02:17:44 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 03 Sep 2014 17:17:44 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <5407ABA7.1080900@stoneleaf.us> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> <20140904013640.10fdaf3e@fsol> <5407ABA7.1080900@stoneleaf.us> Message-ID: <5407AFA8.2040400@stoneleaf.us> On 09/03/2014 05:00 PM, Ethan Furman wrote: > On 09/03/2014 04:36 PM, Antoine Pitrou wrote: >> On Thu, 4 Sep 2014 09:19:56 +1000 >> Nick Coghlan wrote: >>>> >>>> Python is routinely updated to bugfix releases by Linux distributions >>>> and other distribution channels, you usually have no say over what's >>>> shipped in those updates. This is not like changing the major version >>>> used for executing the script, which is normally a manual change. >>> >>> We can potentially deal with the more conservative part of the user base on >>> the redistributor side - so long as the PEP says it's OK for us to not >>> apply this particular change if we deem it appropriate to do so. >> >> So people would believe python.org that they would get HTTPS cert >> validation by default, but their upstream distributor would have >> disabled it for them? That's even worse... > > I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8. If good argument can be made for why we should make validation by default optional, then that point should be well made in Python's release notes, and some easy programmatic way to tell if validation is on or off (which may just be more docs saying call SSLContext and examine the results: xxx means you're validating, yyy means you are not). -- ~Ethan~ From rosuav at gmail.com Thu Sep 4 03:09:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Sep 2014 11:09:54 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <540750A6.9020406@email.de> References: <87oaux8lo5.fsf@uwakimon.sk.tsukuba.ac.jp> <540750A6.9020406@email.de> Message-ID: On Thu, Sep 4, 2014 at 3:32 AM, francis wrote: >> does mean a measure of trust in some external entity, or else some >> very careful rules (mainly firewall), which not every coder will know >> about. > > > Just curious, is there a way to mount the infrastructure the oder way > around? One sets a system polling for sources changes, if so it starts > a build an sends per e-mail the results to some address. I'm more talking about how there's a (virtual) machine that I run, executing code sent to me by someone else (the Python devs). That machine needs a certain measure of access to the internet (to fetch code, and to run certain tests), and the easy and naive way to set it up is to give it full access to everything, which is the trust that I spoke of. Firewalling that box so it can't see anything it's not allowed to see is certainly possible, but that requires sysadmin skills, not coder skills, hence my comment. And it's very easy to make a mistake that you won't notice - everything works just fine, because normally that buildbot won't do anything it shouldn't. ChrisA From a.cavallo at cavallinux.eu Thu Sep 4 03:07:27 2014 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Thu, 04 Sep 2014 02:07:27 +0100 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: <5407BB4F.1040203@cavallinux.eu> I wonder if there is any interest in starting to use the opensuse build servers for continuous build and testing on redhat, fedora suse and (I think) debian: that will solve once for all the maintenance issues on those platforms (and provide a reliable build). Regards, Antonio From benjamin at python.org Thu Sep 4 04:21:59 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 03 Sep 2014 19:21:59 -0700 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: References: Message-ID: <1409797319.3375822.163417941.4024190B@webmail.messagingengine.com> On Wed, Sep 3, 2014, at 17:03, Nick Coghlan wrote: > On 1 September 2014 08:00, Nick Coghlan wrote: > > Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 > > maintenance release in addition to including it with Python 3.4. > > > > That part of the proposal proved to be controversial, so we dropped it from > > the original PEP in order to focus on meeting the Python 3.4 specific > > release deadlines. This also had the benefit of working out the kinks in the > > bootstrapping processing as part of the Python 3.4 release cycle. > > > > However, we still think we should start providing pip by default to Python > > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > > I suggested off-list that, as Python 2.7 release manager, Benjamin > would be a good candidate for BDFL-Delegate for this proposal, and > both Guido and Benjamin agreed. The PEP has been updated accordingly: > http://hg.python.org/peps/rev/909954e4ba43 (I also fixed a > copy-and-paste error in the PEP status!) ... and accepted. Let's make our users' lives' better!. From ncoghlan at gmail.com Thu Sep 4 05:11:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 13:11:38 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <5407ABA7.1080900@stoneleaf.us> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> <20140904013640.10fdaf3e@fsol> <5407ABA7.1080900@stoneleaf.us> Message-ID: On 4 September 2014 10:00, Ethan Furman wrote: > On 09/03/2014 04:36 PM, Antoine Pitrou wrote: >> >> On Thu, 4 Sep 2014 09:19:56 +1000 >> Nick Coghlan wrote: >>>> >>>> >>>> Python is routinely updated to bugfix releases by Linux distributions >>>> and other distribution channels, you usually have no say over what's >>>> shipped in those updates. This is not like changing the major version >>>> used for executing the script, which is normally a manual change. >>> >>> >>> We can potentially deal with the more conservative part of the user base >>> on >>> the redistributor side - so long as the PEP says it's OK for us to not >>> apply this particular change if we deem it appropriate to do so. >> >> >> So people would believe python.org that they would get HTTPS cert >> validation by default, but their upstream distributor would have >> disabled it for them? That's even worse... > > > I agree. If the vendors don't want to have validation by default, they > should stick with 2.7.8. Yes, that's the way it would work in practice - we'd call it 2.7.8, and backport fixes from upstream 2.7.x as needed (as someone put it to me recently, a useful way to think of component version numbers in RHEL is as referring to the *oldest* pieces of code in that component). I've spent quite a bit more time considering the proposal, and I'm now satisfied that making this change upstream isn't likely to cause any major problems, due to the fact that folks who are likely to suffer from this change aren't likely to even be on 2.7 yet. While we managed not to be last this time, the RHEL/CentOS ecosystem is still a useful benchmark for the roll out of Python versions into conservative enterprise organisations, and the more conservative users within *that* ecosystem are likely to wait for the x.1 release (at the earliest) rather than upgrading as soon as x.0 is out. RHEL 7.0 only came out in June, so most of those conservative environments are still going to be on Python 2.6 in RHEL 6. While we shipped 2.7 support well before the release of RHEL 7 as part of Software Collections and OpenShift, the kinds of environments where properly validating SSL by default may cause problems aren't likely to be on the leading edge of adopting new approaches to software deployment like SCL and PaaS. Fixing the HTTPS validation defaults would have several significant positive consequences: - lowers a major barrier to Python adoption (and continued usage) for public internet focused services - fixes a latent security defect for Python applications operating over the public internet - fixes a latent security defect for Python applications operating in a properly configured intranet environment - reveals a latent security defect for Python applications communicating with improperly configured public internet services - reveals a latent security defect for Python applications operating in improperly configured intranet environments The debate is thus solely over how to manage the consequences of the last two, since going from "silent failure" to "noisy failure" looks like going from "working" to "broken" to someone that isn't already intimately familiar with the underlying issues. That question needs to be considered separately for 3 different versions: - 3.5 - 3.4 - 2.7 3.5 is not controversial, the answer is that the standard library's handling of HTTPS URLs should change to verify certificates properly. No ifs, buts, or maybes - Python 3.5 should automatically verify all HTTPS connections, with explicit developer action required to skip (or otherwise relax) the validation check. So far, we have assumed that 3.4 will get at most a warning. However, I have changed my mind on that, because Python 3 is still largely an early adopter driven technology (it's making inroads into more conservative environments, but it's still at least a few years away from catching up to Python 2 on that front). As a result, the kinds of environments RDM and I are worried about will generally *not* be using Python 3, or if they are, it will be for custom scripts that they can fix. I wouldn't suggest actually making that change without getting an explicit +1 from the Canonical folks (since 3.4 is in Ubuntu LTS 14.04), but I would now personally be +1 on just *fixing it* in 3.4.2, rather than doing a bunch of additional development work purely so we can make folks wait another year for the Python 3 standard library to correctly handle HTTPS. That leaves Python 2.7, and I have to say I'm now persuaded that a backport (including any required httplib and urllib features) is the right way to go. One of the tasks I'd been dreading as a follow-on from PEP 466 was organising the code audit to make sure our existing Python 2 applications are properly configuring SSL. If we instead change Python 2.7.9 to validate certificates by default, then the need to do that audit *goes away*, replaced by the far more mundane tasking of doing integration testing on 2.7.9, which we'd have to do *anyway*. Systematically solving the Python 2 HTTPS problem ceases to be something special, and instead just becomes a regular upstream bug fix that will be covered by our normal regression testing processes. There's also the fact that Python 2.7.9 is becoming, in effect, the 2.8 several folks have been asking for (from a HTTPS perspective, anyway), but done in such a way that it feeds more cleanly into the redistributor channels, rather than having the multi-year lead time (and massive additional overhead) that a 2.8 release would suffer. On the redistributor side, we (as in Red Hat) *specifically* offer paid services to help users manage the risks associated with this kind of change (for example https://access.redhat.com/support/policy/updates/errata#Extended_Update_Support), and we charge them extra for it because the extra complexity it introduces *is* a pain to support. If, as a vendor, we're not willing to do something like that as part of our base subscription, then I don't think upstream should feel any obligation to do it for free - the whole *point* of redistributors from a community perspective is for us to handle the boring & annoying stuff whenever possible, so that upstream don't need to worry about it so much. Seriously - we *don't want* the extremely high touch users that need lots of reassurance as direct upstream consumers, as meeting their expectations requires such a high level of responsiveness and stress that it simply isn't ethical to ask people to do it for free. When someone genuinely *wants* that kind of customer-vendor relationship, trying to employ a colleague-colleague community style relationship instead ends up being incredibly unpleasant for both sides. I freely admit that I'm heavily biased on this point - the fact that self-organising community projects are understandably less interested in the boring & annoying bits that some users want or need is ultimately what pays my salary. But this kind of thing is genuinely difficult for a collaborative community driven project to provide, and I think it's reasonable to expect that people who want a slower pace and/or greater selectivity in their upgrades pay for the privilege (or at least rely on a slower moving, freely available, filtered and curated platform environment like CentOS or Ubuntu LTS). Regards, Nick. P.S. I actually believe this is also why we see so much open source development being done on Mac OS X these days, and even a growing presence of Windows - at the platform level, many open source devs *don't want* a collegial relationship with a community, they want a commercial relationship with a vendor so their computer "just works". Take the mindset many of us have towards our OS, move up the stack a few more layers, and we can see how many task oriented application developers might feel about programming language runtimes, web frameworks, etc - for many folks, they're just tools that help to get a job done, not communities to participate in (and that's OK - it just means we need to fully consider the benefits of relying on commercial rather than community focused approaches to meeting their needs). -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 4 06:16:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 14:16:01 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <5407BB4F.1040203@cavallinux.eu> References: <5407BB4F.1040203@cavallinux.eu> Message-ID: On 4 September 2014 11:07, Antonio Cavallo wrote: > I wonder if there is any interest in starting to use the opensuse build > servers for continuous build and testing on redhat, fedora suse and (I > think) debian: that will solve once for all the maintenance issues on those > platforms (and provide a reliable build). It sounds like a good idea in principle, but I suspect there would be a lot of devils in those details. That doesn't make it a bad idea, just something that would need a volunteer to investigate further before we could form a real opinion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 4 06:18:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 14:18:08 +1000 Subject: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7 In-Reply-To: <1409797319.3375822.163417941.4024190B@webmail.messagingengine.com> References: <1409797319.3375822.163417941.4024190B@webmail.messagingengine.com> Message-ID: On 4 September 2014 12:21, Benjamin Peterson wrote: > On Wed, Sep 3, 2014, at 17:03, Nick Coghlan wrote: >> On 1 September 2014 08:00, Nick Coghlan wrote: >> > Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 >> > maintenance release in addition to including it with Python 3.4. >> > >> > That part of the proposal proved to be controversial, so we dropped it from >> > the original PEP in order to focus on meeting the Python 3.4 specific >> > release deadlines. This also had the benefit of working out the kinks in the >> > bootstrapping processing as part of the Python 3.4 release cycle. >> > >> > However, we still think we should start providing pip by default to Python >> > 2.7 users as well, at least as part of the Windows and Mac OS X installers. >> >> I suggested off-list that, as Python 2.7 release manager, Benjamin >> would be a good candidate for BDFL-Delegate for this proposal, and >> both Guido and Benjamin agreed. The PEP has been updated accordingly: >> http://hg.python.org/peps/rev/909954e4ba43 (I also fixed a >> copy-and-paste error in the PEP status!) > > ... and accepted. Let's make our users' lives' better!. Huzzah! Over to you, Ned & Donald. We can give Steve the good news for the Windows side of things as well :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 4 08:06:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 16:06:50 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> Message-ID: On 4 September 2014 15:45, Shorya Raj wrote: > Also, have you considered using Travis? It would allow us to add to the > configurations, at least generally... Many SaaS solution assume the use of specific centralised identity providers, which isn't a function we're prepared to outsource (we know we need to fix our current proliferation of identity silos, but that's a big project and there are other tasks ahead of it on the todo list for the existing infrastructure team). The limiting factor is generally time - various aspects of the current system are clumsy, but they work, so unless someone is particularly keen and willing to work through all the factors that led to the current setup and propose changes (including at least a rough concept of how ongoing maintenance will work), the status quo wins by default. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 4 09:48:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 17:48:59 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> Message-ID: On 4 September 2014 16:59, Shorya Raj wrote: >> The limiting factor is generally time - various aspects of the current >> system are clumsy, but they work, so unless someone is particularly >> keen and willing to work through all the factors that led to the >> current setup and propose changes (including at least a rough concept >> of how ongoing maintenance will work), the status quo wins by default. > > If you could outline some of these, I would be willing to have a look at > what I could do. One that seems promising to me would be to experiment with dynamic testing in OpenStack: http://docs.buildbot.net/latest/manual/cfg-buildslaves.html#openstack Creating test VMs on demand avoids a lot of the security issues otherwise incurred by someone setting up a persistent build slave and provides a more consistent test environment. https://wiki.python.org/moin/BuildBot gives details of the test system configuration details. Due to the overlap with the infrastructure team (to make use of the Rackspace VM resources), infrastructure at python.org may be a better list to ask for more details on what OpenStack resources we have available (this idea really sits somewhere between infrastructure and python-dev, as Antoine does most of the BuildBot master maintenance). If we could get something like that running, then we might be able to trim the preconfigured buildbot fleet back to just the non-x86 machines. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From a.cavallo at cavallinux.eu Thu Sep 4 10:50:07 2014 From: a.cavallo at cavallinux.eu (A. Cavallo) Date: Thu, 04 Sep 2014 09:50:07 +0100 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> Message-ID: <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> Yes there are "details" indeed. But not show stoppers. A prototype can be seen here: http://cclimited.webfactional.com The nice bit is testing will be executed in a total fresh instance, an added bonus On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan wrote: >On 4 September 2014 11:07, Antonio Cavallo >wrote: >> I wonder if there is any interest in starting to use the opensuse >build >> servers for continuous build and testing on redhat, fedora suse and >(I >> think) debian: that will solve once for all the maintenance issues on >those >> platforms (and provide a reliable build). > >It sounds like a good idea in principle, but I suspect there would be >a lot of devils in those details. That doesn't make it a bad idea, >just something that would need a volunteer to investigate further >before we could form a real opinion. > >Cheers, >Nick. > >-- >Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 4 12:29:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 20:29:46 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> References: <5407BB4F.1040203@cavallinux.eu> <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> Message-ID: On 4 September 2014 18:50, A. Cavallo wrote: > Yes there are "details" indeed. But not show stoppers. A prototype can be > seen here: http://cclimited.webfactional.com > The nice bit is testing will be executed in a total fresh instance, an added > bonus Oh, very nice! What is needed to get that up and running? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Sep 4 14:39:03 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Sep 2014 14:39:03 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> <20140904013640.10fdaf3e@fsol> <5407ABA7.1080900@stoneleaf.us> Message-ID: <20140904143903.5e7b14c5@fsol> On Thu, 4 Sep 2014 13:11:38 +1000 Nick Coghlan wrote: > That leaves Python 2.7, and I have to say I'm now persuaded that a > backport (including any required httplib and urllib features) is the > right way to go. One of the tasks I'd been dreading as a follow-on > from PEP 466 was organising the code audit to make sure our existing > Python 2 applications are properly configuring SSL. If we instead > change Python 2.7.9 to validate certificates by default, then the need > to do that audit *goes away*, replaced by the far more mundane tasking > of doing integration testing on 2.7.9, which we'd have to do *anyway*. What are "our existing Python 2 applications"? Is it a Red Hat-specific statement? What is the "code audit" you are talking about? Regards Antoine. From ncoghlan at gmail.com Thu Sep 4 15:31:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Sep 2014 23:31:14 +1000 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <20140904143903.5e7b14c5@fsol> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <20140903203738.516f34cd@fsol> <20140904013640.10fdaf3e@fsol> <5407ABA7.1080900@stoneleaf.us> <20140904143903.5e7b14c5@fsol> Message-ID: On 4 September 2014 22:39, Antoine Pitrou wrote: > On Thu, 4 Sep 2014 13:11:38 +1000 > Nick Coghlan wrote: >> That leaves Python 2.7, and I have to say I'm now persuaded that a >> backport (including any required httplib and urllib features) is the >> right way to go. One of the tasks I'd been dreading as a follow-on >> from PEP 466 was organising the code audit to make sure our existing >> Python 2 applications are properly configuring SSL. If we instead >> change Python 2.7.9 to validate certificates by default, then the need >> to do that audit *goes away*, replaced by the far more mundane tasking >> of doing integration testing on 2.7.9, which we'd have to do *anyway*. > > What are "our existing Python 2 applications"? Is it a Red Hat-specific > statement? What is the "code audit" you are talking about? Yes, that's a Red Hat specific statement. In addition to OpenStack, many of our other products include Python components. We mostly rely on pycurl, pyopenssl, python-nss, etc for SSL/TLS for historical reasons (since ssl wasn't available in the Python standard library until RHEL 6), which was why these concerns got picked up via the OpenStack community rather than the Fedora community. However, being aware of the problem creates that element of doubt as to whether there are *other* cases where someone used the stdlib ssl module rather than one of the alternatives that verifies HTTPS by default. I'd previously been looking at the question while wearing my upstream core developer & CPython redistributor hats. Glyph et al finally prompted me to look at it while wearing my "automated quality assurance toolchain architect for an application vendor" hat (i.e. my actual day job), and that's a *radically* different perspective when it comes to a language runtime that fails to verify HTTPS connections correctly by default. That's the core reason I changed my mind: this conversation forced me to think through how I could ensure correct HTTPS handling in all our Python code, and the only workable solution I now see is to fix it at the standard library level. The alternative is to prepare for the likelihood of future CVEs in Python based tools based on failure to properly verify SSL certificates, as I don't see a practical way of automatically detecting "tried to use HTTPS without verifying the SSL certificate properly". Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rajshorya at gmail.com Thu Sep 4 07:45:26 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Thu, 4 Sep 2014 17:45:26 +1200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> Message-ID: Also, have you considered using Travis? It would allow us to add to the configurations, at least generally... On Thu, Sep 4, 2014 at 4:16 PM, Nick Coghlan wrote: > On 4 September 2014 11:07, Antonio Cavallo > wrote: > > I wonder if there is any interest in starting to use the opensuse build > > servers for continuous build and testing on redhat, fedora suse and (I > > think) debian: that will solve once for all the maintenance issues on > those > > platforms (and provide a reliable build). > > It sounds like a good idea in principle, but I suspect there would be > a lot of devils in those details. That doesn't make it a bad idea, > just something that would need a volunteer to investigate further > before we could form a real opinion. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajshorya at gmail.com Thu Sep 4 08:59:56 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Thu, 4 Sep 2014 18:59:56 +1200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> Message-ID: > > The limiting factor is generally time - various aspects of the current > system are clumsy, but they work, so unless someone is particularly > keen and willing to work through all the factors that led to the > current setup and propose changes (including at least a rough concept > of how ongoing maintenance will work), the status quo wins by default. If you could outline some of these, I would be willing to have a look at what I could do. On Thu, Sep 4, 2014 at 6:06 PM, Nick Coghlan wrote: > On 4 September 2014 15:45, Shorya Raj wrote: > > Also, have you considered using Travis? It would allow us to add to the > > configurations, at least generally... > > Many SaaS solution assume the use of specific centralised identity > providers, which isn't a function we're prepared to outsource (we know > we need to fix our current proliferation of identity silos, but that's > a big project and there are other tasks ahead of it on the todo list > for the existing infrastructure team). > > The limiting factor is generally time - various aspects of the current > system are clumsy, but they work, so unless someone is particularly > keen and willing to work through all the factors that led to the > current setup and propose changes (including at least a rough concept > of how ongoing maintenance will work), the status quo wins by default. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajshorya at gmail.com Thu Sep 4 11:07:44 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Thu, 4 Sep 2014 21:07:44 +1200 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> References: <5407BB4F.1040203@cavallinux.eu> <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> Message-ID: And so what would need to be done to rectify this problem? If there are no objections, I would like to resolve this particular point, at least until better people can be found to do so. On Thu, Sep 4, 2014 at 8:50 PM, A. Cavallo wrote: > Yes there are "details" indeed. But not show stoppers. A prototype can be > seen here: http://cclimited.webfactional.com > The nice bit is testing will be executed in a total fresh instance, an > added bonus > > On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan > wrote: >> >> On 4 September 2014 11:07, Antonio Cavallo wrote: >> >>> >>> I wonder if there is any interest in starting to use the opensuse build >>> servers for continuous build and testing on redhat, fedora suse and (I >>> think) debian: that will solve once for all the maintenance issues on those >>> >>> platforms (and provide a reliable build). >>> >> >> It sounds like a good idea in principle, but I suspect there would be >> a lot of devils in those details. That doesn't make it a bad idea, >> just something that would need a volunteer to investigate further >> >> before we could form a real opinion. >> >> Cheers, >> Nick. >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.wagner at epfl.ch Thu Sep 4 12:06:25 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Thu, 4 Sep 2014 12:06:25 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: References: Message-ID: Hi, Thanks for your comments! I've done more tests on a new machine and, for better or worse, was no longer able to reproduce the issue. Parallel compilation now works fine; I'll keep collecting output from make, and would post it here if the problem appears again. One additional issue appeared, though: it seems that extension modules are always built sequentially no matter what the value of MAKEFLAGS is. I've seen that these are being built by a custom setup.py script. Do you think it would be possible to parallelize this script? Would people be interested in having a parallel version? Cheers, Jonas -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Sep 4 18:09:31 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Sep 2014 18:09:31 +0200 Subject: [Python-Dev] cpython and parallel make References: Message-ID: <20140904180931.46ad9be0@fsol> On Thu, 4 Sep 2014 12:06:25 +0200 Jonas Wagner wrote: > > One additional issue appeared, though: it seems that extension modules are > always built sequentially no matter what the value of MAKEFLAGS is. I've > seen that these are being built by a custom setup.py script. Do you think > it would be possible to parallelize this script? Would people be interested > in having a parallel version? See http://bugs.python.org/issue5309 Regards Antoine. From a.cavallo at cavallinux.eu Fri Sep 5 03:33:32 2014 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Fri, 05 Sep 2014 02:33:32 +0100 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: <5407BB4F.1040203@cavallinux.eu> <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> Message-ID: <540912EC.9000602@cavallinux.eu> Setting up a repo with the code and cleaning a bit here and there. Over the weekend I can put something useable. Nick Coghlan wrote: > On 4 September 2014 18:50, A. Cavallo wrote: >> Yes there are "details" indeed. But not show stoppers. A prototype can be >> seen here: http://cclimited.webfactional.com >> The nice bit is testing will be executed in a total fresh instance, an added >> bonus > > Oh, very nice! > > What is needed to get that up and running? > > Regards, > Nick. > From status at bugs.python.org Fri Sep 5 18:08:11 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 5 Sep 2014 18:08:11 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140905160811.9AAC95641A@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-08-29 - 2014-09-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4640 ( +2) closed 29471 (+40) total 34111 (+42) Open issues with patches: 2191 Issues opened (29) ================== #16037: httplib: header parsing is unlimited http://bugs.python.org/issue16037 reopened by Arfrever #22301: smtplib.SMTP.starttls' documentation is just confusing http://bugs.python.org/issue22301 opened by maker #22302: Windows os.path.isabs UNC path bug http://bugs.python.org/issue22302 opened by akima #22303: Write better test for SSLContext.load_verify_locations http://bugs.python.org/issue22303 opened by pitrou #22305: Documentation on deepcopy's problems is misleading http://bugs.python.org/issue22305 opened by shlsh #22308: add {implementation} to sysconfig.py http://bugs.python.org/issue22308 opened by mattip #22309: distutils/spawn.py handle fork() not implemented. http://bugs.python.org/issue22309 opened by John.Malmberg #22310: Report actual EOF character instead of assuming Ctrl-D http://bugs.python.org/issue22310 opened by John.Malmberg #22313: Make PYLONG_BITS_IN_DIGIT always available to non-core extensi http://bugs.python.org/issue22313 opened by scoder #22314: pydoc.py: TypeError with a $LINES defined to anything http://bugs.python.org/issue22314 opened by arigo #22317: action argument is not documented in argparse add_subparser() http://bugs.python.org/issue22317 opened by Ubik #22319: mailbox.MH chokes on directories without .mh_sequences http://bugs.python.org/issue22319 opened by gumnos #22323: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString( http://bugs.python.org/issue22323 opened by haypo #22324: Use PyUnicode_AsWideCharString() instead of PyUnicode_AsUnicod http://bugs.python.org/issue22324 opened by haypo #22326: tempfile.TemporaryFile fails on NFS v4 filesystems http://bugs.python.org/issue22326 opened by drosera #22327: test_gdb failures on Ubuntu 14.10 http://bugs.python.org/issue22327 opened by barry #22329: Windows installer can't recover partially installed state http://bugs.python.org/issue22329 opened by LlelanD #22330: PyOS_mystricmp is broken http://bugs.python.org/issue22330 opened by kakkoko #22331: test_io.test_interrupted_write_text() hangs on the buildbot Fr http://bugs.python.org/issue22331 opened by haypo #22332: test_multiprocessing_main_handling fail on buildbot "x86 FreeB http://bugs.python.org/issue22332 opened by haypo #22333: test_threaded_import.test_parallel_meta_path() failed on x86 W http://bugs.python.org/issue22333 opened by haypo #22334: test_tcl.test_split() fails on "x86 FreeBSD 7.2 3.x" buildbot http://bugs.python.org/issue22334 opened by haypo #22335: Python 3: Segfault instead of MemoryError when bytearray too b http://bugs.python.org/issue22335 opened by swanson #22336: _tkinter should use Python PyMem_Malloc() instead of Tcl ckall http://bugs.python.org/issue22336 opened by haypo #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 opened by haypo #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 opened by Walkman #22340: Fix Python 3 warnings in Python 2 tests http://bugs.python.org/issue22340 opened by haypo #22341: Python 3 crc32 documentation clarifications http://bugs.python.org/issue22341 opened by vadmium #22342: Fix typo in PEP 380 -- Syntax for Delegating to a Subgenerator http://bugs.python.org/issue22342 opened by Gael.Robin Most recent 15 issues with no replies (15) ========================================== #22342: Fix typo in PEP 380 -- Syntax for Delegating to a Subgenerator http://bugs.python.org/issue22342 #22341: Python 3 crc32 documentation clarifications http://bugs.python.org/issue22341 #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 #22327: test_gdb failures on Ubuntu 14.10 http://bugs.python.org/issue22327 #22317: action argument is not documented in argparse add_subparser() http://bugs.python.org/issue22317 #22314: pydoc.py: TypeError with a $LINES defined to anything http://bugs.python.org/issue22314 #22313: Make PYLONG_BITS_IN_DIGIT always available to non-core extensi http://bugs.python.org/issue22313 #22303: Write better test for SSLContext.load_verify_locations http://bugs.python.org/issue22303 #22300: PEP 446 What's New Updates for 2.7.9 http://bugs.python.org/issue22300 #22294: 2to3 consuming_calls: len, min, max, zip, map, reduce, filter http://bugs.python.org/issue22294 #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 #22286: Allow backslashreplace error handler to be used on input http://bugs.python.org/issue22286 #22268: mrohasattr and mrogetattr http://bugs.python.org/issue22268 #22255: Multiprocessing freeze_support raises RuntimeError http://bugs.python.org/issue22255 #22251: Various markup errors in documentation http://bugs.python.org/issue22251 Most recent 15 issues waiting for review (15) ============================================= #22340: Fix Python 3 warnings in Python 2 tests http://bugs.python.org/issue22340 #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 #22336: _tkinter should use Python PyMem_Malloc() instead of Tcl ckall http://bugs.python.org/issue22336 #22330: PyOS_mystricmp is broken http://bugs.python.org/issue22330 #22324: Use PyUnicode_AsWideCharString() instead of PyUnicode_AsUnicod http://bugs.python.org/issue22324 #22323: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString( http://bugs.python.org/issue22323 #22319: mailbox.MH chokes on directories without .mh_sequences http://bugs.python.org/issue22319 #22308: add {implementation} to sysconfig.py http://bugs.python.org/issue22308 #22301: smtplib.SMTP.starttls' documentation is just confusing http://bugs.python.org/issue22301 #22300: PEP 446 What's New Updates for 2.7.9 http://bugs.python.org/issue22300 #22294: 2to3 consuming_calls: len, min, max, zip, map, reduce, filter http://bugs.python.org/issue22294 #22292: pickle whichmodule RuntimeError http://bugs.python.org/issue22292 #22290: "AMD64 OpenIndiana 3.x" buildbot: assertion failed in PyObject http://bugs.python.org/issue22290 #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 Top 10 most discussed issues (10) ================================= #22194: access to cdecimal / libmpdec API http://bugs.python.org/issue22194 11 msgs #21951: tcl test change crashes AIX http://bugs.python.org/issue21951 8 msgs #22298: Lib/warnings.py _show_warning does not protect against being c http://bugs.python.org/issue22298 6 msgs #22299: resolve() on Windows makes some pathological paths unusable http://bugs.python.org/issue22299 6 msgs #22334: test_tcl.test_split() fails on "x86 FreeBSD 7.2 3.x" buildbot http://bugs.python.org/issue22334 6 msgs #17620: Python interactive console doesn't use sys.stdin for input http://bugs.python.org/issue17620 5 msgs #22043: Use a monotonic clock to compute timeouts http://bugs.python.org/issue22043 5 msgs #22166: test_codecs leaks references http://bugs.python.org/issue22166 5 msgs #22257: PEP 432: Redesign the interpreter startup sequence http://bugs.python.org/issue22257 5 msgs #22297: 2.7 json encoding broken for enums http://bugs.python.org/issue22297 5 msgs Issues closed (39) ================== #1598: unexpected response in imaplib http://bugs.python.org/issue1598 closed by terry.reedy #5833: readline update http://bugs.python.org/issue5833 closed by terry.reedy #6588: insert cookies into cookie jar - cookielib.py http://bugs.python.org/issue6588 closed by terry.reedy #10240: dict.update.__doc__ is misleading http://bugs.python.org/issue10240 closed by rhettinger #15418: 2to3 docs should mention setup.py fixes required to install co http://bugs.python.org/issue15418 closed by terry.reedy #16040: nntplib: unlimited readline() from connection http://bugs.python.org/issue16040 closed by terry.reedy #19461: RawConfigParser modifies empty strings unconditionally http://bugs.python.org/issue19461 closed by lukasz.langa #19546: configparser leaks implementation detail http://bugs.python.org/issue19546 closed by lukasz.langa #19826: Document that bug reporting by email is possible http://bugs.python.org/issue19826 closed by brett.cannon #20421: expose SSL socket protocol version http://bugs.python.org/issue20421 closed by pitrou #20957: test_smptnet Fail instead of Skip if SSL-port is unavailable http://bugs.python.org/issue20957 closed by haypo #21186: RawConfigParser __name__ option handling inconsistent http://bugs.python.org/issue21186 closed by lukasz.langa #21204: multiprocessing example does not work on Windows http://bugs.python.org/issue21204 closed by terry.reedy #21527: concurrent.futures.ThreadPoolExecutor does not use a default v http://bugs.python.org/issue21527 closed by python-dev #21933: Allow the user to change font sizes with the text pane of turt http://bugs.python.org/issue21933 closed by terry.reedy #22098: Behavior of Structure inconsistent with BigEndianStructure whe http://bugs.python.org/issue22098 closed by pitrou #22185: Occasional RuntimeError from Condition.notify http://bugs.python.org/issue22185 closed by pitrou #22216: smtplip STARTTLS fails at second attampt due to unsufficiant q http://bugs.python.org/issue22216 closed by r.david.murray #22221: ast.literal_eval confused by coding declarations http://bugs.python.org/issue22221 closed by serhiy.storchaka #22258: set_inheritable(): ioctl(FIOCLEX) is available but fails with http://bugs.python.org/issue22258 closed by python-dev #22270: cache version selection for documentation http://bugs.python.org/issue22270 closed by terry.reedy #22275: asyncio: enhance documentation of OS support http://bugs.python.org/issue22275 closed by python-dev #22283: "AMD64 FreeBSD 9.0 3.x" fails to build the _decimal module: #e http://bugs.python.org/issue22283 closed by skrah #22296: cookielib uses time.time(), making incorrect checks of expirat http://bugs.python.org/issue22296 closed by regu0004 #22304: Update multiprocessing examples to Py3 and test http://bugs.python.org/issue22304 closed by terry.reedy #22306: Inconsistent division by 0 behaviour in decimal module http://bugs.python.org/issue22306 closed by rhettinger #22307: os.getlogin() documentation has misleading side note http://bugs.python.org/issue22307 closed by python-dev #22311: Pip 404's http://bugs.python.org/issue22311 closed by pitrou #22312: ntpath.splitdrive('//') -> IndexError http://bugs.python.org/issue22312 closed by python-dev #22315: TypeError in error handling in distutils.dir_util.copy_tree http://bugs.python.org/issue22315 closed by python-dev #22316: Add rule about "extraneous whitespace around colon" to "Whites http://bugs.python.org/issue22316 closed by gvanrossum #22318: unitest documentation on assertItemsEqual misleading http://bugs.python.org/issue22318 closed by r.david.murray #22320: Invalid link in General Python FAQ http://bugs.python.org/issue22320 closed by ned.deily #22321: odd result for datetime.time.strftime("%s") http://bugs.python.org/issue22321 closed by r.david.murray #22322: Zip files created by `git archive` result in a SyntaxError (du http://bugs.python.org/issue22322 closed by r.david.murray #22325: wrong subtraction result http://bugs.python.org/issue22325 closed by r.david.murray #22328: ur'foo\d' raises SyntaxError http://bugs.python.org/issue22328 closed by r.david.murray #22337: Modulus not returning proper remainder http://bugs.python.org/issue22337 closed by r.david.murray #1660009: continuing problem with httplib multiple set-cookie headers http://bugs.python.org/issue1660009 closed by terry.reedy From jonas.wagner at epfl.ch Fri Sep 5 10:37:18 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Fri, 5 Sep 2014 10:37:18 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: <20140904180931.46ad9be0@fsol> References: <20140904180931.46ad9be0@fsol> Message-ID: > > Would people be interested in having a parallel version? > > See http://bugs.python.org/issue5309 Cool! I'll look into this. Just as I was thinking that the parallel build problems are solved, the issue surfaced again. I've attached a complete, unmodified log to this mail. Some notes: - asap-clang is a wrapper around Clang that I use for a research project. It should behave the same way as Clang in this case. If anything, it might be less threadsafe. - commands such as configure_and_build_python are from a shell script that invokes configure, make, etc. - This is running on Ubuntu 14.04 with Clang 3.5 and GNU Make 3.81, and a fresh checkout of Python 3.4 - I see two compiler invocations for Parser/pgen.c . Not sure why this happens. It seems to cause the pgen.o file to be empty or truncated. Any ideas? Cheers, Jonas -------------- next part -------------- 16:30:06 + mkdir cpython-asan-build 16:30:06 + cd cpython-asan-build 16:30:06 + asap-clang -asap-init 16:30:06 # Initialized ASAP state in /home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/asap_state. Now run: 16:30:06 export ASAP_STATE_PATH="/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/asap_state" 16:30:06 ++ pwd 16:30:06 + export ASAP_STATE_PATH=/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/asap_state 16:30:06 + ASAP_STATE_PATH=/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/asap_state 16:30:06 + configure_and_build_python '-O3 -fsanitize=address' -fsanitize=address 16:30:06 + configure_python '-O3 -fsanitize=address' -fsanitize=address 16:30:06 + cflags='-O3 -fsanitize=address' 16:30:06 + ldflags=-fsanitize=address 16:30:06 + mkdir build 16:30:06 + cd build 16:30:06 ++ which asap-clang 16:30:06 ++ which asap-clang++ 16:30:06 ++ which asap-ar 16:30:06 ++ which asap-ranlib 16:30:06 + ../../cpython/configure --without-pymalloc --disable-shared --disable-ipv6 CC=/home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang CXX=/home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang++ AR=/home/jenkins/workspace/asap-build/llvm-build/bin/asap-ar RANLIB=/home/jenkins/workspace/asap-build/llvm-build/bin/asap-ranlib 'CFLAGS=-O3 -fsanitize=address' LDFLAGS=-fsanitize=address 16:30:07 checking for hg... found 16:30:07 checking build system type... x86_64-unknown-linux-gnu 16:30:07 checking host system type... x86_64-unknown-linux-gnu 16:30:07 checking for --enable-universalsdk... no 16:30:07 checking for --with-universal-archs... no 16:30:07 checking MACHDEP... linux 16:30:07 checking for --without-gcc... no 16:30:07 checking for gcc... /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang 16:30:07 checking whether the C compiler works... yes 16:30:08 checking for C compiler default output file name... a.out 16:30:08 checking for suffix of executables... 16:30:08 checking whether we are cross compiling... no 16:30:08 checking for suffix of object files... o 16:30:08 checking whether we are using the GNU C compiler... yes 16:30:08 checking whether /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang accepts -g... yes 16:30:08 checking for /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang option to accept ISO C89... none needed 16:30:09 checking for --with-cxx-main=... no 16:30:09 checking for -Wl,--no-as-needed... yes 16:30:09 checking how to run the C preprocessor... /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -E 16:30:09 checking for grep that handles long lines and -e... /bin/grep 16:30:09 checking for egrep... /bin/grep -E 16:30:09 checking for ANSI C header files... yes 16:30:10 checking for sys/types.h... yes 16:30:11 checking for sys/stat.h... yes 16:30:11 checking for stdlib.h... yes 16:30:11 checking for string.h... yes 16:30:11 checking for memory.h... yes 16:30:12 checking for strings.h... yes 16:30:12 checking for inttypes.h... yes 16:30:12 checking for stdint.h... yes 16:30:12 checking for unistd.h... yes 16:30:13 checking minix/config.h usability... no 16:30:13 checking minix/config.h presence... no 16:30:13 checking for minix/config.h... no 16:30:13 checking whether it is safe to define __EXTENSIONS__... yes 16:30:13 checking for --with-suffix... 16:30:13 checking for case-insensitive build directory... no 16:30:14 checking LIBRARY... libpython$(VERSION)$(ABIFLAGS).a 16:30:14 checking LINKCC... $(PURIFY) $(MAINCC) 16:30:14 checking for GNU ld... yes 16:30:14 checking for inline... inline 16:30:14 checking for --enable-shared... no 16:30:14 checking for --enable-profiling... no 16:30:14 checking LDLIBRARY... libpython$(VERSION)$(ABIFLAGS).a 16:30:14 checking for ranlib... /home/jenkins/workspace/asap-build/llvm-build/bin/asap-ranlib 16:30:14 checking for readelf... readelf 16:30:14 checking for python3.4... python3.4 16:30:14 checking for a BSD-compatible install... /usr/bin/install -c 16:30:14 checking for a thread-safe mkdir -p... /bin/mkdir -p 16:30:14 checking for --with-pydebug... no 16:30:14 checking whether /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang accepts and needs -fno-strict-aliasing... no 16:30:14 checking if we can turn off /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang unused result warning... yes 16:30:14 checking for -Werror=declaration-after-statement... yes 16:30:14 checking whether pthreads are available without options... yes 16:30:15 checking whether /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang++ also accepts flags for thread support... no 16:30:15 checking for ANSI C header files... (cached) yes 16:30:15 checking asm/types.h usability... yes 16:30:15 checking asm/types.h presence... yes 16:30:15 checking for asm/types.h... yes 16:30:15 checking conio.h usability... no 16:30:15 checking conio.h presence... no 16:30:16 checking for conio.h... no 16:30:16 checking direct.h usability... no 16:30:16 checking direct.h presence... no 16:30:16 checking for direct.h... no 16:30:16 checking dlfcn.h usability... yes 16:30:16 checking dlfcn.h presence... yes 16:30:16 checking for dlfcn.h... yes 16:30:16 checking errno.h usability... yes 16:30:17 checking errno.h presence... yes 16:30:17 checking for errno.h... yes 16:30:17 checking fcntl.h usability... yes 16:30:17 checking fcntl.h presence... yes 16:30:17 checking for fcntl.h... yes 16:30:17 checking grp.h usability... yes 16:30:18 checking grp.h presence... yes 16:30:18 checking for grp.h... yes 16:30:18 checking ieeefp.h usability... no 16:30:18 checking ieeefp.h presence... no 16:30:18 checking for ieeefp.h... no 16:30:18 checking io.h usability... no 16:30:18 checking io.h presence... no 16:30:19 checking for io.h... no 16:30:19 checking langinfo.h usability... yes 16:30:19 checking langinfo.h presence... yes 16:30:19 checking for langinfo.h... yes 16:30:19 checking libintl.h usability... yes 16:30:19 checking libintl.h presence... yes 16:30:19 checking for libintl.h... yes 16:30:19 checking process.h usability... no 16:30:20 checking process.h presence... no 16:30:20 checking for process.h... no 16:30:20 checking pthread.h usability... yes 16:30:20 checking pthread.h presence... yes 16:30:20 checking for pthread.h... yes 16:30:20 checking sched.h usability... yes 16:30:21 checking sched.h presence... yes 16:30:21 checking for sched.h... yes 16:30:21 checking shadow.h usability... yes 16:30:21 checking shadow.h presence... yes 16:30:21 checking for shadow.h... yes 16:30:21 checking signal.h usability... yes 16:30:21 checking signal.h presence... yes 16:30:22 checking for signal.h... yes 16:30:22 checking for stdint.h... (cached) yes 16:30:22 checking stropts.h usability... yes 16:30:22 checking stropts.h presence... yes 16:30:22 checking for stropts.h... yes 16:30:22 checking termios.h usability... yes 16:30:22 checking termios.h presence... yes 16:30:22 checking for termios.h... yes 16:30:22 checking for unistd.h... (cached) yes 16:30:22 checking utime.h usability... yes 16:30:23 checking utime.h presence... yes 16:30:23 checking for utime.h... yes 16:30:23 checking poll.h usability... yes 16:30:23 checking poll.h presence... yes 16:30:23 checking for poll.h... yes 16:30:23 checking sys/devpoll.h usability... no 16:30:24 checking sys/devpoll.h presence... no 16:30:24 checking for sys/devpoll.h... no 16:30:24 checking sys/epoll.h usability... yes 16:30:24 checking sys/epoll.h presence... yes 16:30:24 checking for sys/epoll.h... yes 16:30:24 checking sys/poll.h usability... yes 16:30:24 checking sys/poll.h presence... yes 16:30:25 checking for sys/poll.h... yes 16:30:25 checking sys/audioio.h usability... no 16:30:25 checking sys/audioio.h presence... no 16:30:25 checking for sys/audioio.h... no 16:30:25 checking sys/xattr.h usability... yes 16:30:25 checking sys/xattr.h presence... yes 16:30:25 checking for sys/xattr.h... yes 16:30:25 checking sys/bsdtty.h usability... no 16:30:26 checking sys/bsdtty.h presence... no 16:30:26 checking for sys/bsdtty.h... no 16:30:26 checking sys/event.h usability... no 16:30:26 checking sys/event.h presence... no 16:30:26 checking for sys/event.h... no 16:30:26 checking sys/file.h usability... yes 16:30:27 checking sys/file.h presence... yes 16:30:27 checking for sys/file.h... yes 16:30:27 checking sys/ioctl.h usability... yes 16:30:27 checking sys/ioctl.h presence... yes 16:30:27 checking for sys/ioctl.h... yes 16:30:27 checking sys/kern_control.h usability... no 16:30:27 checking sys/kern_control.h presence... no 16:30:28 checking for sys/kern_control.h... no 16:30:28 checking sys/loadavg.h usability... no 16:30:28 checking sys/loadavg.h presence... no 16:30:28 checking for sys/loadavg.h... no 16:30:28 checking sys/lock.h usability... no 16:30:28 checking sys/lock.h presence... no 16:30:28 checking for sys/lock.h... no 16:30:28 checking sys/mkdev.h usability... no 16:30:29 checking sys/mkdev.h presence... no 16:30:29 checking for sys/mkdev.h... no 16:30:29 checking sys/modem.h usability... no 16:30:29 checking sys/modem.h presence... no 16:30:29 checking for sys/modem.h... no 16:30:29 checking sys/param.h usability... yes 16:30:29 checking sys/param.h presence... yes 16:30:30 checking for sys/param.h... yes 16:30:30 checking sys/select.h usability... yes 16:30:30 checking sys/select.h presence... yes 16:30:30 checking for sys/select.h... yes 16:30:30 checking sys/sendfile.h usability... yes 16:30:30 checking sys/sendfile.h presence... yes 16:30:30 checking for sys/sendfile.h... yes 16:30:30 checking sys/socket.h usability... yes 16:30:31 checking sys/socket.h presence... yes 16:30:31 checking for sys/socket.h... yes 16:30:31 checking sys/statvfs.h usability... yes 16:30:31 checking sys/statvfs.h presence... yes 16:30:31 checking for sys/statvfs.h... yes 16:30:31 checking for sys/stat.h... (cached) yes 16:30:31 checking sys/syscall.h usability... yes 16:30:32 checking sys/syscall.h presence... yes 16:30:32 checking for sys/syscall.h... yes 16:30:32 checking sys/sys_domain.h usability... no 16:30:32 checking sys/sys_domain.h presence... no 16:30:32 checking for sys/sys_domain.h... no 16:30:32 checking sys/termio.h usability... no 16:30:32 checking sys/termio.h presence... no 16:30:33 checking for sys/termio.h... no 16:30:33 checking sys/time.h usability... yes 16:30:33 checking sys/time.h presence... yes 16:30:33 checking for sys/time.h... yes 16:30:33 checking sys/times.h usability... yes 16:30:33 checking sys/times.h presence... yes 16:30:33 checking for sys/times.h... yes 16:30:33 checking for sys/types.h... (cached) yes 16:30:33 checking sys/uio.h usability... yes 16:30:34 checking sys/uio.h presence... yes 16:30:34 checking for sys/uio.h... yes 16:30:34 checking sys/un.h usability... yes 16:30:34 checking sys/un.h presence... yes 16:30:34 checking for sys/un.h... yes 16:30:34 checking sys/utsname.h usability... yes 16:30:35 checking sys/utsname.h presence... yes 16:30:35 checking for sys/utsname.h... yes 16:30:35 checking sys/wait.h usability... yes 16:30:35 checking sys/wait.h presence... yes 16:30:35 checking for sys/wait.h... yes 16:30:35 checking pty.h usability... yes 16:30:35 checking pty.h presence... yes 16:30:36 checking for pty.h... yes 16:30:36 checking libutil.h usability... no 16:30:36 checking libutil.h presence... no 16:30:36 checking for libutil.h... no 16:30:36 checking sys/resource.h usability... yes 16:30:36 checking sys/resource.h presence... yes 16:30:36 checking for sys/resource.h... yes 16:30:36 checking netpacket/packet.h usability... yes 16:30:37 checking netpacket/packet.h presence... yes 16:30:37 checking for netpacket/packet.h... yes 16:30:37 checking sysexits.h usability... yes 16:30:37 checking sysexits.h presence... yes 16:30:37 checking for sysexits.h... yes 16:30:37 checking bluetooth.h usability... no 16:30:38 checking bluetooth.h presence... no 16:30:38 checking for bluetooth.h... no 16:30:38 checking bluetooth/bluetooth.h usability... no 16:30:38 checking bluetooth/bluetooth.h presence... no 16:30:38 checking for bluetooth/bluetooth.h... no 16:30:38 checking linux/tipc.h usability... yes 16:30:38 checking linux/tipc.h presence... yes 16:30:38 checking for linux/tipc.h... yes 16:30:39 checking spawn.h usability... yes 16:30:39 checking spawn.h presence... yes 16:30:39 checking for spawn.h... yes 16:30:39 checking util.h usability... no 16:30:39 checking util.h presence... no 16:30:39 checking for util.h... no 16:30:39 checking alloca.h usability... yes 16:30:40 checking alloca.h presence... yes 16:30:40 checking for alloca.h... yes 16:30:40 checking endian.h usability... yes 16:30:40 checking endian.h presence... yes 16:30:40 checking for endian.h... yes 16:30:40 checking sys/endian.h usability... no 16:30:41 checking sys/endian.h presence... no 16:30:41 checking for sys/endian.h... no 16:30:41 checking for dirent.h that defines DIR... yes 16:30:41 checking for library containing opendir... none required 16:30:41 checking whether sys/types.h defines makedev... yes 16:30:41 checking for net/if.h... yes 16:30:42 checking for linux/netlink.h... yes 16:30:42 checking for linux/can.h... yes 16:30:42 checking for linux/can/raw.h... yes 16:30:42 checking for linux/can/bcm.h... yes 16:30:42 checking for clock_t in time.h... yes 16:30:42 checking for makedev... yes 16:30:43 checking for le64toh... yes 16:30:43 checking Solaris LFS bug... no 16:30:43 checking for mode_t... yes 16:30:44 checking for off_t... yes 16:30:44 checking for pid_t... yes 16:30:45 checking for size_t... yes 16:30:46 checking for uid_t in sys/types.h... yes 16:30:46 checking for uint32_t... yes 16:30:46 checking for uint32_t... yes 16:30:47 checking for uint64_t... yes 16:30:47 checking for uint64_t... yes 16:30:47 checking for int32_t... yes 16:30:48 checking for int32_t... yes 16:30:49 checking for int64_t... yes 16:30:49 checking for int64_t... yes 16:30:50 checking for ssize_t... yes 16:30:50 checking for __uint128_t... yes 16:30:51 checking size of int... 4 16:30:51 checking size of long... 8 16:30:52 checking size of void *... 8 16:30:52 checking size of short... 2 16:30:53 checking size of float... 4 16:30:53 checking size of double... 8 16:30:54 checking size of fpos_t... 16 16:30:54 checking size of size_t... 8 16:30:55 checking size of pid_t... 4 16:30:55 checking for long long support... yes 16:30:56 checking size of long long... 8 16:30:56 checking for long double support... yes 16:30:56 checking size of long double... 16 16:30:57 checking for _Bool support... yes 16:30:57 checking size of _Bool... 1 16:30:57 checking for uintptr_t... yes 16:30:58 checking size of uintptr_t... 8 16:30:58 checking size of off_t... 8 16:30:58 checking whether to enable large file support... no 16:30:58 checking size of time_t... 8 16:30:59 checking for pthread_t... yes 16:30:59 checking size of pthread_t... 8 16:30:59 checking for --enable-framework... no 16:30:59 checking for dyld... no 16:30:59 checking the extension of shared libraries... .so 16:30:59 checking LDSHARED... $(CC) -shared 16:30:59 checking CCSHARED... -fPIC 16:30:59 checking LINKFORSHARED... -Xlinker -export-dynamic 16:30:59 checking CFLAGSFORSHARED... 16:30:59 checking SHLIBS... $(LIBS) 16:30:59 checking for sendfile in -lsendfile... no 16:31:00 checking for dlopen in -ldl... yes 16:31:00 checking for shl_load in -ldld... no 16:31:00 checking for library containing sem_init... none required 16:31:00 checking for textdomain in -lintl... no 16:31:00 checking aligned memory access is required... no 16:31:01 checking for --with-hash-algorithm... default 16:31:01 checking for t_open in -lnsl... no 16:31:01 checking for socket in -lsocket... no 16:31:01 checking for --with-libs... no 16:31:01 checking for pkg-config... no 16:31:01 checking for --with-system-expat... no 16:31:01 checking for --with-system-ffi... no 16:31:01 checking for --with-system-libmpdec... no 16:31:01 checking for --enable-loadable-sqlite-extensions... no 16:31:01 checking for --with-tcltk-includes... default 16:31:01 checking for --with-tcltk-libs... default 16:31:01 checking for --with-dbmliborder... 16:31:01 checking for --with-signal-module... yes 16:31:01 checking for --with-threads... yes 16:31:01 checking if PTHREAD_SCOPE_SYSTEM is supported... yes 16:31:02 checking for pthread_sigmask... yes 16:31:02 checking for pthread_atfork... yes 16:31:02 checking if --enable-ipv6 is specified... no 16:31:02 checking for OSX 10.5 SDK or later... no 16:31:02 checking for --with-doc-strings... yes 16:31:02 checking for --with-tsc... no 16:31:02 checking for --with-pymalloc... no 16:31:02 checking for --with-valgrind... no 16:31:02 checking for dlopen... yes 16:31:03 checking DYNLOADFILE... dynload_shlib.o 16:31:03 checking MACHDEP_OBJS... none 16:31:03 checking for alarm... yes 16:31:03 checking for accept4... yes 16:31:03 checking for setitimer... yes 16:31:04 checking for getitimer... yes 16:31:04 checking for bind_textdomain_codeset... yes 16:31:04 checking for chown... yes 16:31:04 checking for clock... yes 16:31:05 checking for confstr... yes 16:31:05 checking for ctermid... yes 16:31:05 checking for dup3... yes 16:31:06 checking for execv... yes 16:31:06 checking for faccessat... yes 16:31:06 checking for fchmod... yes 16:31:06 checking for fchmodat... yes 16:31:07 checking for fchown... yes 16:31:07 checking for fchownat... yes 16:31:07 checking for fexecve... yes 16:31:08 checking for fdopendir... yes 16:31:08 checking for fork... yes 16:31:08 checking for fpathconf... yes 16:31:08 checking for fstatat... yes 16:31:09 checking for ftime... yes 16:31:09 checking for ftruncate... yes 16:31:09 checking for futimesat... yes 16:31:10 checking for futimens... yes 16:31:10 checking for futimes... yes 16:31:10 checking for gai_strerror... yes 16:31:10 checking for getgrouplist... yes 16:31:11 checking for getgroups... yes 16:31:11 checking for getlogin... yes 16:31:11 checking for getloadavg... yes 16:31:12 checking for getpeername... yes 16:31:12 checking for getpgid... yes 16:31:12 checking for getpid... yes 16:31:12 checking for getpriority... yes 16:31:13 checking for getresuid... yes 16:31:13 checking for getresgid... yes 16:31:13 checking for getpwent... yes 16:31:14 checking for getspnam... yes 16:31:14 checking for getspent... yes 16:31:14 checking for getsid... yes 16:31:15 checking for getwd... yes 16:31:15 checking for if_nameindex... yes 16:31:15 checking for initgroups... yes 16:31:15 checking for kill... yes 16:31:16 checking for killpg... yes 16:31:16 checking for lchmod... no 16:31:16 checking for lchown... yes 16:31:16 checking for lockf... yes 16:31:17 checking for linkat... yes 16:31:17 checking for lstat... yes 16:31:17 checking for lutimes... yes 16:31:18 checking for mmap... yes 16:31:18 checking for memrchr... yes 16:31:18 checking for mbrtowc... yes 16:31:18 checking for mkdirat... yes 16:31:19 checking for mkfifo... yes 16:31:19 checking for mkfifoat... yes 16:31:19 checking for mknod... yes 16:31:20 checking for mknodat... yes 16:31:20 checking for mktime... yes 16:31:20 checking for mremap... yes 16:31:21 checking for nice... yes 16:31:21 checking for openat... yes 16:31:21 checking for pathconf... yes 16:31:21 checking for pause... yes 16:31:22 checking for pipe2... yes 16:31:22 checking for plock... no 16:31:22 checking for poll... yes 16:31:23 checking for posix_fallocate... yes 16:31:23 checking for posix_fadvise... yes 16:31:23 checking for pread... yes 16:31:23 checking for pthread_init... no 16:31:24 checking for pthread_kill... yes 16:31:24 checking for putenv... yes 16:31:24 checking for pwrite... yes 16:31:25 checking for readlink... yes 16:31:25 checking for readlinkat... yes 16:31:25 checking for readv... yes 16:31:25 checking for realpath... yes 16:31:26 checking for renameat... yes 16:31:26 checking for select... yes 16:31:26 checking for sem_open... yes 16:31:27 checking for sem_timedwait... yes 16:31:27 checking for sem_getvalue... yes 16:31:27 checking for sem_unlink... yes 16:31:28 checking for sendfile... yes 16:31:28 checking for setegid... yes 16:31:28 checking for seteuid... yes 16:31:28 checking for setgid... yes 16:31:29 checking for sethostname... yes 16:31:29 checking for setlocale... yes 16:31:29 checking for setregid... yes 16:31:30 checking for setreuid... yes 16:31:30 checking for setresuid... yes 16:31:30 checking for setresgid... yes 16:31:30 checking for setsid... yes 16:31:31 checking for setpgid... yes 16:31:31 checking for setpgrp... yes 16:31:31 checking for setpriority... yes 16:31:32 checking for setuid... yes 16:31:32 checking for setvbuf... yes 16:31:32 checking for sched_get_priority_max... yes 16:31:32 checking for sched_setaffinity... yes 16:31:33 checking for sched_setscheduler... yes 16:31:33 checking for sched_setparam... yes 16:31:33 checking for sched_rr_get_interval... yes 16:31:34 checking for sigaction... yes 16:31:34 checking for sigaltstack... yes 16:31:34 checking for siginterrupt... yes 16:31:35 checking for sigpending... yes 16:31:35 checking for sigrelse... yes 16:31:35 checking for sigtimedwait... yes 16:31:35 checking for sigwait... yes 16:31:36 checking for sigwaitinfo... yes 16:31:36 checking for snprintf... yes 16:31:36 checking for strftime... yes 16:31:37 checking for strlcpy... no 16:31:37 checking for symlinkat... yes 16:31:37 checking for sync... yes 16:31:37 checking for sysconf... yes 16:31:38 checking for tcgetpgrp... yes 16:31:38 checking for tcsetpgrp... yes 16:31:38 checking for tempnam... yes 16:31:39 checking for timegm... yes 16:31:39 checking for times... yes 16:31:39 checking for tmpfile... yes 16:31:39 checking for tmpnam... yes 16:31:40 checking for tmpnam_r... yes 16:31:40 checking for truncate... yes 16:31:40 checking for uname... yes 16:31:41 checking for unlinkat... yes 16:31:41 checking for unsetenv... yes 16:31:41 checking for utimensat... yes 16:31:42 checking for utimes... yes 16:31:42 checking for waitid... yes 16:31:42 checking for waitpid... yes 16:31:42 checking for wait3... yes 16:31:43 checking for wait4... yes 16:31:43 checking for wcscoll... yes 16:31:43 checking for wcsftime... yes 16:31:44 checking for wcsxfrm... yes 16:31:44 checking for wmemcmp... yes 16:31:44 checking for writev... yes 16:31:44 checking for _getpty... no 16:31:45 checking whether dirfd is declared... yes 16:31:45 checking for chroot... yes 16:31:45 checking for link... yes 16:31:45 checking for symlink... yes 16:31:45 checking for fchdir... yes 16:31:46 checking for fsync... yes 16:31:46 checking for fdatasync... yes 16:31:46 checking for epoll... yes 16:31:46 checking for epoll_create1... yes 16:31:46 checking for kqueue... no 16:31:46 checking for prlimit... yes 16:31:46 checking for ctermid_r... no 16:31:47 checking for flock declaration... yes 16:31:47 checking for flock... yes 16:31:47 checking for getpagesize... yes 16:31:47 checking for broken unsetenv... no 16:31:47 checking for true... true 16:31:47 checking for inet_aton in -lc... yes 16:31:48 checking for chflags... no 16:31:48 checking for lchflags... no 16:31:48 checking for inflateCopy in -lz... yes 16:31:49 checking for hstrerror... yes 16:31:49 checking for inet_aton... yes 16:31:49 checking for inet_pton... yes 16:31:50 checking for setgroups... yes 16:31:50 checking for openpty... no 16:31:50 checking for openpty in -lutil... yes 16:31:50 checking for forkpty... yes 16:31:51 checking for memmove... yes 16:31:51 checking for fseek64... no 16:31:51 checking for fseeko... yes 16:31:51 checking for fstatvfs... yes 16:31:52 checking for ftell64... no 16:31:52 checking for ftello... yes 16:31:52 checking for statvfs... yes 16:31:53 checking for dup2... yes 16:31:53 checking for strdup... yes 16:31:53 checking for getpgrp... yes 16:31:54 checking for setpgrp... (cached) yes 16:31:54 checking for gettimeofday... yes 16:31:54 checking for clock_gettime... yes 16:31:54 checking for clock_getres... yes 16:31:55 checking for major... yes 16:31:55 checking for getaddrinfo... yes 16:31:55 checking getaddrinfo bug... no 16:31:56 checking for getnameinfo... yes 16:31:56 checking whether time.h and sys/time.h may both be included... yes 16:31:57 checking whether struct tm is in sys/time.h or time.h... time.h 16:31:57 checking for struct tm.tm_zone... yes 16:31:57 checking for struct stat.st_rdev... yes 16:31:57 checking for struct stat.st_blksize... yes 16:31:57 checking for struct stat.st_flags... no 16:31:58 checking for struct stat.st_gen... no 16:31:59 checking for struct stat.st_birthtime... no 16:31:59 checking for struct stat.st_blocks... yes 16:32:00 checking for time.h that defines altzone... no 16:32:00 checking whether sys/select.h and sys/time.h may both be included... yes 16:32:00 checking for addrinfo... yes 16:32:00 checking for sockaddr_storage... yes 16:32:00 checking whether char is unsigned... no 16:32:00 checking for an ANSI C-conforming const... yes 16:32:01 checking for working volatile... yes 16:32:01 checking for working signed char... yes 16:32:01 checking for prototypes... yes 16:32:01 checking for variable length prototypes and stdarg.h... yes 16:32:01 checking for socketpair... yes 16:32:01 checking if sockaddr has sa_len member... no 16:32:02 checking whether va_list is an array... yes 16:32:02 checking for gethostbyname_r... yes 16:32:02 checking gethostbyname_r with 6 args... yes 16:32:02 checking for __fpu_control... yes 16:32:02 checking for --with-fpectl... no 16:32:02 checking for --with-libm=STRING... default LIBM="-lm" 16:32:02 checking for --with-libc=STRING... default LIBC="" 16:32:02 checking for x64 gcc inline assembler... yes 16:32:03 checking whether C doubles are little-endian IEEE 754 binary64... yes 16:32:03 checking whether C doubles are big-endian IEEE 754 binary64... no 16:32:03 checking whether C doubles are ARM mixed-endian IEEE 754 binary64... no 16:32:04 checking whether we can use gcc inline assembler to get and set x87 control word... yes 16:32:04 checking for x87-style double rounding... no 16:32:04 checking for acosh... yes 16:32:05 checking for asinh... yes 16:32:05 checking for atanh... yes 16:32:05 checking for copysign... yes 16:32:05 checking for erf... yes 16:32:06 checking for erfc... yes 16:32:06 checking for expm1... yes 16:32:06 checking for finite... yes 16:32:07 checking for gamma... yes 16:32:07 checking for hypot... yes 16:32:07 checking for lgamma... yes 16:32:08 checking for log1p... yes 16:32:08 checking for log2... yes 16:32:08 checking for round... yes 16:32:08 checking for tgamma... yes 16:32:09 checking whether isinf is declared... yes 16:32:09 checking whether isnan is declared... yes 16:32:09 checking whether isfinite is declared... yes 16:32:10 checking whether tanh preserves the sign of zero... yes 16:32:10 checking whether log1p drops the sign of negative zero... no 16:32:10 checking whether POSIX semaphores are enabled... yes 16:32:11 checking for broken sem_getvalue... no 16:32:11 checking digit size for Python's longs... no value specified 16:32:11 checking wchar.h usability... yes 16:32:12 checking wchar.h presence... yes 16:32:12 checking for wchar.h... yes 16:32:12 checking size of wchar_t... 4 16:32:12 checking for UCS-4 tcl... no 16:32:12 checking whether wchar_t is signed... yes 16:32:13 no usable wchar_t found 16:32:13 checking whether byte ordering is bigendian... no 16:32:13 checking ABIFLAGS... 16:32:13 checking SOABI... cpython-34 16:32:13 checking LDVERSION... $(VERSION)$(ABIFLAGS) 16:32:13 checking whether right shift extends the sign bit... yes 16:32:13 checking for getc_unlocked() and friends... yes 16:32:14 checking how to link readline libs... none 16:32:15 checking for rl_callback_handler_install in -lreadline... no 16:32:15 checking for rl_pre_input_hook in -lreadline... no 16:32:15 checking for rl_completion_display_matches_hook in -lreadline... no 16:32:15 checking for rl_completion_matches in -lreadline... no 16:32:16 checking for broken nice()... no 16:32:16 checking for broken poll()... no 16:32:16 checking for struct tm.tm_zone... (cached) yes 16:32:16 checking for working tzset()... yes 16:32:17 checking for tv_nsec in struct stat... yes 16:32:17 checking for tv_nsec2 in struct stat... no 16:32:17 checking curses.h usability... no 16:32:17 checking curses.h presence... no 16:32:18 checking for curses.h... no 16:32:18 checking ncurses.h usability... no 16:32:18 checking ncurses.h presence... no 16:32:18 checking for ncurses.h... no 16:32:18 checking for term.h... no 16:32:18 checking whether mvwdelch is an expression... no 16:32:18 checking whether WINDOW has _flags... no 16:32:18 checking for is_term_resized... no 16:32:18 checking for resize_term... no 16:32:19 checking for resizeterm... no 16:32:19 configure: checking for device files 16:32:19 checking for /dev/ptmx... yes 16:32:19 checking for /dev/ptc... no 16:32:19 checking for %lld and %llu printf() format support... yes 16:32:19 checking for %zd printf() format support... yes 16:32:20 checking for socklen_t... yes 16:32:20 checking for broken mbstowcs... no 16:32:20 checking for --with-computed-gotos... no value specified 16:32:20 checking whether /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang supports computed gotos... yes 16:32:21 checking for build directories... done 16:32:21 checking for -O2... yes 16:32:21 checking for glibc _FORTIFY_SOURCE/memmove bug... no 16:32:21 checking for ensurepip... upgrade 16:32:21 configure: creating ./config.status 16:32:22 config.status: creating Makefile.pre 16:32:22 config.status: creating Modules/Setup.config 16:32:22 config.status: creating Misc/python.pc 16:32:22 config.status: creating Misc/python-config.sh 16:32:22 config.status: creating Modules/ld_so_aix 16:32:22 config.status: creating pyconfig.h 16:32:22 creating Modules/Setup 16:32:22 creating Modules/Setup.local 16:32:22 creating Makefile 16:32:22 + build_python 16:32:22 + cd build 16:32:22 + make clean 16:32:22 find ../../cpython -depth -name '__pycache__' -exec rm -rf {} ';' 16:32:22 find ../../cpython -name '*.py[co]' -exec rm -f {} ';' 16:32:22 find . -name '*.[oa]' -exec rm -f {} ';' 16:32:22 find . -name '*.s[ol]' -exec rm -f {} ';' 16:32:22 find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';' 16:32:22 find build -name 'fficonfig.h' -exec rm -f {} ';' || true 16:32:22 find: `build': No such file or directory 16:32:22 find build -name '*.py' -exec rm -f {} ';' || true 16:32:22 find: `build': No such file or directory 16:32:22 find build -name '*.py[co]' -exec rm -f {} ';' || true 16:32:22 find: `build': No such file or directory 16:32:22 rm -f pybuilddir.txt 16:32:22 rm -f Lib/lib2to3/*Grammar*.pickle 16:32:22 rm -f Modules/_testembed Modules/_freeze_importlib 16:32:22 + make -j 48 all 16:32:22 /bin/mkdir -p Include 16:32:22 python3.4 ../../cpython/Parser/asdl_c.py -h Include ../../cpython/Parser/Python.asdl 16:32:22 make Parser/pgen 16:32:22 ../../cpython/Python/makeopcodetargets.py Python/opcode_targets.h 16:32:22 make Modules/_freeze_importlib 16:32:22 /usr/bin/install -c -m 644 ../../cpython/Tools/gdb/libpython.py python-gdb.py 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Modules/_testembed.o ../../cpython/Modules/_testembed.c 16:32:22 # Substitution happens here, as the completely-expanded BINDIR 16:32:22 # is not available in configure 16:32:22 sed -e "s, at EXENAME@,/usr/local/bin/python3.4," < ../../cpython/Misc/python-config.in >python-config.py 16:32:22 make[1]: Entering directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:22 /bin/mkdir -p Include 16:32:22 # Replace makefile compat. variable references with shell script compat. ones; -> 16:32:22 sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' < Misc/python-config.sh >python-config 16:32:22 python3.4 ../../cpython/Parser/asdl_c.py -h Include ../../cpython/Parser/Python.asdl 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o ../../cpython/Parser/tokenizer_pgen.c 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/printgrammar.o ../../cpython/Parser/printgrammar.c 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parsetok_pgen.o ../../cpython/Parser/parsetok_pgen.c 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgenmain.o ../../cpython/Parser/pgenmain.c 16:32:22 # On Darwin, always use the python version of the script, the shell 16:32:22 make[1]: Entering directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:22 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Modules/_freeze_importlib.o ../../cpython/Modules/_freeze_importlib.c 16:32:22 # version doesn't use the compiler customizations that are provided 16:32:22 # in python (_osx_support.py). 16:32:22 if test `uname -s` = Darwin; then \ 16:32:22 cp python-config.py python-config; \ 16:32:22 fi 16:32:22 /bin/mkdir -p Include 16:32:22 python3.4 ../../cpython/Parser/asdl_c.py -h Include ../../cpython/Parser/Python.asdl 16:32:23 make Parser/pgen 16:32:23 make[2]: Entering directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:23 /bin/mkdir -p Include 16:32:23 python3.4 ../../cpython/Parser/asdl_c.py -h Include ../../cpython/Parser/Python.asdl 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o ../../cpython/Parser/tokenizer_pgen.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/printgrammar.o ../../cpython/Parser/printgrammar.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parsetok_pgen.o ../../cpython/Parser/parsetok_pgen.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgenmain.o ../../cpython/Parser/pgenmain.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Modules/python.o ../../cpython/Modules/python.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/acceler.o ../../cpython/Parser/acceler.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/acceler.o ../../cpython/Parser/acceler.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar1.o ../../cpython/Parser/grammar1.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar1.o ../../cpython/Parser/grammar1.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/listnode.o ../../cpython/Parser/listnode.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/node.o ../../cpython/Parser/node.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/listnode.o ../../cpython/Parser/listnode.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/node.o ../../cpython/Parser/node.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parser.o ../../cpython/Parser/parser.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/bitset.o ../../cpython/Parser/bitset.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parser.o ../../cpython/Parser/parser.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/bitset.o ../../cpython/Parser/bitset.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/metagrammar.o ../../cpython/Parser/metagrammar.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/metagrammar.o ../../cpython/Parser/metagrammar.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/firstsets.o ../../cpython/Parser/firstsets.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar.o ../../cpython/Parser/grammar.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgen.o ../../cpython/Parser/pgen.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/myreadline.o ../../cpython/Parser/myreadline.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parsetok.o ../../cpython/Parser/parsetok.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/firstsets.o ../../cpython/Parser/firstsets.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/tokenizer.o ../../cpython/Parser/tokenizer.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar.o ../../cpython/Parser/grammar.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/abstract.o ../../cpython/Objects/abstract.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgen.o ../../cpython/Parser/pgen.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/accu.o ../../cpython/Objects/accu.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/obmalloc.o ../../cpython/Objects/obmalloc.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/dynamic_annotations.o ../../cpython/Python/dynamic_annotations.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/boolobject.o ../../cpython/Objects/boolobject.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytes_methods.o ../../cpython/Objects/bytes_methods.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/acceler.o ../../cpython/Parser/acceler.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/mysnprintf.o ../../cpython/Python/mysnprintf.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o ../../cpython/Objects/bytearrayobject.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar1.o ../../cpython/Parser/grammar1.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytesobject.o ../../cpython/Objects/bytesobject.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/pyctype.o ../../cpython/Python/pyctype.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/listnode.o ../../cpython/Parser/listnode.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/node.o ../../cpython/Parser/node.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/cellobject.o ../../cpython/Objects/cellobject.c 16:32:23 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/classobject.o ../../cpython/Objects/classobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/codeobject.o ../../cpython/Objects/codeobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/acceler.o ../../cpython/Parser/acceler.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar1.o ../../cpython/Parser/grammar1.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/listnode.o ../../cpython/Parser/listnode.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/node.o ../../cpython/Parser/node.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/complexobject.o ../../cpython/Objects/complexobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/descrobject.o ../../cpython/Objects/descrobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/enumobject.o ../../cpython/Objects/enumobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parser.o ../../cpython/Parser/parser.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parser.o ../../cpython/Parser/parser.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/exceptions.o ../../cpython/Objects/exceptions.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/genobject.o ../../cpython/Objects/genobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/fileobject.o ../../cpython/Objects/fileobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/floatobject.o ../../cpython/Objects/floatobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/bitset.o ../../cpython/Parser/bitset.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/metagrammar.o ../../cpython/Parser/metagrammar.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/firstsets.o ../../cpython/Parser/firstsets.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar.o ../../cpython/Parser/grammar.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/frameobject.o ../../cpython/Objects/frameobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/funcobject.o ../../cpython/Objects/funcobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgen.o ../../cpython/Parser/pgen.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/iterobject.o ../../cpython/Objects/iterobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/listobject.o ../../cpython/Objects/listobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/obmalloc.o ../../cpython/Objects/obmalloc.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/longobject.o ../../cpython/Objects/longobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/dictobject.o ../../cpython/Objects/dictobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/dynamic_annotations.o ../../cpython/Python/dynamic_annotations.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/memoryobject.o ../../cpython/Objects/memoryobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/bitset.o ../../cpython/Parser/bitset.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/metagrammar.o ../../cpython/Parser/metagrammar.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/methodobject.o ../../cpython/Objects/methodobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/moduleobject.o ../../cpython/Objects/moduleobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/namespaceobject.o ../../cpython/Objects/namespaceobject.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/object.o ../../cpython/Objects/object.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/firstsets.o ../../cpython/Parser/firstsets.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/grammar.o ../../cpython/Parser/grammar.c 16:32:24 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/mysnprintf.o ../../cpython/Python/mysnprintf.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/pyctype.o ../../cpython/Python/pyctype.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/obmalloc.o ../../cpython/Objects/obmalloc.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/capsule.o ../../cpython/Objects/capsule.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/rangeobject.o ../../cpython/Objects/rangeobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/setobject.o ../../cpython/Objects/setobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/sliceobject.o ../../cpython/Objects/sliceobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/structseq.o ../../cpython/Objects/structseq.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/tupleobject.o ../../cpython/Objects/tupleobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/typeobject.o ../../cpython/Objects/typeobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/pgen.o ../../cpython/Parser/pgen.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/unicodeobject.o ../../cpython/Objects/unicodeobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/unicodectype.o ../../cpython/Objects/unicodectype.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/weakrefobject.o ../../cpython/Objects/weakrefobject.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/_warnings.o ../../cpython/Python/_warnings.c 16:32:25 /bin/mkdir -p Python 16:32:25 python3.4 ../../cpython/Parser/asdl_c.py -c Python ../../cpython/Parser/Python.asdl 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/asdl.o ../../cpython/Python/asdl.c 16:32:25 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/bltinmodule.o ../../cpython/Python/bltinmodule.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/myreadline.o ../../cpython/Parser/myreadline.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/parsetok.o ../../cpython/Parser/parsetok.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/ceval.o ../../cpython/Python/ceval.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/codecs.o ../../cpython/Python/codecs.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/dynamic_annotations.o ../../cpython/Python/dynamic_annotations.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/errors.o ../../cpython/Python/errors.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Parser/tokenizer.o ../../cpython/Parser/tokenizer.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/frozenmain.o ../../cpython/Python/frozenmain.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/future.o ../../cpython/Python/future.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/abstract.o ../../cpython/Objects/abstract.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/accu.o ../../cpython/Objects/accu.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/getargs.o ../../cpython/Python/getargs.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/boolobject.o ../../cpython/Objects/boolobject.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/getcompiler.o ../../cpython/Python/getcompiler.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/getcopyright.o ../../cpython/Python/getcopyright.c 16:32:26 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -DPLATFORM='"linux"' -o Python/getplatform.o ../../cpython/Python/getplatform.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytes_methods.o ../../cpython/Objects/bytes_methods.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/getversion.o ../../cpython/Python/getversion.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/import.o ../../cpython/Python/import.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -I. -o Python/importdl.o ../../cpython/Python/importdl.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/marshal.o ../../cpython/Python/marshal.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/modsupport.o ../../cpython/Python/modsupport.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/mystrtoul.o ../../cpython/Python/mystrtoul.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o ../../cpython/Objects/bytearrayobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/mysnprintf.o ../../cpython/Python/mysnprintf.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/peephole.o ../../cpython/Python/peephole.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fsanitize=address Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -ldl -lutil -o Parser/pgen 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/bytesobject.o ../../cpython/Objects/bytesobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/pyarena.o ../../cpython/Python/pyarena.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/pyctype.o ../../cpython/Python/pyctype.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/cellobject.o ../../cpython/Objects/cellobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Python/pyfpe.o ../../cpython/Python/pyfpe.c 16:32:27 /usr/bin/ld: cannot find Parser/pgen.o: No such file or directory 16:32:27 clang-3.5: error: linker command failed with exit code 1 (use -v to see invocation) 16:32:27 /home/jenkins/workspace/asap-build/llvm/lib/Transforms/SanityChecks/asap-clang-utils.rb:17:in `run!': Command /home/jenkins/workspace/asap-build/llvm-build/bin/clang failed with status pid 42314 exit 1 (RunExternalCommandError) 16:32:27 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:154:in `do_link' 16:32:27 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:124:in `exec' 16:32:27 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:397:in `main' 16:32:27 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:439:in `
' 16:32:27 make[1]: *** [Parser/pgen] Error 1 16:32:27 make[1]: Leaving directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:27 make: *** [Include/graminit.h] Error 2 16:32:27 make: *** Waiting for unfinished jobs.... 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/classobject.o ../../cpython/Objects/classobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/codeobject.o ../../cpython/Objects/codeobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/complexobject.o ../../cpython/Objects/complexobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/descrobject.o ../../cpython/Objects/descrobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/enumobject.o ../../cpython/Objects/enumobject.c 16:32:27 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/exceptions.o ../../cpython/Objects/exceptions.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/genobject.o ../../cpython/Objects/genobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/fileobject.o ../../cpython/Objects/fileobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/floatobject.o ../../cpython/Objects/floatobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/frameobject.o ../../cpython/Objects/frameobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/funcobject.o ../../cpython/Objects/funcobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/iterobject.o ../../cpython/Objects/iterobject.c 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fsanitize=address Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -ldl -lutil -o Parser/pgen 16:32:28 /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -fsanitize=address -Werror=declaration-after-statement -I. -IInclude -I../../cpython/Include -DPy_BUILD_CORE -o Objects/listobject.o ../../cpython/Objects/listobject.c 16:32:28 Parser/pgen.o: file not recognized: File truncated 16:32:28 clang-3.5: error: linker command failed with exit code 1 (use -v to see invocation) 16:32:28 /home/jenkins/workspace/asap-build/llvm/lib/Transforms/SanityChecks/asap-clang-utils.rb:17:in `run!': Command /home/jenkins/workspace/asap-build/llvm-build/bin/clang failed with status pid 42440 exit 1 (RunExternalCommandError) 16:32:28 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:154:in `do_link' 16:32:28 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:124:in `exec' 16:32:28 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:397:in `main' 16:32:28 from /home/jenkins/workspace/asap-build/llvm-build/bin/asap-clang:439:in `
' 16:32:28 make[2]: *** [Parser/pgen] Error 1 16:32:28 make[2]: Leaving directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:28 make[1]: *** [Include/graminit.h] Error 2 16:32:28 make[1]: *** Waiting for unfinished jobs.... 16:32:39 make[1]: Leaving directory `/home/jenkins/workspace/asap-benchmarks/python34-experiments/cpython-asan-build/build' 16:32:39 make: *** [Python/importlib.h] Error 2 From jonas.wagner at epfl.ch Fri Sep 5 12:15:15 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Fri, 5 Sep 2014 12:15:15 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: References: <20140904180931.46ad9be0@fsol> Message-ID: >> > Would people be interested in having a parallel version? >> >> See http://bugs.python.org/issue5309 > > Cool! I'll look into this. The patch there works well for me. I've made one small update, and submitted the new version in the bug tracker. Regarding the other build problem, I might have found some hint: - Parser/pgen.o ends up in both the PARSER_OBJS and PGENOBJS variables in the Makefile - PARSER_OBJS is depended upon in a few places, hence it could be that make starts to build Parser/pgen.o - PGENOBJS is built when building PGEN, which happens *in a different make that is called recursively* I think the culprit is the rule for GRAMMAR_H which calls make recursively. Is there a reason that GRAMMAR_H has to generate PGEN like this? Couldn't it just depend on PGEN? Cheers, Jonas From jonas.wagner at epfl.ch Fri Sep 5 15:05:49 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Fri, 5 Sep 2014 15:05:49 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: References: <20140904180931.46ad9be0@fsol> Message-ID: Hi again, the attached Makefile patch seems to fix the parallel build problems. Is there a chance of getting this into trunk? Should I open an issue or send the patch somewhere else? Cheers, Jonas On Fri, Sep 5, 2014 at 12:15 PM, Jonas Wagner wrote: >>> > Would people be interested in having a parallel version? >>> >>> See http://bugs.python.org/issue5309 >> >> Cool! I'll look into this. > > The patch there works well for me. I've made one small update, and > submitted the new version in the bug tracker. > > Regarding the other build problem, I might have found some hint: > - Parser/pgen.o ends up in both the PARSER_OBJS and PGENOBJS variables > in the Makefile > - PARSER_OBJS is depended upon in a few places, hence it could be that > make starts to build Parser/pgen.o > - PGENOBJS is built when building PGEN, which happens *in a different > make that is called recursively* > > I think the culprit is the rule for GRAMMAR_H which calls make > recursively. Is there a reason that GRAMMAR_H has to generate PGEN > like this? Couldn't it just depend on PGEN? > > Cheers, > Jonas -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile_parallel.patch Type: text/x-patch Size: 1349 bytes Desc: not available URL: From brett at python.org Fri Sep 5 19:56:58 2014 From: brett at python.org (Brett Cannon) Date: Fri, 05 Sep 2014 17:56:58 +0000 Subject: [Python-Dev] cpython and parallel make References: <20140904180931.46ad9be0@fsol> Message-ID: Open an issue on bugs.python.org and attach the patch there (it should also ask you so sign the contributor agreement, but if not then please also sign that). On Fri Sep 05 2014 at 12:52:45 PM Jonas Wagner wrote: > Hi again, > > the attached Makefile patch seems to fix the parallel build problems. > > Is there a chance of getting this into trunk? Should I open an issue > or send the patch somewhere else? > > Cheers, > Jonas > > On Fri, Sep 5, 2014 at 12:15 PM, Jonas Wagner > wrote: > >>> > Would people be interested in having a parallel version? > >>> > >>> See http://bugs.python.org/issue5309 > >> > >> Cool! I'll look into this. > > > > The patch there works well for me. I've made one small update, and > > submitted the new version in the bug tracker. > > > > Regarding the other build problem, I might have found some hint: > > - Parser/pgen.o ends up in both the PARSER_OBJS and PGENOBJS variables > > in the Makefile > > - PARSER_OBJS is depended upon in a few places, hence it could be that > > make starts to build Parser/pgen.o > > - PGENOBJS is built when building PGEN, which happens *in a different > > make that is called recursively* > > > > I think the culprit is the rule for GRAMMAR_H which calls make > > recursively. Is there a reason that GRAMMAR_H has to generate PGEN > > like this? Couldn't it just depend on PGEN? > > > > Cheers, > > Jonas > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zuo at kaliszewski.net Sat Sep 6 13:34:37 2014 From: zuo at kaliszewski.net (Jan Kaliszewski) Date: Sat, 6 Sep 2014 13:34:37 +0200 Subject: [Python-Dev] Python docs about comparisons vs. CPython reality Message-ID: <20140906133437.1e18bbfb@grzmot> Hello, Are they bugs in the Python docs or just some CPython implementation details that are purposely not documented? (but then, again, some of the docs seem to be at least not precise...): In https://docs.python.org/3.4/reference/datamodel.html#object.__eq__ there is the statement: > There are no implied relationships among the comparison operators. > The truth of x==y does not imply that x!=y is false. Accordingly, > when defining __eq__(), one should also define __ne__() so that the > operators will behave as expected. On the other hand, in https://docs.python.org/3.4/library/stdtypes.html#comparisons we read: > (in general, __lt__() and __eq__() are sufficient, if you want the > conventional meanings of the comparison operators) And, when I try the __eq__() stuff in CPython it seems that, indeed, the language provides a proper __ne__() implementation for me automatically (without need to implement __ne__() explicitly by myself): Python 3.4.0 (default, Mar 20 2014, 01:28:00) [...] >>> class A: ... def __eq__(self, other): ... if hasattr(self, 'x') and hasattr(other, 'x'): ... return self.x == other.x ... return NotImplemented ... >>> A() == A() False >>> A() != A() True >>> a = A() >>> a.x = 1 >>> a1 = A() >>> a1.x = 1 >>> a2 = A() >>> a2.x = 2 >>> a == a1 True >>> a != a1 False >>> a1 == a1 True >>> a1 != a1 False >>> a1 == a2 False >>> a1 != a2 True Is it a language guarantee (then, I believe, it should be documented) or just an implementation accident? (then, I believe, it still could be documented as a CPython implementation detail). See also the Python equivalent of the SimpleNamespace class (without __ne__() implemented explicitly): https://docs.python.org/3/library/types.html#types.SimpleNamespace On the other hand, the "__lt__() and __eq__() are sufficient" statement seems not to be true: >>> a < a1 False >>> a <= a1 Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: A() <= A() >>> a > a1 False >>> a >= a1 Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: A() >= A() >>> a1 < a2 True >>> a1 <= a2 Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: A() <= A() >>> a1 > a2 False >>> a1 >= a2 Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: A() >= A() On yet another hand, adding __le__() to that class seems to be perfectly sufficient (without adding __gt__() and __ge__()): >>> def le(self, other): ... if hasattr(self, 'x') and hasattr(other, 'x'): ... return self.x <= other.x ... return NotImplemented ... >>> A.__le__ = le >>> a < a1 False >>> a <= a1 True >>> a > a1 False >>> a >= a1 True >>> a1 < a2 True >>> a1 <= a2 True >>> a1 > a2 False >>> a1 >= a2 False What of all this stuff is a language guarantee and what is just an implementation accident? Shouldn't it be documented more accurately? Cheers. *j From solipsis at pitrou.net Sat Sep 6 14:27:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 6 Sep 2014 14:27:28 +0200 Subject: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation References: <3hqshz5QhVz7MpF@mail.python.org> Message-ID: <20140906142728.3c46a7b0@fsol> On Sat, 6 Sep 2014 12:40:19 +0200 (CEST) nick.coghlan wrote: > > The following command will install the latest version of a module and its > dependencies from the Python Package Index:: > > - pip install SomePackage > + python -m pip install SomePackage > + > +.. note:: > + > + For POSIX users (including Mac OS X and Linux users), the examples in > + this guide assume the use of a :term:`virtual environment`. Why not advocate --user instead? It is simpler than messing around with virtual environments and will suffice for most use cases. Regards Antoine. From skip at pobox.com Sat Sep 6 14:37:11 2014 From: skip at pobox.com (Skip Montanaro) Date: Sat, 6 Sep 2014 07:37:11 -0500 Subject: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation In-Reply-To: <20140906142728.3c46a7b0@fsol> References: <3hqshz5QhVz7MpF@mail.python.org> <20140906142728.3c46a7b0@fsol> Message-ID: On Sat, Sep 6, 2014 at 7:27 AM, Antoine Pitrou wrote: > Why not advocate --user instead? It is simpler than messing around with virtual > environments and will suffice for most use cases. I agree, however, --user needs to be more fully integrated into pip's behavior. For example, if I execute pip install --user SomePackage today, when SomePackage is installed someplace outside ~/.local, pip complains that SomePackage is already installed. If I then execute pip install --user --upgrade SomePackage it tries to remove the outdated more global version of SomePackage. Skip From skip at pobox.com Sat Sep 6 14:47:17 2014 From: skip at pobox.com (Skip Montanaro) Date: Sat, 6 Sep 2014 07:47:17 -0500 Subject: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation In-Reply-To: References: <3hqshz5QhVz7MpF@mail.python.org> <20140906142728.3c46a7b0@fsol> Message-ID: > If I then execute > > pip install --user --upgrade SomePackage > > it tries to remove the outdated more global version of SomePackage. BTW, I believe this is a known issue: https://github.com/pypa/pip/issues/1851 https://github.com/pypa/pip/issues/1122 Based on the comment in the second issue, it doesn't appear this will be resolved until 1.7 at the earliest. Skip From p.f.moore at gmail.com Sat Sep 6 15:13:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 6 Sep 2014 14:13:35 +0100 Subject: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation In-Reply-To: References: <3hqshz5QhVz7MpF@mail.python.org> <20140906142728.3c46a7b0@fsol> Message-ID: On 6 September 2014 13:47, Skip Montanaro wrote: > Based on the comment in the second issue, it doesn't appear this will > be resolved until 1.7 at the earliest. The second issue is specific to setuptools, where we have some very unpleasant hacks to deal with the setuptools/distribute mess. Having said that I don't have an immediate feel for what proportion of the issues around --user are fixed in 1.6, as I don't personally use --user much. Paul From rdmurray at bitdance.com Sat Sep 6 17:18:38 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 06 Sep 2014 11:18:38 -0400 Subject: [Python-Dev] Python docs about comparisons vs. CPython reality In-Reply-To: <20140906133437.1e18bbfb@grzmot> References: <20140906133437.1e18bbfb@grzmot> Message-ID: <20140906151839.BF0A0250E23@webabinitio.net> On Sat, 06 Sep 2014 13:34:37 +0200, Jan Kaliszewski wrote: > Are they bugs in the Python docs or just some CPython implementation > details that are purposely not documented? (but then, again, some of > the docs seem to be at least not precise...): You might want to read http://bugs.python.org/issue12067 and contribute your thoughts there. There are indeed documentation improvements under discussion. --David From ncoghlan at gmail.com Sat Sep 6 21:36:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 7 Sep 2014 05:36:12 +1000 Subject: [Python-Dev] cpython (3.4): Issue #22295: Adopt 'python -m pip' as the preferred invocation In-Reply-To: References: <3hqshz5QhVz7MpF@mail.python.org> <20140906142728.3c46a7b0@fsol> Message-ID: On 6 Sep 2014 23:15, "Paul Moore" wrote: > > On 6 September 2014 13:47, Skip Montanaro wrote: > > Based on the comment in the second issue, it doesn't appear this will > > be resolved until 1.7 at the earliest. > > The second issue is specific to setuptools, where we have some very > unpleasant hacks to deal with the setuptools/distribute mess. > > Having said that I don't have an immediate feel for what proportion of > the issues around --user are fixed in 1.6, as I don't personally use > --user much. Using virtual environments also avoids some bad interactions between pkg_resources.requires() and the way Fedora (et al) handle parallel installation of mutually incompatible versions of components that use the same module namespace (e.g. CherryPy 2 vs CherryPy 3) into the system Python. The specific reason for the assumption of virtual environment usage on POSIX is slightly different though: inside a virtual environment created with pyvenv, "python" always refers to Python 3, regardless of platform. Thus, the assumption let me avoid a long-winded explanation of PEP 394 and why the system-wide "python" refers to Python 2 and you need to run "python3 -m pip" to install into Python 3 instead. It was the system level pip vs pip3 distinction that prompted the original change to recommending "-m pip", and I realised explaining the vagaries of python vs python3 would be just as distracting & uninteresting for most users. There's a question at the end of the document about installing into the system Python that could probably be expanded on with more details, but there really are enough ugly edge cases to running custom code in the system Python (at least currently) that the simplest answer is "don't". Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Sep 7 14:39:56 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 07 Sep 2014 05:39:56 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 Message-ID: <540C521C.7070802@hastings.org> Matthias asked me when I was going to release 3.4.2. I propose the following schedule: Tag 3.4.2rc1 Friday Sep 12 2014 Release 3.4.2rc1 Saturday Sep 13 2014 Tag 3.4.2 final Saturday Sep 27 2014 Release 3.4.2 final Sunday Sep 28 2014 Normally I want to tag on Saturdays and release on Sundays, however I propose moving rc1 a day earlier because I'll be unavailable that Sunday. Any objections? I will interpret silence as tacit approval. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Sun Sep 7 20:49:17 2014 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sun, 07 Sep 2014 19:49:17 +0100 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <540912EC.9000602@cavallinux.eu> References: <5407BB4F.1040203@cavallinux.eu> <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> <540912EC.9000602@cavallinux.eu> Message-ID: <540CA8AD.5020809@cavallinux.eu> Ok, I've put everything under the tools directory of the build server for the project home:cavallo71:opt-python-interpreters: https://build.opensuse.org/package/show/home:cavallo71:opt-python-interpreters/tools To get started you need using the osc client (it uses the same subversion syntax, so is very simple): $> osc co home:cavallo71:opt-python-interpreters $> make -f tools/Makefile update (for the 2.7 branch) $> make -f tools/Makefile BRANCH=3 update (for the 3.x branch, I haven't fixed few issue with the build) I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if there's a general consensus on that (maybe having the python project official release build on susebuild?). Then there's the testing bit (it's separate from the python binaries build to insulate the two). It's very similar and I'll post more details when ready. I hope this helps, Thnaks Antonio Cavallo wrote: > Setting up a repo with the code and cleaning a bit here and there. > Over the weekend I can put something useable. > > > > Nick Coghlan wrote: >> On 4 September 2014 18:50, A. Cavallo wrote: >>> Yes there are "details" indeed. But not show stoppers. A prototype >>> can be >>> seen here: http://cclimited.webfactional.com >>> The nice bit is testing will be executed in a total fresh instance, >>> an added >>> bonus >> >> Oh, very nice! >> >> What is needed to get that up and running? >> >> Regards, >> Nick. >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu > From ncoghlan at gmail.com Sun Sep 7 23:24:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 8 Sep 2014 07:24:24 +1000 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: <540CA8AD.5020809@cavallinux.eu> References: <5407BB4F.1040203@cavallinux.eu> <90e33a7b-c519-4e2c-a1f0-2e11124a1dcd@email.android.com> <540912EC.9000602@cavallinux.eu> <540CA8AD.5020809@cavallinux.eu> Message-ID: On 8 Sep 2014 04:51, "Antonio Cavallo" wrote: > > Ok, > > I've put everything under the tools directory of the build server for the project home:cavallo71:opt-python-interpreters: > > https://build.opensuse.org/package/show/home:cavallo71:opt-python-interpreters/tools > > To get started you need using the osc client (it uses the same subversion syntax, so is very simple): > > $> osc co home:cavallo71:opt-python-interpreters > $> make -f tools/Makefile update (for the 2.7 branch) > $> make -f tools/Makefile BRANCH=3 update (for the 3.x branch, I > haven't fixed few issue with the build) Thanks for posting this! > I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if there's a general consensus on that (maybe having the python project official release build on susebuild?). Integration into the release process will be up to the current release managers (Larry Hastings for 3.4/3.5 and Benjamin Peterson for 2.7). That may be worthwhile as a distro independent way of publishing "opt builds" of Python for Linux platforms (I'd suggest using the softwarecollections.org convention and put them under "/opt/psf/pythonXY"). Note that I think it's worth pursuing this as an integration testing tool, regardless of whether or not we end up incorporating it into the release process. > Then there's the testing bit (it's separate from the python binaries build to insulate the two). It's very similar and I'll post more details when ready. Thanks again! Regards, Nick. > > I hope this helps, > Thnaks > > > > Antonio Cavallo wrote: >> >> Setting up a repo with the code and cleaning a bit here and there. >> Over the weekend I can put something useable. >> >> >> >> Nick Coghlan wrote: >>> >>> On 4 September 2014 18:50, A. Cavallo wrote: >>>> >>>> Yes there are "details" indeed. But not show stoppers. A prototype >>>> can be >>>> seen here: http://cclimited.webfactional.com >>>> The nice bit is testing will be executed in a total fresh instance, >>>> an added >>>> bonus >>> >>> >>> Oh, very nice! >>> >>> What is needed to get that up and running? >>> >>> Regards, >>> Nick. >>> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Mon Sep 8 06:28:25 2014 From: nad at acm.org (Ned Deily) Date: Sun, 07 Sep 2014 21:28:25 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 References: <540C521C.7070802@hastings.org> Message-ID: In article <540C521C.7070802 at hastings.org>, Larry Hastings wrote: > Matthias asked me when I was going to release 3.4.2. I propose the > following schedule: > > Tag 3.4.2rc1 Friday Sep 12 2014 > Release 3.4.2rc1 Saturday Sep 13 2014 > Tag 3.4.2 final Saturday Sep 27 2014 > Release 3.4.2 final Sunday Sep 28 2014 > > Normally I want to tag on Saturdays and release on Sundays, however I > propose moving rc1 a day earlier because I'll be unavailable that Sunday. > > Any objections? I will interpret silence as tacit approval. As I've already discussed with Larry, I think adding a week to the scheduled dates would be preferable. The original dates give pretty short notice and there are a number of open issues that would be good to resolve now in 3.4.2. -- Ned Deily, nad at acm.org From mal at egenix.com Mon Sep 8 10:09:26 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 08 Sep 2014 10:09:26 +0200 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: References: <5400DBC0.1020700@egenix.com> <5400DD64.4050308@stoneleaf.us> <20140831032525.19b7e48c@fsol> <20140831022149.F0493250E30@webabinitio.net> <540396E9.2020800@python.org> <54041C9D.1080404@python.org> Message-ID: <540D6436.4020906@egenix.com> On 01.09.2014 10:09, Nick Coghlan wrote: > On 1 September 2014 17:13, Christian Heimes wrote: >> On 01.09.2014 08:44, Nick Coghlan wrote: >>> Yes, it would have exactly the same security failure modes as >>> sitecustomize, except it would only fire if the application >>> imported the ssl module. >>> >>> The "-S" and "-I" switches would need to disable the implied >>> "sslcustomize", just as they disable "import site". >> >> A malicious package can already play havoc with your installation with >> a custom ssl module. If somebody is able to sneak in a ssl.py then you >> are screwed anyway. sslcustomize is not going to make the situation worse. > > That's not quite true - we're fairly careful about putting the > standard library before userspace directories, so aside from the > "current directory" problem, shadowing "ssl" itself can be tricky to > arrange. It's really easy to modify sys.modules to override any module that has already been loaded or add new ones bypassing sys.path entirely, so the sys.path layout doesn't provide any protection against such hacks. If you gain access to one of the dirs on sys.path, you can play such tricks in sitecustomize.py. Any 3rd party package can do the same. We'd have to add digital API signatures to the ssl module to prevent such stunts :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 11 days to go 2014-09-27: PyDDF Sprint 2014 ... 19 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Sep 8 14:08:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 8 Sep 2014 22:08:21 +1000 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: On 8 September 2014 14:28, Ned Deily wrote: > In article <540C521C.7070802 at hastings.org>, > Larry Hastings wrote: >> Matthias asked me when I was going to release 3.4.2. I propose the >> following schedule: >> >> Tag 3.4.2rc1 Friday Sep 12 2014 >> Release 3.4.2rc1 Saturday Sep 13 2014 >> Tag 3.4.2 final Saturday Sep 27 2014 >> Release 3.4.2 final Sunday Sep 28 2014 >> >> Normally I want to tag on Saturdays and release on Sundays, however I >> propose moving rc1 a day earlier because I'll be unavailable that Sunday. >> >> Any objections? I will interpret silence as tacit approval. > > As I've already discussed with Larry, I think adding a week to the > scheduled dates would be preferable. The original dates give pretty > short notice and there are a number of open issues that would be good to > resolve now in 3.4.2. It would also be good to get Guido's official verdict on PEP 476 (the switch to validating HTTPS by default) in time for 3.4.2. Based on the previous discussion, Alex updated the PEP to suggest "just fix it" for all of 3.5, 3.4.x and 2.7.x (including the httplib SSLContext support backport in the latter case). I think that would be feasible with an rc on the 20th, but challenging if the rc is this coming weekend. Note, as I stated in the previous thread, I'm now +1 on that PEP, because I don't see any way to write an automated scan for a large code base that ensures we're not relying on the default handling at all. If the default behaviour is to validate HTTPS, the lack of such a scanner isn't a problem - any failures to cope with self-signed or invalid certs will be noisy, and we can either fix the certs, patch the code to cope with them appropriately, or (for the self-signed cert case) configure OpenSSL via environment variables. If dealing with invalid certs is truly necessary, and the code can't be updated either, then I'm OK with the answer being "keep using an older version of Python, as that's going to be the least of your security concerns". Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jonas.wagner at epfl.ch Mon Sep 8 10:56:13 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Mon, 8 Sep 2014 10:56:13 +0200 Subject: [Python-Dev] cpython and parallel make In-Reply-To: References: <20140904180931.46ad9be0@fsol> Message-ID: Hi, Open an issue on bugs.python.org and attach the patch there (it should also > ask you so sign the contributor agreement, but if not then please also sign > that). > Submitted as http://bugs.python.org/issue22359 , and signed the agreement. Cheers, Jonas -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Sep 8 18:58:21 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 09:58:21 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: On Mon, Sep 8, 2014 at 5:08 AM, Nick Coghlan wrote: > > It would also be good to get Guido's official verdict on PEP 476 (the > switch to validating HTTPS by default) in time for 3.4.2. Based on the > previous discussion, Alex updated the PEP to suggest "just fix it" for > all of 3.5, 3.4.x and 2.7.x (including the httplib SSLContext support > backport in the latter case). > My opinion hasn't changed since the last time I opened my mouth prematurely. :-) I would very much like these to go in, but for 2.7 I am now worried about what we should tell people who have a script that uses an https URL to access a service that can only be accessed via SSL/TLS to a self-signed or otherwise mis-configured cert. I am not insisting on an environment variable to disable this (too easy) but I do think it must be possible to make a simple change to the code, on the order of tracking down the urlopen() call and adding a new keyword parameter. Such a band-aid needn't be backward compatible (we can introduce a new keyword parameter for this purpose) and it needn't be totally straightforward (we can assume some modicum of understanding of finding and editing .py files) but it should definitely not require a refactor of the script (e.g. swapping out urlopen and replacing it with httplib or requests would be too much of a burden). And we should have prominent documentation (perhaps in FAQ form?) with an example of how to do it. > I think that would be feasible with an rc on the 20th, but challenging > if the rc is this coming weekend. > > Note, as I stated in the previous thread, I'm now +1 on that PEP, > because I don't see any way to write an automated scan for a large > code base that ensures we're not relying on the default handling at > all. If the default behaviour is to validate HTTPS, the lack of such a > scanner isn't a problem - any failures to cope with self-signed or > invalid certs will be noisy, and we can either fix the certs, patch > the code to cope with them appropriately, or (for the self-signed cert > case) configure OpenSSL via environment variables. If dealing with > invalid certs is truly necessary, and the code can't be updated > either, then I'm OK with the answer being "keep using an older version > of Python, as that's going to be the least of your security concerns". > Yeah, I am not interested in helping out the case where the user is incapable (for whatever reason) of tracking down and changing a couple of lines of code. Such users are dependent on someone else with wizard powers anyway (who gave them the script?). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Mon Sep 8 19:23:32 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 08 Sep 2014 10:23:32 -0700 (PDT) Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: Message-ID: <540de614.075b8c0a.5db6.5396@mx.google.com> Summary: There needs to be a simple way to opt out at install time. It would be far better to offer more fine-grained control, but leaving that better solution to downstream is acceptable. On 3 September 2014 01:19, Antoine Pitrou wrote: > RFC 2818 (HTTP over TLS) has the following language in section 3.1: >> If the hostname is available, the client MUST check it against the >> server's identity as presented in the server's Certificate message, >> in order to prevent man-in-the-middle attacks. >> If the client has external information as to the expected identity of >> the server, the hostname check MAY be omitted. This second case is pretty common, in my experience. I still see it on the public internet, but mismatches are almost the expected case on the intranet, and many installation guides begin by saying to ignore the security warnings. I think it best not to name my employer in this context, but I work for an IT firm large enough that you've heard of it. As bad as our internal situation is, it is still better than a typical client's infrastructure, except that clients often have fewer surfaces to expose in the first place. Internal websites and applications tend to have information that "needs protection" only because saying otherwise requires a long bureaucratic process with little payoff. (Also true at many clients.) Nick has already posted a subset of the reasons why a site may be "signed" with a certificate that is self-signed, expired, and/or limited to the wrong hostname/subdomain. In the long run, I agree that it is better to default to secure. But in the short and medium term, there has to be a workaround, and I would prefer that the simplest workaround not be "retire the application, and don't use python again." I believe that the minimal acceptable workaround is that the Release Notes have an URL pointing to an install-time recipe telling the admin how to change the default back globally. Examples of good enough: "Add this file to site-packages" "Add this environment variable with this setting" "Add this command line switch to your launch script" Examples of not good enough: "Edit your application to change ..." "Edit your system store ..." (affecting more than python) Obviously, it would be great to offer finer control, so that the stricter default can be used when it is OK. (Per installation? Per application? Per run? Per domain? Per protocol? Per certificate? Per rejection reason? Treat anything in subdomain1.example.com as valid for hostname.example.com? Self-signing is OK for this IP range?) I would be pleasantly surprised if this level of API can even be standardized in time, and I agree that it is reasonable to leave it to 3rd party modules and downstream distributions. But I think Python itself should provide at least the single big hammer -- and that hammer should be something that can be used once at installation time (perhaps by changing the launch script), instead of requiring user interaction. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From alex.gaynor at gmail.com Mon Sep 8 19:25:14 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 8 Sep 2014 17:25:14 +0000 (UTC) Subject: [Python-Dev] Proposed schedule for 3.4.2 References: <540C521C.7070802@hastings.org> Message-ID: Guido van Rossum python.org> writes: > > Would you be willing to officially pronounce on PEP-476 in the context of 3.4.x, so we can get it into the release, and then we can defer on officially approving it for 2.7.X until we figure out all the moving pieces? Cheers, Alex From guido at python.org Mon Sep 8 19:35:57 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 10:35:57 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: I will pronounce for 3.4 once you point me to the documentation that explains how to disable cert validation for an example program that currently pulls down an https URL using urlopen. Without adding package dependencies. On Mon, Sep 8, 2014 at 10:25 AM, Alex Gaynor wrote: > Guido van Rossum python.org> writes: > > > > > > > Would you be willing to officially pronounce on PEP-476 in the context of > 3.4.x, > so we can get it into the release, and then we can defer on officially > approving > it for 2.7.X until we figure out all the moving pieces? > > Cheers, > Alex > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Mon Sep 8 19:44:51 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 8 Sep 2014 10:44:51 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: *Shifts uncomfortably* it looks like presently there's not a good way to change anything about the SSL configuration for urllib.request.urlopen. It does not take a `context` argument, as the http.client API does: https://docs.python.org/3/library/urllib.request.html#module-urllib.request and instead takes the cafile, capath, cadefault args. This would need to be updated first, once it *did* take such an argument, this would be accomplished by: context = ssl.create_default_context() context.verify_mode = CERT_OPTIONACERT_NONE context.verify_hostname = False urllib.request.urlopen("https://something-i-apparently-dont-care-much-about", context=context) Alex On Mon, Sep 8, 2014 at 10:35 AM, Guido van Rossum wrote: > I will pronounce for 3.4 once you point me to the documentation that > explains how to disable cert validation for an example program that > currently pulls down an https URL using urlopen. Without adding package > dependencies. > > On Mon, Sep 8, 2014 at 10:25 AM, Alex Gaynor > wrote: > >> Guido van Rossum python.org> writes: >> >> > >> > >> >> Would you be willing to officially pronounce on PEP-476 in the context of >> 3.4.x, >> so we can get it into the release, and then we can defer on officially >> approving >> it for 2.7.X until we figure out all the moving pieces? >> >> Cheers, >> Alex >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Sep 8 19:46:40 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 10:46:40 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: Well, get cracking then! :-) On Mon, Sep 8, 2014 at 10:44 AM, Alex Gaynor wrote: > *Shifts uncomfortably* it looks like presently there's not a good way to > change anything about the SSL configuration for urllib.request.urlopen. It > does not take a `context` argument, as the http.client API does: > https://docs.python.org/3/library/urllib.request.html#module-urllib.request > and instead takes the cafile, capath, cadefault args. > > This would need to be updated first, once it *did* take such an argument, > this would be accomplished by: > > context = ssl.create_default_context() > context.verify_mode = CERT_OPTIONACERT_NONE > context.verify_hostname = False > urllib.request.urlopen(" > https://something-i-apparently-dont-care-much-about", context=context) > > Alex > > > On Mon, Sep 8, 2014 at 10:35 AM, Guido van Rossum > wrote: > >> I will pronounce for 3.4 once you point me to the documentation that >> explains how to disable cert validation for an example program that >> currently pulls down an https URL using urlopen. Without adding package >> dependencies. >> >> On Mon, Sep 8, 2014 at 10:25 AM, Alex Gaynor >> wrote: >> >>> Guido van Rossum python.org> writes: >>> >>> > >>> > >>> >>> Would you be willing to officially pronounce on PEP-476 in the context >>> of 3.4.x, >>> so we can get it into the release, and then we can defer on officially >>> approving >>> it for 2.7.X until we figure out all the moving pieces? >>> >>> Cheers, >>> Alex >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Sep 8 19:57:17 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 8 Sep 2014 13:57:17 -0400 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: <20140908135717.34c781a1@anarchist.wooz.org> On Sep 08, 2014, at 10:44 AM, Alex Gaynor wrote: >*Shifts uncomfortably* it looks like presently there's not a good way to >change anything about the SSL configuration for urllib.request.urlopen. It >does not take a `context` argument, as the http.client API does: >https://docs.python.org/3/library/urllib.request.html#module-urllib.request >and instead takes the cafile, capath, cadefault args. > >This would need to be updated first, once it *did* take such an argument, >this would be accomplished by: > >context = ssl.create_default_context() >context.verify_mode = CERT_OPTIONACERT_NONE >context.verify_hostname = False >urllib.request.urlopen("https://something-i-apparently-dont-care-much-about", >context=context) There's probably an ugly hack possibility that uses unittest.mock.patch. ;) -Barry From jim.baker at python.org Mon Sep 8 20:11:31 2014 From: jim.baker at python.org (Jim Baker) Date: Mon, 8 Sep 2014 12:11:31 -0600 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: +1 for the suggested change to 2.7. Something I have put off in the work on SSL support in Jython 2.7 is what to do about the possibility of adding a large security hole to support standard Python behavior here with CERT_NONE. By default, we use the standard trust database and corresponding manager from Java, as augmented by any provided ca_certs. In practice, I don't think people are really noticing that it's currently locked down in the latest beta, which added this SSL support. Although it's very easy to open such a hole on Jython/Java, it's much nicer if we require the user has to do some work to do so, and on a per connection basis. - Jim On Mon, Sep 8, 2014 at 10:58 AM, Guido van Rossum wrote: > On Mon, Sep 8, 2014 at 5:08 AM, Nick Coghlan wrote: > >> >> It would also be good to get Guido's official verdict on PEP 476 (the >> switch to validating HTTPS by default) in time for 3.4.2. Based on the >> previous discussion, Alex updated the PEP to suggest "just fix it" for >> all of 3.5, 3.4.x and 2.7.x (including the httplib SSLContext support >> backport in the latter case). >> > > My opinion hasn't changed since the last time I opened my mouth > prematurely. :-) I would very much like these to go in, but for 2.7 I am > now worried about what we should tell people who have a script that uses an > https URL to access a service that can only be accessed via SSL/TLS to a > self-signed or otherwise mis-configured cert. I am not insisting on an > environment variable to disable this (too easy) but I do think it must be > possible to make a simple change to the code, on the order of tracking down > the urlopen() call and adding a new keyword parameter. Such a band-aid > needn't be backward compatible (we can introduce a new keyword parameter > for this purpose) and it needn't be totally straightforward (we can assume > some modicum of understanding of finding and editing .py files) but it > should definitely not require a refactor of the script (e.g. swapping out > urlopen and replacing it with httplib or requests would be too much of a > burden). And we should have prominent documentation (perhaps in FAQ form?) > with an example of how to do it. > > >> I think that would be feasible with an rc on the 20th, but challenging >> if the rc is this coming weekend. >> >> Note, as I stated in the previous thread, I'm now +1 on that PEP, >> because I don't see any way to write an automated scan for a large >> code base that ensures we're not relying on the default handling at >> all. If the default behaviour is to validate HTTPS, the lack of such a >> scanner isn't a problem - any failures to cope with self-signed or >> invalid certs will be noisy, and we can either fix the certs, patch >> the code to cope with them appropriately, or (for the self-signed cert >> case) configure OpenSSL via environment variables. If dealing with >> invalid certs is truly necessary, and the code can't be updated >> either, then I'm OK with the answer being "keep using an older version >> of Python, as that's going to be the least of your security concerns". >> > > Yeah, I am not interested in helping out the case where the user is > incapable (for whatever reason) of tracking down and changing a couple of > lines of code. Such users are dependent on someone else with wizard powers > anyway (who gave them the script?). > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Sep 8 21:44:52 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Sep 2014 20:44:52 +0100 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: <540de614.075b8c0a.5db6.5396@mx.google.com> References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: On 8 September 2014 18:23, Jim J. Jewett wrote: > Summary: There needs to be a simple way to opt out at install time. > It would be far better to offer more fine-grained control, but leaving > that better solution to downstream is acceptable. Does this argument apply to a hypothetical 2.7 backport of this change, or does it apply to making the change in 3.5? (Or of course both.) From greg at krypto.org Mon Sep 8 23:05:55 2014 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 8 Sep 2014 14:05:55 -0700 Subject: [Python-Dev] Sad status of Python 3.x buildbots In-Reply-To: References: Message-ID: On Wed, Sep 3, 2014 at 2:52 PM, Victor Stinner wrote: > > > ARMv7 3.x: "Read-only file system", Mercurial fails... > > I sent an email to Gregory P. Smith (owner). > For mine, its a case of me not having any monitoring for it. Sending me an email worked. In this case the SSD (a very fast usb3 memory stick) died after 9 months of use. Not a surprise. I brought it back up on the internal flash for the time being. I am in the process of setting up a new system (nvidia jetson tk1) to replace this one that will be 1.5-3x faster depending on how you measure it. Monitoring idea for buildbots: If builders for all non-custom branches on a buildbot are failing for more than some period of time or number of builds each in a row, that is likely a sign of broken infrastructure or a common platform specific code change that needs to be made. anyone want to implement that? -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Sep 8 23:35:19 2014 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 8 Sep 2014 14:35:19 -0700 Subject: [Python-Dev] PEP 476: Enabling certificate validation by default! In-Reply-To: <87d2bc8gnu.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5B98529F-CD35-4D0E-B60F-0221441395A8@twistedmatrix.com> <20140902233208.4e580ef4@fsol> <55B9F814-3CB7-4ECF-8AFC-ECAD182BAFF6@twistedmatrix.com> <20140903012917.1E8D3B1408D@webabinitio.net> <20140903163113.2165f744@fsol> <20140903155808.08F50250E04@webabinitio.net> <87d2bc8gnu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 3, 2014 at 3:48 PM, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > > lot: five years ago (when I worked at Google!) it was common to find > > internal services that required SSL but had a misconfigured certificate, > > and the only way to access those services was to override the browser > > complaints. Today (working at Dropbox, a much smaller company!) I don't > > even remember the last time I had to deal with such a browser complaint > -- > > I would tend to discount your recent experience, then. Smaller (and > possibly even more important in this fast-developing area, younger) > organizations are a lot more nimble about things like this. > As a defensive data point: I don't remember a single instance of this happening for Google internal services, at least since I arrived in 2007. I'm not doubting that Guido remembers some thing(s) but in general people here at Google would not stand for that, then or now. I would not call it common, especially five years ago. Common things I _have_ encountered over the years everywhere I've been both internal and external: services that listen on the https port 443 but don't have a valid cert as they are intended only for http port 80 access. Those are becoming somewhat less common, the only thing I regularly see that on anymore is random home router web config UIs as issuing a signed server certificate for security hole ridden commodity embedded box is... a challenge. (I'm not commenting on the PEP plans as it seems the right things are happening for now) -gps @ Google -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 00:20:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 08:20:52 +1000 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <20140908135717.34c781a1@anarchist.wooz.org> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> Message-ID: On 9 Sep 2014 04:00, "Barry Warsaw" wrote: > > > >This would need to be updated first, once it *did* take such an argument, > >this would be accomplished by: > > > >context = ssl.create_default_context() > >context.verify_mode = CERT_OPTIONACERT_NONE > >context.verify_hostname = False > >urllib.request.urlopen(" https://something-i-apparently-dont-care-much-about", > >context=context) > > There's probably an ugly hack possibility that uses unittest.mock.patch. ;) We could actually make it an "official" hack: import urllib.request urllib.request.urlopen = urllib.request._unverified_urlopen Or else the user can just change the code to call the unverified one directly. All we'd have to do is keep the existing version that doesn't validate certs properly around under the name "_unverified_urlopen". I like this for a few reasons: 1. It doesn't get much easier than calling function A instead of function B 2. Monkeypatching lets you do a process global hack 3. The name tells you exactly why this is a bad idea 4. It's easy to grep for later after you fix your certs 5. The leading underscore acts as a strong "keep away" signal 6. The leading underscore makes it clear this function may not always be available (e.g. Jython, older versions of Python) Cheers, Nick. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Sep 9 00:28:46 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 8 Sep 2014 18:28:46 -0400 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> Message-ID: <20140908182846.2f021575@anarchist.wooz.org> On Sep 09, 2014, at 08:20 AM, Nick Coghlan wrote: >We could actually make it an "official" hack: > > import urllib.request > urllib.request.urlopen = urllib.request._unverified_urlopen > >Or else the user can just change the code to call the unverified one >directly. > >All we'd have to do is keep the existing version that doesn't validate >certs properly around under the name "_unverified_urlopen". > >I like this for a few reasons: > >1. It doesn't get much easier than calling function A instead of function B >2. Monkeypatching lets you do a process global hack >3. The name tells you exactly why this is a bad idea >4. It's easy to grep for later after you fix your certs >5. The leading underscore acts as a strong "keep away" signal >6. The leading underscore makes it clear this function may not always be >available (e.g. Jython, older versions of Python) +1. This would also make it easy to mock in a context manager if you just wanted to ignore certs for a small section of code. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From donald at stufft.io Tue Sep 9 00:30:35 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 8 Sep 2014 18:30:35 -0400 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> Message-ID: <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> > On Sep 8, 2014, at 6:20 PM, Nick Coghlan wrote: > > > On 9 Sep 2014 04:00, "Barry Warsaw" > wrote: > > > > > >This would need to be updated first, once it *did* take such an argument, > > >this would be accomplished by: > > > > > >context = ssl.create_default_context() > > >context.verify_mode = CERT_OPTIONACERT_NONE > > >context.verify_hostname = False > > >urllib.request.urlopen("https://something-i-apparently-dont-care-much-about ", > > >context=context) > > > > There's probably an ugly hack possibility that uses unittest.mock.patch. ;) > > We could actually make it an "official" hack: > > import urllib.request > urllib.request.urlopen = urllib.request._unverified_urlopen > > Or else the user can just change the code to call the unverified one directly. > > All we'd have to do is keep the existing version that doesn't validate certs properly around under the name "_unverified_urlopen". > > I like this for a few reasons: > > 1. It doesn't get much easier than calling function A instead of function B > 2. Monkeypatching lets you do a process global hack > 3. The name tells you exactly why this is a bad idea > 4. It's easy to grep for later after you fix your certs > 5. The leading underscore acts as a strong "keep away" signal > 6. The leading underscore makes it clear this function may not always be available (e.g. Jython, older versions of Python) > > If someone wants to do this, can?t they write their own 6 line function? import ssl import urllib.request _real_urlopen = urllib.request.urlopen def _unverified(*args, **kwargs): if not kwargs.keys() & {?context?, ?cafile?, ?capath?, ?cadefault?}: ctx = ssl.create_default_context() ctx.verify_mode = CERT_NONE ctx.verify_hostname = False kwargs[?context?] = ctx return _real_urlopen(*args, **kwargs) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 00:35:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 08:35:03 +1000 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> Message-ID: On 9 Sep 2014 08:20, "Nick Coghlan" wrote: > > > On 9 Sep 2014 04:00, "Barry Warsaw" wrote: > > > > > >This would need to be updated first, once it *did* take such an argument, > > >this would be accomplished by: > > > > > >context = ssl.create_default_context() > > >context.verify_mode = CERT_OPTIONACERT_NONE > > >context.verify_hostname = False > > >urllib.request.urlopen(" https://something-i-apparently-dont-care-much-about", > > >context=context) > > > > There's probably an ugly hack possibility that uses unittest.mock.patch. ;) > > We could actually make it an "official" hack: > > import urllib.request > urllib.request.urlopen = urllib.request._unverified_urlopen Thinking about it a bit more, I suspect httplib would be the right level for such a hack. Either way, I actually think a monkeypatching based solution is a reasonable choice here. You can downgrade back to the old behaviour selectively (calling the unverified version or monkeypatching the calling module) or globally (monkeypatching the httplib module) If folks go "Ewww, I'm going to fix my code or certs instead", that's a good outcome :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 00:27:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 08:27:54 +1000 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: <540de614.075b8c0a.5db6.5396@mx.google.com> References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: On 9 Sep 2014 03:25, "Jim J. Jewett" wrote: > Examples of good enough: > "Add this file to site-packages" > "Add this environment variable with this setting" > "Add this command line switch to your launch script" The monkeypatching hack I proposed would work correctly in sitecustomize (or a *.pth file). Cost would be loading httplib in all cases. A -X option for "noverifyhttps" might also be feasible. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 00:43:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 08:43:51 +1000 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> Message-ID: On 9 Sep 2014 08:30, "Donald Stufft" wrote: > > If someone wants to do this, can?t they write their own 6 line function? Unfortunately not, as the domain knowledge required to know what those six lines should look like is significant. Keeping the old unsafe behaviour around with a more obviously dangerous name is much simpler than explaining to people "Here, copy this chunk of code you don't understand". If we were starting with a blank slate there's no way we'd offer such a thing, but as Jim pointed out, we do want to make it relatively easy for Standard Operating Environment maintainers to hack around it if necessary. Cheers, Nick. > > import ssl > import urllib.request > _real_urlopen = urllib.request.urlopen > def _unverified(*args, **kwargs): > if not kwargs.keys() & {?context?, ?cafile?, ?capath?, ?cadefault?}: > ctx = ssl.create_default_context() > ctx.verify_mode = CERT_NONE > ctx.verify_hostname = False > kwargs[?context?] = ctx > return _real_urlopen(*args, **kwargs) > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Sep 9 00:48:39 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 8 Sep 2014 18:48:39 -0400 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> Message-ID: <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> > On Sep 8, 2014, at 6:43 PM, Nick Coghlan wrote: > > > On 9 Sep 2014 08:30, "Donald Stufft" > wrote: > > > > If someone wants to do this, can?t they write their own 6 line function? > > Unfortunately not, as the domain knowledge required to know what those six lines should look like is significant. > > Keeping the old unsafe behaviour around with a more obviously dangerous name is much simpler than explaining to people "Here, copy this chunk of code you don't understand". > > If we were starting with a blank slate there's no way we'd offer such a thing, but as Jim pointed out, we do want to make it relatively easy for Standard Operating Environment maintainers to hack around it if necessary. > > Cheers, > Nick. > > > > > import ssl > > import urllib.request > > _real_urlopen = urllib.request.urlopen > > def _unverified(*args, **kwargs): > > if not kwargs.keys() & {?context?, ?cafile?, ?capath?, ?cadefault?}: > > ctx = ssl.create_default_context() > > ctx.verify_mode = CERT_NONE > > ctx.verify_hostname = False > > kwargs[?context?] = ctx > > return _real_urlopen(*args, **kwargs) > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > Why isn?t documentation with appropriate red warnings a suitable place if we really must have it? That sounds like a much better solution that some weird function people monkeypatch. It gives them more control over things (maybe they have a valid certificate chain, but an invalid host name!), it?ll work across all Python implementations, and most importantly, it gives us a place where there is some long form location to be like ?yea you really probably don?t want to be doing this? in big red letters. Overall I?m -1 on either offering the function or documenting it at all, but if we must do something then I think documentation is more than enough. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 9 01:31:24 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 16:31:24 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> Message-ID: I still prefer having a parameter on urlopen (or thereabouts) -- it feels wrong to make it easier to change this globally than on a per-call basis, and if you don't understand monkey-patching, it's impossible to debug if you put the patch in the wrong place. For the poor soul who has a script with many urlopen("https"//") calls, well, they probably don't mind the busywork of editing each and every one of them. I'm fine with giving the actual keyword parameter a scary-sounding ugly name. On Mon, Sep 8, 2014 at 3:48 PM, Donald Stufft wrote: > > On Sep 8, 2014, at 6:43 PM, Nick Coghlan wrote: > > > On 9 Sep 2014 08:30, "Donald Stufft" wrote: > > > > If someone wants to do this, can?t they write their own 6 line function? > > Unfortunately not, as the domain knowledge required to know what those six > lines should look like is significant. > > Keeping the old unsafe behaviour around with a more obviously dangerous > name is much simpler than explaining to people "Here, copy this chunk of > code you don't understand". > > If we were starting with a blank slate there's no way we'd offer such a > thing, but as Jim pointed out, we do want to make it relatively easy for > Standard Operating Environment maintainers to hack around it if necessary. > > Cheers, > Nick. > > > > > import ssl > > import urllib.request > > _real_urlopen = urllib.request.urlopen > > def _unverified(*args, **kwargs): > > if not kwargs.keys() & {?context?, ?cafile?, ?capath?, ?cadefault?}: > > ctx = ssl.create_default_context() > > ctx.verify_mode = CERT_NONE > > ctx.verify_hostname = False > > kwargs[?context?] = ctx > > return _real_urlopen(*args, **kwargs) > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > > Why isn?t documentation with appropriate red warnings a suitable place if > we really must have it? That sounds like a much better solution that some > weird function people monkeypatch. It gives them more control over things > (maybe they have a valid certificate chain, but an invalid host name!), > it?ll work across all Python implementations, and most importantly, it > gives us a place where there is some long form location to be like ?yea you > really probably don?t want to be doing this? in big red letters. > > Overall I?m -1 on either offering the function or documenting it at all, > but if we must do something then I think documentation is more than enough. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Tue Sep 9 02:40:43 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 8 Sep 2014 20:40:43 -0400 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: On Mon, Sep 8, 2014 at 3:44 PM, Cory Benfield wrote: > On 8 September 2014 18:23, Jim J. Jewett wrote: >> Summary: There needs to be a simple way to opt out at install time. >> It would be far better to offer more fine-grained control, but leaving >> that better solution to downstream is acceptable. > Does this argument apply to a hypothetical 2.7 backport of this > change, or does it apply to making the change in 3.5? (Or of course > both.) I believe the argument applies even to 3.5, given that there was no deprecation period. The concern is obviously stronger for maintenance releases. I am not saying that secure-by-default should wait until until 3.6; I am saying that the "rush" requires even more attention than usual to backwards compatibility. This actually argues *for* backporting the fix as at least opt-in, so that 2.7/3.4 can serve as the "make your changes now, test them without all the other new features" releases. Nick's suggestion of a monkey-patching .pth file would be sufficient backwards compatibility support, if the recipe were referenced from the release notes (not just the python lib documentation). Support for partial opt-in -- whether per-process, per call, per address, etc -- would be nice, but it isn't required for backwards compatibility. I think that means an -X option for "noverifyhttps" should NOT be added. It doesn't get users closer to the final solution; it just adds the noise of a different workaround. I assume that adding _unverified_urlopen or urlopen(context=...) do provide incremental improvements compatible with the eventual full opt-in. If so, adding them is probably reasonable, but I think the PEP should explicitly list all such approved half-measures as a guard against API feature creep. -jJ From v+python at g.nevcal.com Tue Sep 9 04:46:07 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 08 Sep 2014 19:46:07 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> Message-ID: <540E69EF.6090005@g.nevcal.com> Well, this thread seems to be top-posted.... so... Why not provide _urlopen_with_scary_keyword_parameter as the monkey-patch option? So after the (global to the module) monkeypatch, they would _still_ have to add the keyword parameter. On 9/8/2014 4:31 PM, Guido van Rossum wrote: > I still prefer having a parameter on urlopen (or thereabouts) -- it > feels wrong to make it easier to change this globally than on a > per-call basis, and if you don't understand monkey-patching, it's > impossible to debug if you put the patch in the wrong place. > > For the poor soul who has a script with many > urlopen("https"//") calls, well, they probably don't mind > the busywork of editing each and every one of them. > > I'm fine with giving the actual keyword parameter a scary-sounding > ugly name. > > On Mon, Sep 8, 2014 at 3:48 PM, Donald Stufft > wrote: > > >> On Sep 8, 2014, at 6:43 PM, Nick Coghlan > > wrote: >> >> >> On 9 Sep 2014 08:30, "Donald Stufft" > > wrote: >> > >> > If someone wants to do this, can?t they write their own 6 line >> function? >> >> Unfortunately not, as the domain knowledge required to know what >> those six lines should look like is significant. >> >> Keeping the old unsafe behaviour around with a more obviously >> dangerous name is much simpler than explaining to people "Here, >> copy this chunk of code you don't understand". >> >> If we were starting with a blank slate there's no way we'd offer >> such a thing, but as Jim pointed out, we do want to make it >> relatively easy for Standard Operating Environment maintainers to >> hack around it if necessary. >> >> Cheers, >> Nick. >> >> > >> > import ssl >> > import urllib.request >> > _real_urlopen = urllib.request.urlopen >> > def _unverified(*args, **kwargs): >> > if not kwargs.keys() & {?context?, ?cafile?, ?capath?, >> ?cadefault?}: >> > ctx = ssl.create_default_context() >> > ctx.verify_mode = CERT_NONE >> > ctx.verify_hostname = False >> > kwargs[?context?] = ctx >> > return _real_urlopen(*args, **kwargs) >> > >> > --- >> > Donald Stufft >> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > >> > > Why isn?t documentation with appropriate red warnings a suitable > place if we really must have it? That sounds like a much better > solution that some weird function people monkeypatch. It gives > them more control over things (maybe they have a valid certificate > chain, but an invalid host name!), it?ll work across all Python > implementations, and most importantly, it gives us a place where > there is some long form location to be like ?yea you really > probably don?t want to be doing this? in big red letters. > > Overall I?m -1 on either offering the function or documenting it > at all, but if we must do something then I think documentation is > more than enough. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 05:03:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 13:03:53 +1000 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: On 9 Sep 2014 10:48, "Jim J. Jewett" wrote: > I assume that adding _unverified_urlopen or urlopen(context=...) do > provide incremental improvements compatible with the eventual full > opt-in. If so, adding them is probably reasonable, but I think the > PEP should explicitly list all such approved half-measures as a guard > against API feature creep. >From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now): 1. "context" parameter support in urllib.request (to opt out on a per-call basis) 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching) The former change seems non-controversial. I think the more fine-grained solution for the latter can wait until 3.5 (and will be an independent PEP), we just need an interim workaround for 3.4 that could conceivably be backported to 2.7. On that front, ssl currently has two context factories: get_default_context() and _get_stdlib_context. One possible option would be to: 1. Rename "_get_stdlib_context" to "_get_unverified_context" 2. Add "_get_https_context" as an alias for "get_default_context" Opting out on a per-call basis means passing an unverified context. Opting out globally would mean monkeypatching _get_https_context to refer to _get_unverified_context instead. These would both be documented as part of transition, but with clear security warnings. The use of the leading underscores in the names is intended to emphasise "you probably don't want to be using this". Regards, Nick. > > -jJ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 9 05:32:24 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 20:32:24 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <540E69EF.6090005@g.nevcal.com> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> <540E69EF.6090005@g.nevcal.com> Message-ID: Replacing urllib.urlopen(url) with urllib._unsafe_urlopen_without_secure_https(url) would be fine too (actual name to be picked by whoever writes the code) but I don't see that it offers much more of a barrier against abuse of this compatibility feature compared to a keyword argument. Requiring a monkeypatch feels unnecessarily mean -- I see no reason why the code can't be in the standard library. It's a bit like the emergency hammer on a train -- what keeps riders from misusing it is convention (and the sign next to it), since locking it up would miss the point. Do note that there are a couple of different common patterns for how this is used in legacy code, e.g. urllib vs.urllib2, URLOpener vs FancyURLOpener, urlopen vs. urlretrieve; there are also some internal calls, e.g. in response to redirects. The ultimate form of the solution (keyword argument of alternate function or whatever) may depend on the needs of these various (ancient) architectures. Regarding 3.4 and 3.5, there's presumably much less legacy code for 3.4, but its expected lifetime is also much shorter than 2.7's (since we're already close to releasing 3.5). So I'm still a bit torn -- in the end one reason to do it in 3.4 is that 3.4 shouldn't have a weaker default than 2.7. Onwards, On Mon, Sep 8, 2014 at 7:46 PM, Glenn Linderman wrote: > Well, this thread seems to be top-posted.... so... > > Why not provide _urlopen_with_scary_keyword_parameter as the monkey-patch > option? > > So after the (global to the module) monkeypatch, they would _still_ have > to add the keyword parameter. > > > > On 9/8/2014 4:31 PM, Guido van Rossum wrote: > > I still prefer having a parameter on urlopen (or thereabouts) -- it feels > wrong to make it easier to change this globally than on a per-call basis, > and if you don't understand monkey-patching, it's impossible to debug if > you put the patch in the wrong place. > > For the poor soul who has a script with many urlopen("https"//") > calls, well, they probably don't mind the busywork of editing each and > every one of them. > > I'm fine with giving the actual keyword parameter a scary-sounding ugly > name. > > On Mon, Sep 8, 2014 at 3:48 PM, Donald Stufft wrote: > >> >> On Sep 8, 2014, at 6:43 PM, Nick Coghlan wrote: >> >> >> On 9 Sep 2014 08:30, "Donald Stufft" wrote: >> > >> > If someone wants to do this, can?t they write their own 6 line function? >> >> Unfortunately not, as the domain knowledge required to know what those >> six lines should look like is significant. >> >> Keeping the old unsafe behaviour around with a more obviously dangerous >> name is much simpler than explaining to people "Here, copy this chunk of >> code you don't understand". >> >> If we were starting with a blank slate there's no way we'd offer such a >> thing, but as Jim pointed out, we do want to make it relatively easy for >> Standard Operating Environment maintainers to hack around it if necessary. >> >> Cheers, >> Nick. >> >> > >> > import ssl >> > import urllib.request >> > _real_urlopen = urllib.request.urlopen >> > def _unverified(*args, **kwargs): >> > if not kwargs.keys() & {?context?, ?cafile?, ?capath?, ?cadefault?}: >> > ctx = ssl.create_default_context() >> > ctx.verify_mode = CERT_NONE >> > ctx.verify_hostname = False >> > kwargs[?context?] = ctx >> > return _real_urlopen(*args, **kwargs) >> > >> > --- >> > Donald Stufft >> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > >> >> >> Why isn?t documentation with appropriate red warnings a suitable place >> if we really must have it? That sounds like a much better solution that >> some weird function people monkeypatch. It gives them more control over >> things (maybe they have a valid certificate chain, but an invalid host >> name!), it?ll work across all Python implementations, and most importantly, >> it gives us a place where there is some long form location to be like ?yea >> you really probably don?t want to be doing this? in big red letters. >> >> Overall I?m -1 on either offering the function or documenting it at >> all, but if we must do something then I think documentation is more than >> enough. >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 9 05:40:18 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Sep 2014 20:40:18 -0700 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: The multiple threads going on are confusing (or maybe GMail makes them more confusing), but the architecture you are sketching here sounds good. I can't find get_default_context() in the repo, but perhaps I need to refresh, or perhaps you're talking about a design in a PEP. On Mon, Sep 8, 2014 at 8:03 PM, Nick Coghlan wrote: > > On 9 Sep 2014 10:48, "Jim J. Jewett" wrote: > > I assume that adding _unverified_urlopen or urlopen(context=...) do > > provide incremental improvements compatible with the eventual full > > opt-in. If so, adding them is probably reasonable, but I think the > > PEP should explicitly list all such approved half-measures as a guard > > against API feature creep. > > From Guido's and your feedback, I think we may need two things to approve > this for 3.4.2 (putting 2.7 aside for now): > > 1. "context" parameter support in urllib.request (to opt out on a per-call > basis) > 2. a documented way to restore the old behaviour via sitecustomize (which > may involve monkeypatching) > > The former change seems non-controversial. > > I think the more fine-grained solution for the latter can wait until 3.5 > (and will be an independent PEP), we just need an interim workaround for > 3.4 that could conceivably be backported to 2.7. > > On that front, ssl currently has two context factories: > get_default_context() and _get_stdlib_context. One possible option would be > to: > > 1. Rename "_get_stdlib_context" to "_get_unverified_context" > 2. Add "_get_https_context" as an alias for "get_default_context" > > Opting out on a per-call basis means passing an unverified context. > > Opting out globally would mean monkeypatching _get_https_context to refer > to _get_unverified_context instead. > > These would both be documented as part of transition, but with clear > security warnings. The use of the leading underscores in the names is > intended to emphasise "you probably don't want to be using this". > > Regards, > Nick. > > > > > > -jJ > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Sep 9 05:41:52 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 09 Sep 2014 12:41:52 +0900 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <540E69EF.6090005@g.nevcal.com> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> <540E69EF.6090005@g.nevcal.com> Message-ID: <87egvl794v.fsf@uwakimon.sk.tsukuba.ac.jp> Glenn Linderman writes: > Well, this thread seems to be top-posted.... so... Not a good enough reason for me! > Why not provide _urlopen_with_scary_keyword_parameter as the > monkey-patch option? > > So after the (global to the module) monkeypatch, they would _still_ have > to add the keyword parameter. I understand the hardline position, though I don't like it: "if you don't know how to do it yourself, we won't help you do it at all."[1] But this "defense in depth" suggestion really violates the "consenting adults" principle. One warning in the docs and another in the name itself should be enough, and if it isn't, Mommy should take Jimmy's RaspberryPi away. Footnotes: [1] Personally, I think that taken seriously, this reasoning applies to anybody who uses computers for anything other than programming, though. Should anybody be allowed to use computers, given that they're going to put their personal data on Facebook for their stalkers to see or inadvertently install botnet software with whatever warez they are weak for? From donald at stufft.io Tue Sep 9 05:44:46 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 8 Sep 2014 23:44:46 -0400 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: It?s create_default_context() -> https://docs.python.org/3.4/library/ssl.html#ssl.create_default_context > On Sep 8, 2014, at 11:40 PM, Guido van Rossum wrote: > > The multiple threads going on are confusing (or maybe GMail makes them more confusing), but the architecture you are sketching here sounds good. I can't find get_default_context() in the repo, but perhaps I need to refresh, or perhaps you're talking about a design in a PEP. > > On Mon, Sep 8, 2014 at 8:03 PM, Nick Coghlan > wrote: > > On 9 Sep 2014 10:48, "Jim J. Jewett" > wrote: > > I assume that adding _unverified_urlopen or urlopen(context=...) do > > provide incremental improvements compatible with the eventual full > > opt-in. If so, adding them is probably reasonable, but I think the > > PEP should explicitly list all such approved half-measures as a guard > > against API feature creep. > > From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now): > > 1. "context" parameter support in urllib.request (to opt out on a per-call basis) > 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching) > > The former change seems non-controversial. > > I think the more fine-grained solution for the latter can wait until 3.5 (and will be an independent PEP), we just need an interim workaround for 3.4 that could conceivably be backported to 2.7. > > On that front, ssl currently has two context factories: get_default_context() and _get_stdlib_context. One possible option would be to: > > 1. Rename "_get_stdlib_context" to "_get_unverified_context" > 2. Add "_get_https_context" as an alias for "get_default_context" > > Opting out on a per-call basis means passing an unverified context. > > Opting out globally would mean monkeypatching _get_https_context to refer to _get_unverified_context instead. > > These would both be documented as part of transition, but with clear security warnings. The use of the leading underscores in the names is intended to emphasise "you probably don't want to be using this". > > Regards, > Nick. > > > > > > > -jJ > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Sep 9 06:32:32 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 08 Sep 2014 21:32:32 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: <87egvl794v.fsf@uwakimon.sk.tsukuba.ac.jp> References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> <75C56C44-B034-4991-80FB-D9EB6D42AF67@stufft.io> <69A191A5-6EF3-4B35-8FA8-B410A1631245@stufft.io> <540E69EF.6090005@g.nevcal.com> <87egvl794v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <540E82E0.1090205@g.nevcal.com> On 9/8/2014 8:41 PM, Stephen J. Turnbull wrote: > > Why not provide _urlopen_with_scary_keyword_parameter as the > > monkey-patch option? > > > > So after the (global to the module) monkeypatch, they would_still_ have > > to add the keyword parameter. > > I understand the hardline position, though I don't like it: "if you > don't know how to do it yourself, we won't help you do it at all."[1] > > But this "defense in depth" suggestion really violates the "consenting > adults" principle. One warning in the docs and another in the name > itself should be enough, and if it isn't, Mommy should take Jimmy's > RaspberryPi away. I was assuming, because of the suggestion for a monkey patch at all, in response to Guido's suggestion of a keyword parameter, that there was a problem adding a keyword parameter to urlopen. If there is, then the combo above could be useful in making them track down and adjust the places that need it, without forcing them to adjust the places that don't need it? If there is not, then no need for the monkey patch at all, they can just change add the keyword parameter. If the alternate function doesn't have an extra keyword parameter, the monkeypatch solution would be the "easy" way to apply the change globally, even to places that don't need it, or optionally don't need it... too big a hammer. Having the extra parameter also might make them not apply it globally, and think more about what and why they are doing what they are doing. Whatever, the idea is out there. If no one likes it, let it die. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 12:52:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Sep 2014 20:52:12 +1000 Subject: [Python-Dev] Suggested changes to verify HTTPS by default (was Re: Proposed schedule for 3.4.2) Message-ID: On 9 September 2014 03:44, Alex Gaynor wrote: > *Shifts uncomfortably* it looks like presently there's not a good way to > change anything about the SSL configuration for urllib.request.urlopen. It > does not take a `context` argument, as the http.client API does: > https://docs.python.org/3/library/urllib.request.html#module-urllib.request > and instead takes the cafile, capath, cadefault args. > > This would need to be updated first, once it *did* take such an argument, > this would be accomplished by: > > context = ssl.create_default_context() > context.verify_mode = CERT_OPTIONACERT_NONE > context.verify_hostname = False > urllib.request.urlopen("https://something-i-apparently-dont-care-much-about", > context=context) I'd never needed to use the existing global configuration settings in urllib.request before, but it actually *does* already support setting the default opener for urllib.urlopen. To explicitly set it to use verified HTTPS by default: import ssl, urllib.request https_handler = HTTPSHandler(context=ssl.create_default_context(), check_hostname=True) urllib.request.install_opener(urllib.request.build_opener(https_handler) When the default changes, turning off verification by default for urllib.request.urlopen would look like: import ssl, urllib.request unverified_context = ssl.create_default_context() unverified_context.verify_mode = CERT_OPTIONACERT_NONE unverified_context.verify_hostname = False unverified_handler = HTTPSHandler(context=unverified_context, check_hostname=False) urllib.request.install_opener(urllib.request.build_opener(unverified_handler) However, even setting the opener like that still leaves http.client.HTTPSConnection, urllib.request.URLOpener and urllib.request.FancyURLOpener using unverified HTTPS with no easy way to change their default behaviour. That means some other approach to global configuration is going to be needed to cover the "coping with legacy corporate infrastructure" case, and I still think a monkeypatching based hack is likely to be our best bet. So, focusing on 3.4, but in a way that should be feasible to backport, the changes that I now believe would be needed are: 1. Add "context" arguments to urlopen, URLOpener and FancyURLOpener (the latter two have been deprecated since 3.3, but this would make things easier for a subsequent 2.7 backport) 2. Add a ssl._create_https_context() alias for ssl.create_default_context() 3. Change urllib.request.urlopen() and http.client.HTTPSConnection to call ssl_create_https_context() rather than ssl._create_stdlib_context() 4. Rename ssl._create_stdlib_context() to ssl._create_unverified_context() (updating call sites accordingly) To revert any given call site to the old behaviour: http.client.HTTPSConnection(context=ssl._create_unverified_context()) urllib.request.urlopen(context=ssl._create_unverified_context()) urllib.request.URLOpener(context=ssl._create_unverified_context()) urllib.request.FancyURLOpener(context=ssl._create_unverified_context()) And to revert to the old default behaviour globally: import ssl ssl._create_https_context = ssl._create_unverified_context The backport to 2.7 would then be a matter of bringing urllib, urllib2, httplib and ssl into line with their 3.4.2 counterparts. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Tue Sep 9 18:11:56 2014 From: christian at python.org (Christian Heimes) Date: Tue, 09 Sep 2014 18:11:56 +0200 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> Message-ID: <540F26CC.4010608@python.org> On 09.09.2014 05:03, Nick Coghlan wrote: > > On 9 Sep 2014 10:48, "Jim J. Jewett" > wrote: >> I assume that adding _unverified_urlopen or urlopen(context=...) do >> provide incremental improvements compatible with the eventual full >> opt-in. If so, adding them is probably reasonable, but I think the >> PEP should explicitly list all such approved half-measures as a guard >> against API feature creep. > > From Guido's and your feedback, I think we may need two things to > approve this for 3.4.2 (putting 2.7 aside for now): > > 1. "context" parameter support in urllib.request (to opt out on a > per-call basis) > 2. a documented way to restore the old behaviour via sitecustomize > (which may involve monkeypatching) What's with our plan to introduce sslcustomize? Is the idea for a configuration module and named contexts off the table? For reference: I came up with the idea to introduce the module "sslcustomize" similar to sitecustomize. Contrary to sitecustomize the module is imported at the end of the ssl module. Based on my idea Nick proposed another addition to the SSL module. He proposed a ssl.named_contexts mapping from module names to factory functions that create SSLContext objects. http://permalink.gmane.org/gmane.comp.python.devel/149292 I still prefer the general idea over the monkey patching idea because it provides a clean but simple interface for structured configuration. Monkey patching of stdlib modules is ugly and error-prone. Christian From guido at python.org Tue Sep 9 19:08:51 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Sep 2014 10:08:51 -0700 Subject: [Python-Dev] Suggested changes to verify HTTPS by default (was Re: Proposed schedule for 3.4.2) In-Reply-To: References: Message-ID: I'm going to leave the design up to Nick and friends for a while. Let me know when there is a patch to review. On Tue, Sep 9, 2014 at 3:52 AM, Nick Coghlan wrote: > On 9 September 2014 03:44, Alex Gaynor wrote: > > *Shifts uncomfortably* it looks like presently there's not a good way to > > change anything about the SSL configuration for urllib.request.urlopen. > It > > does not take a `context` argument, as the http.client API does: > > > https://docs.python.org/3/library/urllib.request.html#module-urllib.request > > and instead takes the cafile, capath, cadefault args. > > > > This would need to be updated first, once it *did* take such an argument, > > this would be accomplished by: > > > > context = ssl.create_default_context() > > context.verify_mode = CERT_OPTIONACERT_NONE > > context.verify_hostname = False > > urllib.request.urlopen(" > https://something-i-apparently-dont-care-much-about", > > context=context) > > I'd never needed to use the existing global configuration settings in > urllib.request before, but it actually *does* already support setting > the default opener for urllib.urlopen. > > To explicitly set it to use verified HTTPS by default: > > import ssl, urllib.request > https_handler = HTTPSHandler(context=ssl.create_default_context(), > check_hostname=True) > > urllib.request.install_opener(urllib.request.build_opener(https_handler) > > When the default changes, turning off verification by default for > urllib.request.urlopen would look like: > > import ssl, urllib.request > unverified_context = ssl.create_default_context() > unverified_context.verify_mode = CERT_OPTIONACERT_NONE > unverified_context.verify_hostname = False > unverified_handler = HTTPSHandler(context=unverified_context, > check_hostname=False) > > urllib.request.install_opener(urllib.request.build_opener(unverified_handler) > > However, even setting the opener like that still leaves > http.client.HTTPSConnection, urllib.request.URLOpener and > urllib.request.FancyURLOpener using unverified HTTPS with no easy way > to change their default behaviour. > > That means some other approach to global configuration is going to be > needed to cover the "coping with legacy corporate infrastructure" > case, and I still think a monkeypatching based hack is likely to be > our best bet. > > So, focusing on 3.4, but in a way that should be feasible to backport, > the changes that I now believe would be needed are: > > 1. Add "context" arguments to urlopen, URLOpener and FancyURLOpener > (the latter two have been deprecated since 3.3, but this would make > things easier for a subsequent 2.7 backport) > 2. Add a ssl._create_https_context() alias for ssl.create_default_context() > 3. Change urllib.request.urlopen() and http.client.HTTPSConnection to > call ssl_create_https_context() rather than > ssl._create_stdlib_context() > 4. Rename ssl._create_stdlib_context() to > ssl._create_unverified_context() (updating call sites accordingly) > > To revert any given call site to the old behaviour: > > http.client.HTTPSConnection(context=ssl._create_unverified_context()) > urllib.request.urlopen(context=ssl._create_unverified_context()) > urllib.request.URLOpener(context=ssl._create_unverified_context()) > urllib.request.FancyURLOpener(context=ssl._create_unverified_context()) > > And to revert to the old default behaviour globally: > > import ssl > ssl._create_https_context = ssl._create_unverified_context > > The backport to 2.7 would then be a matter of bringing urllib, > urllib2, httplib and ssl into line with their 3.4.2 counterparts. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 9 22:31:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Sep 2014 06:31:58 +1000 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: <540F26CC.4010608@python.org> References: <540de614.075b8c0a.5db6.5396@mx.google.com> <540F26CC.4010608@python.org> Message-ID: On 10 Sep 2014 02:11, "Christian Heimes" wrote: > > On 09.09.2014 05:03, Nick Coghlan wrote: > > > > On 9 Sep 2014 10:48, "Jim J. Jewett" > > wrote: > >> I assume that adding _unverified_urlopen or urlopen(context=...) do > >> provide incremental improvements compatible with the eventual full > >> opt-in. If so, adding them is probably reasonable, but I think the > >> PEP should explicitly list all such approved half-measures as a guard > >> against API feature creep. > > > > From Guido's and your feedback, I think we may need two things to > > approve this for 3.4.2 (putting 2.7 aside for now): > > > > 1. "context" parameter support in urllib.request (to opt out on a > > per-call basis) > > 2. a documented way to restore the old behaviour via sitecustomize > > (which may involve monkeypatching) > > What's with our plan to introduce sslcustomize? Is the idea for a > configuration module and named contexts off the table? I'd still like to do that for 3.5+ as the proper long term fix, but I don't think it's necessary for 2.7 or 3.4. Cheers, Nick. > > > For reference: > > I came up with the idea to introduce the module "sslcustomize" similar > to sitecustomize. Contrary to sitecustomize the module is imported at > the end of the ssl module. > > Based on my idea Nick proposed another addition to the SSL module. He > proposed a ssl.named_contexts mapping from module names to factory > functions that create SSLContext objects. > http://permalink.gmane.org/gmane.comp.python.devel/149292 > > I still prefer the general idea over the monkey patching idea because it > provides a clean but simple interface for structured configuration. > Monkey patching of stdlib modules is ugly and error-prone. > > Christian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Tue Sep 9 23:13:34 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Tue, 9 Sep 2014 17:13:34 -0400 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: <540F26CC.4010608@python.org> References: <540de614.075b8c0a.5db6.5396@mx.google.com> <540F26CC.4010608@python.org> Message-ID: On Tue, Sep 9, 2014 at 12:11 PM, Christian Heimes wrote: > On 09.09.2014 05:03, Nick Coghlan wrote: >> >> On 9 Sep 2014 10:48, "Jim J. Jewett" > > wrote: >> From Guido's and your feedback, I think we may need two things to >> approve this for 3.4.2 (putting 2.7 aside for now): >> 1. "context" parameter support in urllib.request (to opt out on a >> per-call basis) >> 2. a documented way to restore the old behaviour via sitecustomize >> (which may involve monkeypatching) > What's with our plan to introduce sslcustomize? Is the idea for a > configuration module and named contexts off the table? In a perfect world, half-measures would not be needed, and so neither would sslcustomize. In the real world, half-measures are needed, but offering too many of them adds so much confusion that things can actually get worse in practice. In other words, sslcustomize could be great, but getting it wrong would be a step backwards -- so start it as a 3rd party module. Since the biggest users are likely supported customers of downstream distributions, it makes sense to let them take the lead, though I'm sure they would appreciate a proof of concept. > I still prefer the general idea over the monkey patching idea because it > provides a clean but simple interface for structured configuration. > Monkey patching of stdlib modules is ugly and error-prone. The primary use case for monkey patching is to support Separation of Roles. (Exact titles will of course differ by business.) If you need structured configuration, then you are already treating some calls differently from others, which means that you are already doing partial remediation. I agree that monkey patching is the wrong choice if you are doing partial remediation. But this partial remediation also implies that a "Developer" and "Business Owner" are involved to decide which calls need to be changed, and whether to change the call vs dropping the functionality vs convincing the owner of the other end of the connection to do things right in the first place. A Developer in charge of her own environment doesn't need to monkey patch -- but she could just do the right thing today, or switch to a framework that does. sslcustomize may be a really good way for her to document "these are the strange exceptions in our existing environment", if it is done right. A "Deployment Engineer" may not even know python, and is certainly not authorized to make changes beyond "configuration". Convincing someone that a .py file is a configuration knob probably requires an exception that is painful to get. (And saying "oh, this is just where we list security stuff that we're ignoring" won't make it easier.) Changing the the urlopen calls would therefore be unacceptable even if source code were available -- and sometimes it isn't. The Deployment Engineer is often responsible for upgrading the infrastructure components (possibly including python) for security patches, so he has to be able to deploy 3.4.x or 2.7.y (though *probably* not 3.5.0) without any changes to the application itself -- and usually without access to whatever regression tests the application itself uses. (Ideally, someone else who does have that access is involved, but ... not always.) What the Deployment Engineer *can* do is modify the environment around the application. He can write a shell script that sets environment variables and or command line options. He can probably even add a required component -- which might in practice just be a pre-written module like sslcustomize, or a .pth file that does the monkey patch on each launch. But *adding* such a component is orders of magnitude simpler (from a bureaucratic perspective) than *modifying* one that already exists. The end user often can't do anything outside the application's own UI, which is why the change has to be handled once at deployment, instead of as break-fix per call site or per bad certificate. -jJ From ncoghlan at gmail.com Wed Sep 10 06:26:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Sep 2014 14:26:27 +1000 Subject: [Python-Dev] Backwards compatibility after certificate autovalidation In-Reply-To: References: <540de614.075b8c0a.5db6.5396@mx.google.com> <540F26CC.4010608@python.org> Message-ID: On 10 September 2014 07:13, Jim J. Jewett wrote: > On Tue, Sep 9, 2014 at 12:11 PM, Christian Heimes wrote: >> On 09.09.2014 05:03, Nick Coghlan wrote: >>> >>> On 9 Sep 2014 10:48, "Jim J. Jewett" >> > wrote: >>> From Guido's and your feedback, I think we may need two things to >>> approve this for 3.4.2 (putting 2.7 aside for now): > >>> 1. "context" parameter support in urllib.request (to opt out on a >>> per-call basis) >>> 2. a documented way to restore the old behaviour via sitecustomize >>> (which may involve monkeypatching) > >> What's with our plan to introduce sslcustomize? Is the idea for a >> configuration module and named contexts off the table? > > In a perfect world, half-measures would not be needed, and so neither > would sslcustomize. > > In the real world, half-measures are needed, but offering too many of > them adds so much confusion that things can actually get worse in > practice. > > In other words, sslcustomize could be great, but getting it wrong > would be a step backwards -- so start it as a 3rd party module. Since > the biggest users are likely supported customers of downstream > distributions, it makes sense to let them take the lead, though I'm > sure they would appreciate a proof of concept. I see the use case for sslcustomize as being different: it wouldn't be about turning the stdlib SSL security settings *down*, it would be about selectively turning them *up*. As PEP 476 says, turning cert validation on for protocols like SMTP is currently going to cause significantly more problems than turning it on for HTTPS. The cleaner, more comprehensive sslcustomize model would provide a way to opt-in on a virtualenv and installation wide basis to those higher security settings. The fact it would also offer a slightly cleaner alternative to the HTTPS downgrade monkeypatch would be an incidental side effect. The change would touch many more modules than the current proposal though, which is why I now believe it's better tackled as a 3.5+ only feature. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 10 09:17:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Sep 2014 17:17:57 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog Message-ID: Since it may come in handy when discussing "Why was Python 3 necessary?" with folks, I wanted to point out that my article on the transition to multilingual programming has now been reposted on the Red Hat developer blog: http://developerblog.redhat.com/2014/09/09/transition-to-multilingual-programming-python/ I wouldn't normally bring the Red Hat brand into an upstream discussion like that, but this myth that Python 3 is killing the language, and that Python 2 could have continued as a viable development platform indefinitely "if only Guido and the core development team hadn't decided to go ahead and create Python 3", is just plain wrong, and it really needs to die. I'm hoping that borrowing a bit of Red Hat's enterprise credibility will finally get people to understand that we really do have some idea what we're doing, which is why most of our redistributors and many of our key users are helping to push the migration forward, while we also continue to support existing Python 2 users :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Sep 10 12:08:56 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 10 Sep 2014 20:08:56 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: Message-ID: <20140910100856.GC9293@ando.pearwood.info> On Wed, Sep 10, 2014 at 05:17:57PM +1000, Nick Coghlan wrote: > Since it may come in handy when discussing "Why was Python 3 > necessary?" with folks, I wanted to point out that my article on the > transition to multilingual programming has now been reposted on the > Red Hat developer blog: > http://developerblog.redhat.com/2014/09/09/transition-to-multilingual-programming-python/ That's awesome! Thank you Nick. -- Steven From larry at hastings.org Wed Sep 10 22:34:17 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 10 Sep 2014 13:34:17 -0700 Subject: [Python-Dev] Proposed schedule for 3.4.2 In-Reply-To: References: <540C521C.7070802@hastings.org> Message-ID: <5410B5C9.5040209@hastings.org> On 09/08/2014 05:08 AM, Nick Coghlan wrote: > On 8 September 2014 14:28, Ned Deily wrote: >> As I've already discussed with Larry, I think adding a week to the >> scheduled dates would be preferable. The original dates give pretty >> short notice and there are a number of open issues that would be good to >> resolve now in 3.4.2. > It would also be good to get Guido's official verdict on PEP 476 (the > switch to validating HTTPS by default) in time for 3.4.2. Based on the > previous discussion, Alex updated the PEP to suggest "just fix it" for > all of 3.5, 3.4.x and 2.7.x (including the httplib SSLContext support > backport in the latter case). > > I think that would be feasible with an rc on the 20th, but challenging > if the rc is this coming weekend. Okay, the thread has died down. It seems like pushing back a week will possibly help. So the new dates: 3.4.2 rc 1: September 22, 2014 3.4.2 final: October 6, 2014 //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jazeltq at 163.com Thu Sep 11 04:55:40 2014 From: jazeltq at 163.com (Li Tianqing) Date: Thu, 11 Sep 2014 10:55:40 +0800 (CST) Subject: [Python-Dev] recursive closure Message-ID: <45b3bb0.545e.14862a34501.Coremail.jazeltq@163.com> Hello, Can someone explain me why gc(CPython) can not collect recursive closure's cycle reference? There is no __del__ here, why gc can not collect? Thanks a lot. For example: #!/usr/bin/env python import ipdb import gc gc.set_debug(gc.DEBUG_LEAK) def A(): N = [1] def aa(n): if n in N: return 1 else: return n * aa(n-1) x = 33 + aa(10) #ipdb.set_trace() print x if __name__ == '__main__': while xrange(1000): A() -- Best Li Tianqing -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Sep 11 05:38:10 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Sep 2014 13:38:10 +1000 Subject: [Python-Dev] recursive closure In-Reply-To: <45b3bb0.545e.14862a34501.Coremail.jazeltq@163.com> References: <45b3bb0.545e.14862a34501.Coremail.jazeltq@163.com> Message-ID: On Thu, Sep 11, 2014 at 12:55 PM, Li Tianqing wrote: > Hello, > Can someone explain me why gc(CPython) can not collect recursive > closure's cycle reference? There is no __del__ here, why gc can not > collect? Can you start by explaining what's not getting collected and what is? Where's the cycle you're looking at? Also, this thread would do better on python-list at python.org rather than python-dev. ChrisA From farowl at blueyonder.co.uk Thu Sep 11 09:25:30 2014 From: farowl at blueyonder.co.uk (Jeff Allen) Date: Thu, 11 Sep 2014 08:25:30 +0100 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: Message-ID: <54114E6A.1070506@blueyonder.co.uk> A welcome article. One correction should be made, I believe: the area of code point space used for the smuggling of bytes under PEP-383 is not a "Unicode Private Use Area", but a portion of the trailing surrogate range. This is a code violation, which I imagine is why "surrogateescape" is an error handler, not a codec. http://www.unicode.org/faq/private_use.html I believe the private use area was considered and rejected for PEP-383. In an implementation of the type unicode based on UTF-16 (Jython), lone surrogates preclude a naive use of the platform string library. This is on my mind at the moment as I'm working several bugs in Jython's unicode type, and can see why it has been too difficult. Jeff On 10/09/2014 08:17, Nick Coghlan wrote: > Since it may come in handy when discussing "Why was Python 3 > necessary?" with folks, I wanted to point out that my article on the > transition to multilingual programming has now been reposted on the > Red Hat developer blog: > http://developerblog.redhat.com/2014/09/09/transition-to-multilingual-programming-python/ > > I wouldn't normally bring the Red Hat brand into an upstream > discussion like that, but this myth that Python 3 is killing the > language, and that Python 2 could have continued as a viable > development platform indefinitely "if only Guido and the core > development team hadn't decided to go ahead and create Python 3", is > just plain wrong, and it really needs to die. > > I'm hoping that borrowing a bit of Red Hat's enterprise credibility > will finally get people to understand that we really do have some idea > what we're doing, which is why most of our redistributors and many of > our key users are helping to push the migration forward, while we also > continue to support existing Python 2 users :) > > Cheers, > Nick. > From solipsis at pitrou.net Thu Sep 11 20:36:29 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 11 Sep 2014 20:36:29 +0200 Subject: [Python-Dev] Proposed schedule for 3.4.2 References: <540C521C.7070802@hastings.org> Message-ID: <20140911203629.5f091a9f@fsol> On Mon, 8 Sep 2014 10:44:51 -0700 Alex Gaynor wrote: > *Shifts uncomfortably* it looks like presently there's not a good way to > change anything about the SSL configuration for urllib.request.urlopen. It > does not take a `context` argument, as the http.client API does: > https://docs.python.org/3/library/urllib.request.html#module-urllib.request > and instead takes the cafile, capath, cadefault args. The reason it doesn't take a context argument is that SSL contexts are hard to understand and configure for the average urlopen() user (why should people know about ssl module specifics to open an HTTPS URL?). http.client is more low-level and therefore more amenable to such powerful knobs. Regards Antoine. From solipsis at pitrou.net Thu Sep 11 20:39:50 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 11 Sep 2014 20:39:50 +0200 Subject: [Python-Dev] Proposed schedule for 3.4.2 References: <540C521C.7070802@hastings.org> <20140908135717.34c781a1@anarchist.wooz.org> Message-ID: <20140911203950.3590bf36@fsol> On Tue, 9 Sep 2014 08:20:52 +1000 Nick Coghlan wrote: > On 9 Sep 2014 04:00, "Barry Warsaw" wrote: > > > > > >This would need to be updated first, once it *did* take such an argument, > > >this would be accomplished by: > > > > > >context = ssl.create_default_context() > > >context.verify_mode = CERT_OPTIONACERT_NONE > > >context.verify_hostname = False > > >urllib.request.urlopen(" > https://something-i-apparently-dont-care-much-about", > > >context=context) > > > > There's probably an ugly hack possibility that uses unittest.mock.patch. > ;) > > We could actually make it an "official" hack: > > import urllib.request > urllib.request.urlopen = urllib.request._unverified_urlopen -1. Instead of disabling cert verification for *one* urlopen() call site, you're doing it for *every* urlopen call site in the program - or, even worse, for every urlopen that's imported after the monkey-patching (which makes the final effect potentially dependent on module import order, and import style). It may affect third-party libraries that have their own REST calls, or whatever. Regards Antoine. From turnbull at sk.tsukuba.ac.jp Fri Sep 12 05:28:54 2014 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 12 Sep 2014 12:28:54 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <54114E6A.1070506@blueyonder.co.uk> References: <54114E6A.1070506@blueyonder.co.uk> Message-ID: <877g195xft.fsf@uwakimon.sk.tsukuba.ac.jp> Jeff Allen writes: > A welcome article. One correction should be made, I believe: the area of > code point space used for the smuggling of bytes under PEP-383 is not a > "Unicode Private Use Area", but a portion of the trailing surrogate > range. Nice catch. Note that the surrogate range was originally part of the Private Use Area, but it was carved out with the adoption of UTF-16 in about 1993. In practice, I doubt that there are any current implementations claiming compatibility with Unicode 1.0 (IIRC, UTF-16 was made mandatory in Unicode 1.1). > This is a code violation, which I imagine is why > "surrogateescape" is an error handler, not a codec. Yes. > I believe the private use area was considered and rejected for PEP-383. > In an implementation of the type unicode based on UTF-16 (Jython), lone > surrogates preclude a naive use of the platform string library. This is > on my mind at the moment as I'm working several bugs in Jython's unicode > type, and can see why it has been too difficult. I've always thought that the "right" way to handle the private use area for "platforms" like Python and Emacs, which may need to use it for their own purposes (such as "undecodable bytes") but want to respect its use by applications, is to create an auxiliary table mapping the private use area to objects describing the characters represented by the private use code points. These objects would have attributes such as external representation for text I/O, glyph (for GUI display), repr (for TTY display), various Unicode properties, etc. From ja.py at farowl.co.uk Fri Sep 12 08:54:56 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Fri, 12 Sep 2014 07:54:56 +0100 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <877g195xft.fsf@uwakimon.sk.tsukuba.ac.jp> References: <54114E6A.1070506@blueyonder.co.uk> <877g195xft.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <541298C0.6040302@farowl.co.uk> On 12/09/2014 04:28, Stephen J. Turnbull wrote: > Jeff Allen writes: > > > A welcome article. One correction should be made, I believe: the area of > > code point space used for the smuggling of bytes under PEP-383 is not a > > "Unicode Private Use Area", but a portion of the trailing surrogate > > range. > > Nice catch. Note that the surrogate range was originally part of the > Private Use Area, but it was carved out with the adoption of UTF-16 in > about 1993. In practice, I doubt that there are any current > implementations claiming compatibility with Unicode 1.0 (IIRC, UTF-16 > was made mandatory in Unicode 1.1). That's a helpful bit of history that explains the uncharacteristic inaccuracy. Most I can do to keep the current position clear in my head. > I've always thought that the "right" way to handle the private use > area for "platforms" like Python and Emacs, which may need to use it > for their own purposes (such as "undecodable bytes") but want to > respect its use by applications, is to create an auxiliary table > mapping the private use area to objects describing the characters > represented by the private use code points. These objects would have > attributes such as external representation for text I/O, glyph (for > GUI display), repr (for TTY display), various Unicode properties, etc. Simply having a block "for private use" seems to create an unmanaged space for conflict, reminiscent of the "other 128 characters" in bilingual programming. I wondered if the way to respect use by applications might be to make it private to a particular sub-class of str, idly however. Jeff Allen From solipsis at pitrou.net Fri Sep 12 10:37:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 12 Sep 2014 10:37:26 +0200 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog References: <54114E6A.1070506@blueyonder.co.uk> <877g195xft.fsf@uwakimon.sk.tsukuba.ac.jp> <541298C0.6040302@farowl.co.uk> Message-ID: <20140912103726.73d40ffe@fsol> On Fri, 12 Sep 2014 07:54:56 +0100 Jeff Allen wrote: > Simply having a block "for private use" seems to create an unmanaged > space for conflict, reminiscent of the "other 128 characters" in > bilingual programming. I wondered if the way to respect use by > applications might be to make it private to a particular sub-class of > str, idly however. It's not private from Python's point of view, it's actually specified in a PEP. So all Python 3 code has to follow the rule, and there's no conflict internally. The characters shouldn't leak out to other applications, unless the user's code does its I/O very badly :-) Regards Antoine. From jimjjewett at gmail.com Fri Sep 12 17:37:59 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 12 Sep 2014 08:37:59 -0700 (PDT) Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <54114E6A.1070506@blueyonder.co.uk> Message-ID: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> On September 11, 2014, Jeff Allen wrote: > ... the area of code point > space used for the smuggling of bytes under PEP-383 is not a > "Unicode Private Use Area", but a portion of the trailing surrogate > range. This is a code violation, which I imagine is why > "surrogateescape" is an error handler, not a codec. True, but I believe that is a CPython implementation detail. Other implementations (including jython) should implement the surrogatescape API, but I don't think it is important to use the same internal representation for the invalid bytes. (Well, unless you want to communicate with external tools (GUIs?) that are trying to directly use (effectively bytes rather than strings) in that particular internal encoding when communicating with python.) > lone surrogates preclude a naive use of the platform string library Invalid input often causes problems. Are you saying that there are situations where the platform string library could easily handle invalid characters in general, but has a problem with the specific case of lone surrogates? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From status at bugs.python.org Fri Sep 12 18:08:10 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 12 Sep 2014 18:08:10 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140912160810.3968556348@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-09-05 - 2014-09-12) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4652 (+12) closed 29509 (+38) total 34161 (+50) Open issues with patches: 2196 Issues opened (39) ================== #16662: load_tests not invoked in package/__init__.py http://bugs.python.org/issue16662 reopened by haypo #22343: Install bash activate script on Windows when using venv http://bugs.python.org/issue22343 opened by marfire #22344: Reorganize unittest.mock docs into linear manner http://bugs.python.org/issue22344 opened by py.user #22347: mimetypes.guess_type("//example.com") misinterprets host name http://bugs.python.org/issue22347 opened by vadmium #22348: Documentation of asyncio.StreamWriter.drain() http://bugs.python.org/issue22348 opened by martius #22350: nntplib file write failure causes exception from QUIT command http://bugs.python.org/issue22350 opened by vadmium #22351: NNTP constructor exception leaves socket for garbage collector http://bugs.python.org/issue22351 opened by vadmium #22352: Ensure opcode names and args fit in disassembly output http://bugs.python.org/issue22352 opened by ncoghlan #22354: Highlite tabs in the IDLE http://bugs.python.org/issue22354 opened by Christian.Kleineidam #22355: inconsistent results with inspect.getsource() / inspect.getsou http://bugs.python.org/issue22355 opened by isedev #22356: mention explicitly that stdlib assumes gmtime(0) epoch is 1970 http://bugs.python.org/issue22356 opened by akira #22357: inspect module documentation makes no reference to __qualname_ http://bugs.python.org/issue22357 opened by isedev #22359: Remove incorrect uses of recursive make http://bugs.python.org/issue22359 opened by Sjlver #22360: Adding manually offset parameter to str/bytes split function http://bugs.python.org/issue22360 opened by cwr #22361: Ability to join() threads in concurrent.futures.ThreadPoolExec http://bugs.python.org/issue22361 opened by dktrkranz #22362: Warn about octal escapes > 0o377 in re http://bugs.python.org/issue22362 opened by serhiy.storchaka #22363: argparse AssertionError with add_mutually_exclusive_group and http://bugs.python.org/issue22363 opened by Zacrath #22364: Unify error messages of re and regex http://bugs.python.org/issue22364 opened by serhiy.storchaka #22365: SSLContext.load_verify_locations(cadata) does not accept CRLs http://bugs.python.org/issue22365 opened by Ralph.Broenink #22366: urllib.request.urlopen shoudl take a "context" (SSLContext) ar http://bugs.python.org/issue22366 opened by alex #22367: Please add F_OFD_SETLK, etc support to fcntl.lockf http://bugs.python.org/issue22367 opened by Andrew.Lutomirski #22370: pathlib OS detection http://bugs.python.org/issue22370 opened by Antony.Lee #22371: tests failing with -uall and http_proxy and https_proxy set http://bugs.python.org/issue22371 opened by doko #22374: Replace contextmanager example and improve explanation http://bugs.python.org/issue22374 opened by terry.reedy #22376: urllib2.urlopen().read().splitlines() opening a directory in a http://bugs.python.org/issue22376 opened by alanoe #22377: %Z in strptime doesn't match EST and others http://bugs.python.org/issue22377 opened by cool-RR #22378: SO_MARK support for Linux http://bugs.python.org/issue22378 opened by jpv #22379: Empty exception message of str.join http://bugs.python.org/issue22379 opened by fossilet #22382: sqlite3 connection built from apsw connection should raise Int http://bugs.python.org/issue22382 opened by wtonkin #22384: Tk.report_callback_exception kills process when run with pytho http://bugs.python.org/issue22384 opened by Aivar.Annamaa #22385: Allow 'x' and 'X' to accept bytes-like objects in string forma http://bugs.python.org/issue22385 opened by ncoghlan #22387: Making tempfile.NamedTemporaryFile a class http://bugs.python.org/issue22387 opened by Antony.Lee #22388: Unify style of "Contributed by" notes http://bugs.python.org/issue22388 opened by serhiy.storchaka #22389: Generalize contextlib.redirect_stdout http://bugs.python.org/issue22389 opened by barry #22390: test.regrtest should complain if a test doesn't remove tempora http://bugs.python.org/issue22390 opened by haypo #22391: MSILIB truncates last character in summary information stream http://bugs.python.org/issue22391 opened by Kevin.Phillips #22392: Clarify documentation of __getinitargs__ http://bugs.python.org/issue22392 opened by David.Gilman #22393: multiprocessing.Pool shouldn't hang forever if a worker proces http://bugs.python.org/issue22393 opened by dan.oreilly #22394: Update documentation building to use venv and pip http://bugs.python.org/issue22394 opened by brett.cannon Most recent 15 issues with no replies (15) ========================================== #22394: Update documentation building to use venv and pip http://bugs.python.org/issue22394 #22393: multiprocessing.Pool shouldn't hang forever if a worker proces http://bugs.python.org/issue22393 #22392: Clarify documentation of __getinitargs__ http://bugs.python.org/issue22392 #22388: Unify style of "Contributed by" notes http://bugs.python.org/issue22388 #22378: SO_MARK support for Linux http://bugs.python.org/issue22378 #22370: pathlib OS detection http://bugs.python.org/issue22370 #22367: Please add F_OFD_SETLK, etc support to fcntl.lockf http://bugs.python.org/issue22367 #22357: inspect module documentation makes no reference to __qualname_ http://bugs.python.org/issue22357 #22356: mention explicitly that stdlib assumes gmtime(0) epoch is 1970 http://bugs.python.org/issue22356 #22354: Highlite tabs in the IDLE http://bugs.python.org/issue22354 #22352: Ensure opcode names and args fit in disassembly output http://bugs.python.org/issue22352 #22351: NNTP constructor exception leaves socket for garbage collector http://bugs.python.org/issue22351 #22350: nntplib file write failure causes exception from QUIT command http://bugs.python.org/issue22350 #22347: mimetypes.guess_type("//example.com") misinterprets host name http://bugs.python.org/issue22347 #22344: Reorganize unittest.mock docs into linear manner http://bugs.python.org/issue22344 Most recent 15 issues waiting for review (15) ============================================= #22393: multiprocessing.Pool shouldn't hang forever if a worker proces http://bugs.python.org/issue22393 #22390: test.regrtest should complain if a test doesn't remove tempora http://bugs.python.org/issue22390 #22389: Generalize contextlib.redirect_stdout http://bugs.python.org/issue22389 #22388: Unify style of "Contributed by" notes http://bugs.python.org/issue22388 #22384: Tk.report_callback_exception kills process when run with pytho http://bugs.python.org/issue22384 #22379: Empty exception message of str.join http://bugs.python.org/issue22379 #22378: SO_MARK support for Linux http://bugs.python.org/issue22378 #22366: urllib.request.urlopen shoudl take a "context" (SSLContext) ar http://bugs.python.org/issue22366 #22363: argparse AssertionError with add_mutually_exclusive_group and http://bugs.python.org/issue22363 #22362: Warn about octal escapes > 0o377 in re http://bugs.python.org/issue22362 #22359: Remove incorrect uses of recursive make http://bugs.python.org/issue22359 #22356: mention explicitly that stdlib assumes gmtime(0) epoch is 1970 http://bugs.python.org/issue22356 #22348: Documentation of asyncio.StreamWriter.drain() http://bugs.python.org/issue22348 #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 #22330: PyOS_mystricmp is broken http://bugs.python.org/issue22330 Top 10 most discussed issues (10) ================================= #22299: resolve() on Windows makes some pathological paths unusable http://bugs.python.org/issue22299 19 msgs #22302: Windows os.path.isabs UNC path bug http://bugs.python.org/issue22302 12 msgs #9951: introduce bytes.hex method http://bugs.python.org/issue9951 11 msgs #16662: load_tests not invoked in package/__init__.py http://bugs.python.org/issue16662 11 msgs #22166: test_codecs leaks references http://bugs.python.org/issue22166 10 msgs #21228: Missing enumeration of HTTPResponse Objects methods of urllib. http://bugs.python.org/issue21228 9 msgs #7559: TestLoader.loadTestsFromName swallows import errors http://bugs.python.org/issue7559 7 msgs #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 7 msgs #22240: argparse support for "python -m module" in help http://bugs.python.org/issue22240 6 msgs #22360: Adding manually offset parameter to str/bytes split function http://bugs.python.org/issue22360 6 msgs Issues closed (37) ================== #4972: context management support in imaplib, smtplib, ftplib http://bugs.python.org/issue4972 closed by serhiy.storchaka #8315: ./python -m unittest test.test_importlib doesn't work http://bugs.python.org/issue8315 closed by berker.peksag #11487: build_installer.py should avoid relying on a young Python http://bugs.python.org/issue11487 closed by ned.deily #13968: Support recursive globs http://bugs.python.org/issue13968 closed by serhiy.storchaka #15594: test_copyfile_named_pipe() fails on Mac OS X Snow Leopard: OSE http://bugs.python.org/issue15594 closed by ned.deily #16104: Compileall script: add option to use multiple cores http://bugs.python.org/issue16104 closed by brett.cannon #19232: Speed up _decimal import http://bugs.python.org/issue19232 closed by skrah #19524: ResourceWarning when urlopen() forgets the HTTPConnection obje http://bugs.python.org/issue19524 closed by serhiy.storchaka #20417: ensurepip should not be installed with --without-ensurepip http://bugs.python.org/issue20417 closed by ncoghlan #20643: Strange dot in documentation (after generator.close) http://bugs.python.org/issue20643 closed by serhiy.storchaka #21122: CPython fails to build modules with LLVM LTO on Mac OS X http://bugs.python.org/issue21122 closed by Sjlver #21147: sqlite3 doesn't complain if the request contains a null charac http://bugs.python.org/issue21147 closed by serhiy.storchaka #21951: tcl test change crashes AIX http://bugs.python.org/issue21951 closed by haypo #22131: uuid.bytes optimization http://bugs.python.org/issue22131 closed by serhiy.storchaka #22192: dict_values objects are hashable http://bugs.python.org/issue22192 closed by roippi #22212: zipfile.py fails if zlib.so module fails to build. http://bugs.python.org/issue22212 closed by ned.deily #22215: "embedded NUL character" exceptions http://bugs.python.org/issue22215 closed by serhiy.storchaka #22226: Refactor dict result handling in Tkinter http://bugs.python.org/issue22226 closed by serhiy.storchaka #22284: decimal module contains less symbols when the _decimal module http://bugs.python.org/issue22284 closed by skrah #22285: The Modules/ directory should not be added to sys.path http://bugs.python.org/issue22285 closed by ned.deily #22295: Clarify available commands for package installation http://bugs.python.org/issue22295 closed by ncoghlan #22336: _tkinter should use Python PyMem_Malloc() instead of Tcl ckall http://bugs.python.org/issue22336 closed by python-dev #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 closed by haypo #22340: Fix Python 3 warnings in Python 2 tests http://bugs.python.org/issue22340 closed by haypo #22342: Fix typo in PEP 380 -- Syntax for Delegating to a Subgenerator http://bugs.python.org/issue22342 closed by python-dev #22345: https://docs.python.org/release/1.4/ returns 403 http://bugs.python.org/issue22345 closed by benjamin.peterson #22346: asyncio documentation does not mention provisional status http://bugs.python.org/issue22346 closed by gvanrossum #22349: Remove more unnecessary version checks from distutils http://bugs.python.org/issue22349 closed by python-dev #22353: re.findall() documentation lacks information about finding THE http://bugs.python.org/issue22353 closed by gvanrossum #22358: Unnecessary JUMP_FORWARD(0) (NOP) in if statements without els http://bugs.python.org/issue22358 closed by serhiy.storchaka #22369: "context management protocol" vs "context manager protocol" http://bugs.python.org/issue22369 closed by serhiy.storchaka #22373: PyArray_FromAny tries to deallocate double: 12 (d) http://bugs.python.org/issue22373 closed by benjamin.peterson #22375: urllib2.urlopen().read().splitlines() opening a directory in a http://bugs.python.org/issue22375 closed by alanoe #22380: Y2K compliance section in FAQ is 14 years too old http://bugs.python.org/issue22380 closed by python-dev #22381: update zlib in 2.7 to 1.2.8 http://bugs.python.org/issue22381 closed by doko #22383: Crazy unicode : How g and ?? look the same but are two differe http://bugs.python.org/issue22383 closed by ezio.melotti #22386: Python 3.4 logging.getLevelName() no longer maps string to lev http://bugs.python.org/issue22386 closed by python-dev From stephen at xemacs.org Fri Sep 12 19:24:15 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 13 Sep 2014 02:24:15 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <541298C0.6040302@farowl.co.uk> References: <54114E6A.1070506@blueyonder.co.uk> <877g195xft.fsf@uwakimon.sk.tsukuba.ac.jp> <541298C0.6040302@farowl.co.uk> Message-ID: <874mwc69c0.fsf@uwakimon.sk.tsukuba.ac.jp> Jeff Allen writes: > Simply having a block "for private use" seems to create an unmanaged > space for conflict, No. The uncharted range of human language (including recently- invented nonsense like "emoticons" and the annual "design a character" contest run by a newpaper in Taipei, with the grand prize being your character gets added to the national standard IIRC, but maybe it's just that newspaper's collection of private space characters) already contains those conflicts. Believe me, "private use space, manage it yourself" was the best they could do. I've been working with the beureaucratic insanity of the Japanese national standard -- it took almost 3 decades before every Japanese citizen could store their names in a computer using government- approved codes -- and the chaos of the Taiwanese national standard -- which contains hordes of characters with one known use and no known meaning, many of them duplicates -- for twenty years now. Neither approach works as well as Unicode's, despite its design-by-committee flaws overlaid with national animosities that can flare into linguicidal vetoes and code-space-stuffing logrolling. > reminiscent of the "other 128 characters" in bilingual > programming. I wondered if the way to respect use by applications > might be to make it private to a particular sub-class of str, idly > however. If I understand your suggestion, that's precisely the intent of PEP 383, to make undecodable bytes in a coded character stream private. But they need to be in the stream one way or another. So PEP 383 chose to use a non-Unicode encoding (based on the "lone surrogate" device invented by Markus Kuhn for utf-8b) to deal with that, and that does effectively make those elements private to Python (but of course not in the Unicode sense, as they're not even characters in Unicode). But I gather the "native" Unicode type in Java doesn't allow you to use that dodge because it checks for malformed Unicode internally (ie, at a level not controllable by Jython). So you have to embed such stream elements in the space of Unicode characters. You have the option of the private space or unallocated (reserved) space. The latter seems like asking for trouble, and the only way to avoid it would be to be prepared to move that data around in case of collision. But that's precisely what I'm suggesting doing in private space. Same issue, either way. Private space with a local registry seems saner. From ja.py at farowl.co.uk Sat Sep 13 00:16:30 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Fri, 12 Sep 2014 23:16:30 +0100 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> References: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> Message-ID: <541370BE.8030700@farowl.co.uk> Jim, Stephen: It seems like we're off topic here, but to answer all as briefly as possible: 1. Java does not really have a Unicode type, therefore not one that validates. It has a String type that is a sequence of UTF-16 code units. There are some String methods and Character methods that deal with code points represented as int. I can put any 16-bit values I like in a String. 2. With proper accounting for indices, and as long as surrogates appear in pairs, I believe operations like find or endswith give correct answers about the unicode, when applied to the UTF-16. This is an attractive implementation option, and mostly what we do. 3. I'm fixing some bugs where we get it wrong beyond the BMP, and the fix involves banning lone surrogates (completely). At present you can't type them in literals but you can sneak them in from Java. 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it would have to do it the same way as CPython, as it is visible. It's not impossible (I think), but is messy. Some are strongly against. Jeff Allen On 12/09/2014 16:37, Jim J. Jewett wrote: > > > On September 11, 2014, Jeff Allen wrote: > >> ... "surrogateescape" is an error handler, not a codec. > True, but I believe that is a CPython implementation detail. > > Other implementations (including jython) should implement the > surrogatescape API, but I don't think it is important to use the > same internal representation for the invalid bytes. > >> lone surrogates preclude a naive use of the platform string library > Invalid input often causes problems. Are you saying that there are > situations where the platform string library could easily handle > invalid characters in general, but has a problem with the specific > case of lone surrogates? > From benjamin at python.org Sat Sep 13 02:34:48 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 12 Sep 2014 20:34:48 -0400 Subject: [Python-Dev] new hg.python.org server Message-ID: <1410568488.3502994.166945269.219D0D6F@webmail.messagingengine.com> I just switched hg.python.org from a OSUOSL VM to a Rackspace VM. The new VM is a bit beefier and has what I think is better network connectivity, so hopefully that will improving the speed of repository operations. We also now support HTTPS for repository browsing and cloning, so update all your links to https://hg.python.org! IPv6 support has also returned for those who like that sort of thing. Note the host keys changed, so you'll probably have to futz with known_hosts to quiet ssh down. I apologize, but I noticed that that the current RSA host key is 1024 bits, so I decided to upgrade it to 2048 during the transition. Thanks to Donald Stufft for helping me set this up. From raymond.hettinger at gmail.com Sat Sep 13 03:02:42 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 12 Sep 2014 18:02:42 -0700 Subject: [Python-Dev] [python-committers] new hg.python.org server In-Reply-To: <1410568488.3502994.166945269.219D0D6F@webmail.messagingengine.com> References: <1410568488.3502994.166945269.219D0D6F@webmail.messagingengine.com> Message-ID: <020FC186-AD45-46D0-AFF4-5E84FF70CCD9@gmail.com> On Sep 12, 2014, at 5:34 PM, Benjamin Peterson wrote: > The > new VM is a bit beefier and has what I think is better network > connectivity, so hopefully that will improving the speed of repository > operations. Thanks Benjamin, the repo is noticeably faster. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajshorya at gmail.com Sat Sep 13 03:52:02 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Sat, 13 Sep 2014 13:52:02 +1200 Subject: [Python-Dev] [python-committers] new hg.python.org server In-Reply-To: <020FC186-AD45-46D0-AFF4-5E84FF70CCD9@gmail.com> References: <1410568488.3502994.166945269.219D0D6F@webmail.messagingengine.com> <020FC186-AD45-46D0-AFF4-5E84FF70CCD9@gmail.com> Message-ID: Just wondering - are there any sys-adminy sort of tasks that could be completed? I mean, I have some (note, some) experience doing this, and I wouldn't mind helping out (I inquired in the buildbot thread as well, but there wasn't much of a response). Thanks Shorya Raj On Sat, Sep 13, 2014 at 1:02 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > On Sep 12, 2014, at 5:34 PM, Benjamin Peterson > wrote: > > The > new VM is a bit beefier and has what I think is better network > connectivity, so hopefully that will improving the speed of repository > operations. > > > Thanks Benjamin, the repo is noticeably faster. > > > Raymond > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Sep 13 04:15:33 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 12 Sep 2014 22:15:33 -0400 Subject: [Python-Dev] [python-committers] new hg.python.org server In-Reply-To: References: <1410568488.3502994.166945269.219D0D6F@webmail.messagingengine.com> <020FC186-AD45-46D0-AFF4-5E84FF70CCD9@gmail.com> Message-ID: <1410574533.3522004.166962429.2D7D7CC6@webmail.messagingengine.com> On Fri, Sep 12, 2014, at 21:52, Shorya Raj wrote: > Just wondering - are there any sys-adminy sort of tasks that could be > completed? I mean, I have some (note, some) experience doing this, and I > wouldn't mind helping out (I inquired in the buildbot thread as well, but > there wasn't much of a response). Well, hg.python.org is basically done now. The main thing now is understanding how other services (planet.python.org, bugs.python.org) are setup and moving them to config management. From ncoghlan at gmail.com Sat Sep 13 11:06:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Sep 2014 21:06:21 +1200 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <541370BE.8030700@farowl.co.uk> References: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> <541370BE.8030700@farowl.co.uk> Message-ID: On 13 Sep 2014 10:18, "Jeff Allen" wrote: > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it would have to do it the same way as CPython, as it is visible. It's not impossible (I think), but is messy. Some are strongly against. It may be worth trying *without* it (i.e. treat "surrogateescape" as equivalent to "strict" initially), and seeing how you go. The main purpose of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data round trips work on POSIX" aspect of CPython 2, which doesn't apply in exactly the same way on Jython. Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2, "surrogateescape is equivalent to strict" seems like a relatively small discrepancy in behaviour. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sat Sep 13 15:31:50 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 13 Sep 2014 09:31:50 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> <541370BE.8030700@farowl.co.uk> Message-ID: <20140913133151.098A3250E0B@webabinitio.net> On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan wrote: > On 13 Sep 2014 10:18, "Jeff Allen" wrote: > > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it > would have to do it the same way as CPython, as it is visible. It's not > impossible (I think), but is messy. Some are strongly against. > > It may be worth trying *without* it (i.e. treat "surrogateescape" as > equivalent to "strict" initially), and seeing how you go. The main purpose > of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data > round trips work on POSIX" aspect of CPython 2, which doesn't apply in > exactly the same way on Jython. > > Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2, > "surrogateescape is equivalent to strict" seems like a relatively small > discrepancy in behaviour. That would totally break the email package. It would of course be possible to rewrite email to not use surrogate escape, but it is a seriously non-trivial undertaking. --David From tlesher at gmail.com Sat Sep 13 17:29:42 2014 From: tlesher at gmail.com (Tim Lesher) Date: Sat, 13 Sep 2014 15:29:42 +0000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog References: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> <541370BE.8030700@farowl.co.uk> <20140913133151.098A3250E0B@webabinitio.net> Message-ID: On Sat, Sep 13, 2014, 09:33 R. David Murray wrote: > On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan > wrote: > > On 13 Sep 2014 10:18, "Jeff Allen" wrote: > > > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, > it > > would have to do it the same way as CPython, as it is visible. It's not > > impossible (I think), but is messy. Some are strongly against. > > > > It may be worth trying *without* it (i.e. treat "surrogateescape" as > > equivalent to "strict" initially), and seeing how you go. The main > purpose > > of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data > > round trips work on POSIX" aspect of CPython 2, which doesn't apply in > > exactly the same way on Jython. > > > > Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2, > > "surrogateescape is equivalent to strict" seems like a relatively small > > discrepancy in behaviour. > > That would totally break the email package. > > It would of course be possible to rewrite email to not use surrogate > escape, but it is a seriously non-trivial undertaking. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > tlesher%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Sep 13 21:40:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Sep 2014 07:40:04 +1200 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140913133151.098A3250E0B@webabinitio.net> References: <54131357.67258c0a.58d3.ffff8a9a@mx.google.com> <541370BE.8030700@farowl.co.uk> <20140913133151.098A3250E0B@webabinitio.net> Message-ID: On 14 Sep 2014 01:33, "R. David Murray" wrote: > > On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan wrote: > > On 13 Sep 2014 10:18, "Jeff Allen" wrote: > > > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it > > would have to do it the same way as CPython, as it is visible. It's not > > impossible (I think), but is messy. Some are strongly against. > > > > It may be worth trying *without* it (i.e. treat "surrogateescape" as > > equivalent to "strict" initially), and seeing how you go. The main purpose > > of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data > > round trips work on POSIX" aspect of CPython 2, which doesn't apply in > > exactly the same way on Jython. > > > > Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2, > > "surrogateescape is equivalent to strict" seems like a relatively small > > discrepancy in behaviour. > > That would totally break the email package. > > It would of course be possible to rewrite email to not use surrogate > escape, but it is a seriously non-trivial undertaking. That does indeed make for a compelling use case :) Cheers, Nick. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From binni at binnisb.com Sun Sep 14 12:13:03 2014 From: binni at binnisb.com (=?UTF-8?Q?Brynjar_Sm=C3=A1ri_Bjarnason?=) Date: Sun, 14 Sep 2014 12:13:03 +0200 Subject: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1 Message-ID: Hi. I am using Python 3.4.1 installed with Anaconda. I tried the following (expecting an OrderedDict as result): >>>from collections import namedtuple >>>NT = namedtuple("NT",["a","b"]) >>>nt = NT(1,2) >>>print(vars(nt)) {} so the result is an empty dict. In Python 3.3.2 (downgraded in the same Anaconda environment) results in: >>>print(vars(nt)) OrderedDict([('a', 1), ('b', 2)]) I haven't looked at the source code, I can do that later in the week and see if I can pinpoint where this is happening. But any suggestions what and where to fix this would be good :) Cheers, Brynjar From rosuav at gmail.com Sun Sep 14 18:13:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Sep 2014 02:13:53 +1000 Subject: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1 In-Reply-To: References: Message-ID: On Sun, Sep 14, 2014 at 8:13 PM, Brynjar Sm?ri Bjarnason wrote: > I am using Python 3.4.1 installed with Anaconda. I tried the following > (expecting an OrderedDict as result): > >>>>from collections import namedtuple >>>>NT = namedtuple("NT",["a","b"]) >>>>nt = NT(1,2) >>>>print(vars(nt)) > {} > > so the result is an empty dict. In Python 3.3.2 (downgraded in the > same Anaconda environment) results in: > >>>>print(vars(nt)) > OrderedDict([('a', 1), ('b', 2)]) For what it's worth, I can't reproduce the issue on trunk CPython (built from default branch on Aug 21, so it's a little old now), nor on 3.4.1 as currently shipping with Debian Jessie, nor with 3.4.0 on Windows. So this may be an Anaconda issue. Do you know if it's meant to be a patched install of Python? ChrisA From solipsis at pitrou.net Sun Sep 14 18:27:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 14 Sep 2014 18:27:35 +0200 Subject: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1 References: Message-ID: <20140914182735.258b9362@fsol> On Mon, 15 Sep 2014 02:13:53 +1000 Chris Angelico wrote: > On Sun, Sep 14, 2014 at 8:13 PM, Brynjar Sm?ri Bjarnason > wrote: > > I am using Python 3.4.1 installed with Anaconda. I tried the following > > (expecting an OrderedDict as result): > > > >>>>from collections import namedtuple > >>>>NT = namedtuple("NT",["a","b"]) > >>>>nt = NT(1,2) > >>>>print(vars(nt)) > > {} > > > > so the result is an empty dict. In Python 3.3.2 (downgraded in the > > same Anaconda environment) results in: > > > >>>>print(vars(nt)) > > OrderedDict([('a', 1), ('b', 2)]) > > For what it's worth, I can't reproduce the issue on trunk CPython > (built from default branch on Aug 21, so it's a little old now), nor > on 3.4.1 as currently shipping with Debian Jessie, nor with 3.4.0 on > Windows. So this may be an Anaconda issue. Do you know if it's meant > to be a patched install of Python? There may be a couple distutils-specific patches, but that's all. Anaconda issues should be reported at https://github.com/ContinuumIO/anaconda-issues/issues/ Regards Antoine. From technews.full at gmail.com Mon Sep 15 05:18:21 2014 From: technews.full at gmail.com (Harish Tech) Date: Mon, 15 Sep 2014 08:48:21 +0530 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append Message-ID: I had a list a = [1, 2, 3] when I did a.insert(100, 100) [1, 2, 3, 100] as list was originally of size 4 and I was trying to insert value at index 100 , it behaved like append instead of throwing any errors as I was trying to insert in an index that did not even existed . Should it not throw IndexError: list assignment index out of range exception as it throws when I attempt doing a[100] = 100 Question : 1. Any idea Why has it been designed to silently handle this instead of informing the user with an exception ? Personal Opinion : Lets see how other dynamic languages behave in such a situation : Ruby : > a = [1, 2] > a[100] = 100 > a => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] The way ruby handles this is pretty clear and sounds meaningful (and this is how I expected to behave and it behaved as per my expectation) at least to me . So what I felt was either it should throw exception or do the way ruby handles it . Is ruby way of handling not the obvious way ? I even raised it in stackoverflow http://stackoverflow.com/questions/25840177/list-insert-at-index-that-is-well-out-of-range-behaves-like-append and got some responses . -------------- next part -------------- An HTML attachment was scrubbed... URL: From taleinat at gmail.com Mon Sep 15 13:31:22 2014 From: taleinat at gmail.com (Tal Einat) Date: Mon, 15 Sep 2014 14:31:22 +0300 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: References: Message-ID: On Mon, Sep 15, 2014 at 6:18 AM, Harish Tech wrote: > I had a list > > a = [1, 2, 3] > > when I did > > a.insert(100, 100) > > [1, 2, 3, 100] > > as list was originally of size 4 and I was trying to insert value at index > 100 , it behaved like append instead of throwing any errors as I was trying > to insert in an index that did not even existed . > > > Should it not throw > > > IndexError: list assignment index out of range > > > exception as it throws when I attempt doing > > > a[100] = 100 > > Question : 1. Any idea Why has it been designed to silently handle this > instead of informing the user with an exception ? > > > Personal Opinion : Lets see how other dynamic languages behave in such a > situation : Ruby : > > > > a = [1, 2] > > > a[100] = 100 > > > a > > => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] > > The way ruby handles this is pretty clear and sounds meaningful (and this is > how I expected to behave and it behaved as per my expectation) at least to > me . So what I felt was either it should throw exception or do the way ruby > handles it . > > > Is ruby way of handling not the obvious way ? > > I even raised it in stackoverflow > http://stackoverflow.com/questions/25840177/list-insert-at-index-that-is-well-out-of-range-behaves-like-append > > and got some responses . Hello Harish, The appropriate place to ask questions like this is python-list [1], or perhaps Stack Overflow. If you meant to suggest changing the behavior of Python in such cases, you should first discuss this on python-list, and then post a clearly written suggestion to python-ideas [2]. This list, python-dev, is used for discussing the development *of* the Python language. See the "Python Mailing Lists" page [3] for more information. Regards, - Tal Einat ..[1]: http://mail.python.org/mailman/listinfo/python-list ..[2]: http://mail.python.org/mailman/listinfo/python-ideas ..[3]: https://www.python.org/community/lists/ From technews.full at gmail.com Mon Sep 15 14:17:33 2014 From: technews.full at gmail.com (Harish Tech) Date: Mon, 15 Sep 2014 17:47:33 +0530 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: References: Message-ID: Hi , Sorry for that mistake . Now I have posted it in python-list mailing list . Thanks for your guidance. Harish On Mon, Sep 15, 2014 at 5:01 PM, Tal Einat wrote: > On Mon, Sep 15, 2014 at 6:18 AM, Harish Tech > wrote: > > I had a list > > > > a = [1, 2, 3] > > > > when I did > > > > a.insert(100, 100) > > > > [1, 2, 3, 100] > > > > as list was originally of size 4 and I was trying to insert value at > index > > 100 , it behaved like append instead of throwing any errors as I was > trying > > to insert in an index that did not even existed . > > > > > > Should it not throw > > > > > > IndexError: list assignment index out of range > > > > > > exception as it throws when I attempt doing > > > > > > a[100] = 100 > > > > Question : 1. Any idea Why has it been designed to silently handle this > > instead of informing the user with an exception ? > > > > > > Personal Opinion : Lets see how other dynamic languages behave in such a > > situation : Ruby : > > > > > > > a = [1, 2] > > > > > a[100] = 100 > > > > > a > > > > => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, > nil, > > nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] > > > > The way ruby handles this is pretty clear and sounds meaningful (and > this is > > how I expected to behave and it behaved as per my expectation) at least > to > > me . So what I felt was either it should throw exception or do the way > ruby > > handles it . > > > > > > Is ruby way of handling not the obvious way ? > > > > I even raised it in stackoverflow > > > http://stackoverflow.com/questions/25840177/list-insert-at-index-that-is-well-out-of-range-behaves-like-append > > > > and got some responses . > > Hello Harish, > > The appropriate place to ask questions like this is python-list [1], > or perhaps Stack Overflow. > > If you meant to suggest changing the behavior of Python in such cases, > you should first discuss this on python-list, and then post a clearly > written suggestion to python-ideas [2]. > > This list, python-dev, is used for discussing the development *of* the > Python language. > > See the "Python Mailing Lists" page [3] for more information. > > Regards, > - Tal Einat > > ..[1]: http://mail.python.org/mailman/listinfo/python-list > ..[2]: http://mail.python.org/mailman/listinfo/python-ideas > ..[3]: https://www.python.org/community/lists/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Mon Sep 15 19:52:52 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Mon, 15 Sep 2014 10:52:52 -0700 Subject: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1 In-Reply-To: <20140914182735.258b9362@fsol> References: <20140914182735.258b9362@fsol> Message-ID: FWIW I cannot reproduce the bug with Anaconda's Python 3.4.1 (from a miniconda install): $ python Python 3.4.1 |Continuum Analytics, Inc.| (default, Sep 2 2014, 14:00:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from collections import namedtuple; vars(namedtuple("T", "a b")(1, 2)) OrderedDict([('a', 1), ('b', 2)]) Antony 2014-09-14 9:27 GMT-07:00 Antoine Pitrou : > On Mon, 15 Sep 2014 02:13:53 +1000 > Chris Angelico wrote: > > On Sun, Sep 14, 2014 at 8:13 PM, Brynjar Sm?ri Bjarnason > > wrote: > > > I am using Python 3.4.1 installed with Anaconda. I tried the following > > > (expecting an OrderedDict as result): > > > > > >>>>from collections import namedtuple > > >>>>NT = namedtuple("NT",["a","b"]) > > >>>>nt = NT(1,2) > > >>>>print(vars(nt)) > > > {} > > > > > > so the result is an empty dict. In Python 3.3.2 (downgraded in the > > > same Anaconda environment) results in: > > > > > >>>>print(vars(nt)) > > > OrderedDict([('a', 1), ('b', 2)]) > > > > For what it's worth, I can't reproduce the issue on trunk CPython > > (built from default branch on Aug 21, so it's a little old now), nor > > on 3.4.1 as currently shipping with Debian Jessie, nor with 3.4.0 on > > Windows. So this may be an Anaconda issue. Do you know if it's meant > > to be a patched install of Python? > > There may be a couple distutils-specific patches, but that's all. > Anaconda issues should be reported at > https://github.com/ContinuumIO/anaconda-issues/issues/ > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/antony.lee%40berkeley.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Mon Sep 15 20:35:01 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 15 Sep 2014 11:35:01 -0700 (PDT) Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <541370BE.8030700@farowl.co.uk> Message-ID: <54173155.087fe00a.7669.4f8e@mx.google.com> On Sat Sep 13 00:16:30 CEST 2014, Jeff Allen wrote: > 1. Java does not really have a Unicode type, therefore not one that > validates. It has a String type that is a sequence of UTF-16 code units. > There are some String methods and Character methods that deal with code > points represented as int. I can put any 16-bit values I like in a String. Including lone surrogates, and invalid characters in general? > 2. With proper accounting for indices, and as long as surrogates appear > in pairs, I believe operations like find or endswith give correct > answers about the unicode, when applied to the UTF-16. This is an > attractive implementation option, and mostly what we do. So use it. The fact that you're having to smuggle bytes already guarantees that your data is either invalid or misinterpreted, and bug-free isn't possible. In terms of best-effort, it is reasonable to treat the smuggled bytes as representing a character outside of your unicode repertoire -- so it won't ever match entirely valid strings, except perhaps via a wildcard. And it should still work for .endswith(). > 3. I'm fixing some bugs where we get it wrong beyond the BMP, and the > fix involves banning lone surrogates (completely). At present you can't > type them in literals but you can sneak them in from Java. So how will you ban them, and what will you do when some java class sends you an invalid sequence anyhow? That is exactly the use case for these smuggled bytes... If you distinguish between a fully constructed PyString and a code-unit-sequence-that-could-be-made-into-a-PyString-later, then you could always have your constructor return an InvalidPyString subclass on the rare occasions when one is needed. If you want to avoid invalid surrogates even then, just use the replacement character and keep a separate list of "original characters that got replaced in this string" -- a hassle, but no worse than tracking indices for surrogates. > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it > would have to do it the same way as CPython, as it is visible. It's not > impossible (I think), but is messy. Some are strongly against. If you allow direct write access to the underlying charsequence (as CPython does to C extensions), then you can't really ban invalid sequences. If callers have to go through an API -- even something as minimal as getBytes or getChars -- then you can use whatever internal representation you prefer. Hopefully, the vast majority of strings won't actually have smuggled bytes. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From mark at hotpy.org Tue Sep 16 00:29:34 2014 From: mark at hotpy.org (Mark Shannon) Date: Mon, 15 Sep 2014 23:29:34 +0100 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: References: Message-ID: <5417684E.5020002@hotpy.org> On 15/09/14 12:31, Tal Einat wrote: > On Mon, Sep 15, 2014 at 6:18 AM, Harish Tech wrote: >> I had a list >> >> a = [1, 2, 3] >> >> when I did >> >> a.insert(100, 100) >> >> [1, 2, 3, 100] >> >> as list was originally of size 4 and I was trying to insert value at index >> 100 , it behaved like append instead of throwing any errors as I was trying >> to insert in an index that did not even existed . >> >> >> Should it not throw >> >> >> IndexError: list assignment index out of range >> >> >> exception as it throws when I attempt doing >> >> >> a[100] = 100 >> >> Question : 1. Any idea Why has it been designed to silently handle this >> instead of informing the user with an exception ? >> >> >> Personal Opinion : Lets see how other dynamic languages behave in such a >> situation : Ruby : >> >> >> > a = [1, 2] >> >> > a[100] = 100 >> >> > a >> >> => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] >> >> The way ruby handles this is pretty clear and sounds meaningful (and this is >> how I expected to behave and it behaved as per my expectation) at least to >> me . So what I felt was either it should throw exception or do the way ruby >> handles it . >> >> >> Is ruby way of handling not the obvious way ? >> >> I even raised it in stackoverflow >> http://stackoverflow.com/questions/25840177/list-insert-at-index-that-is-well-out-of-range-behaves-like-append >> >> and got some responses . > > Hello Harish, > > The appropriate place to ask questions like this is python-list [1], > or perhaps Stack Overflow. I think this is an OK forum for this question. If someone isn't sure if something is a bug or not, then why not ask here before reporting it on the bug tracker? This does seem strange behaviour, and the documentation for list.insert gives no clue as to why this behaviour was chosen. Cheers, Mark. From breamoreboy at yahoo.co.uk Tue Sep 16 00:46:03 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 15 Sep 2014 23:46:03 +0100 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: <5417684E.5020002@hotpy.org> References: <5417684E.5020002@hotpy.org> Message-ID: On 15/09/2014 23:29, Mark Shannon wrote: > > > On 15/09/14 12:31, Tal Einat wrote: >> On Mon, Sep 15, 2014 at 6:18 AM, Harish Tech >> wrote: >>> I had a list >>> >>> a = [1, 2, 3] >>> >>> when I did >>> >>> a.insert(100, 100) >>> >>> [1, 2, 3, 100] >>> >>> as list was originally of size 4 and I was trying to insert value at >>> index >>> 100 , it behaved like append instead of throwing any errors as I was >>> trying >>> to insert in an index that did not even existed . >>> >>> >>> Should it not throw >>> >>> >>> IndexError: list assignment index out of range >>> >>> >>> exception as it throws when I attempt doing >>> >>> >>> a[100] = 100 >>> >>> Question : 1. Any idea Why has it been designed to silently handle this >>> instead of informing the user with an exception ? >>> >>> >>> Personal Opinion : Lets see how other dynamic languages behave in such a >>> situation : Ruby : >>> >>> >>> > a = [1, 2] >>> >>> > a[100] = 100 >>> >>> > a >>> >>> => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] >>> >>> The way ruby handles this is pretty clear and sounds meaningful (and >>> this is >>> how I expected to behave and it behaved as per my expectation) at >>> least to >>> me . So what I felt was either it should throw exception or do the >>> way ruby >>> handles it . >>> >>> >>> Is ruby way of handling not the obvious way ? >>> >>> I even raised it in stackoverflow >>> http://stackoverflow.com/questions/25840177/list-insert-at-index-that-is-well-out-of-range-behaves-like-append >>> >>> >>> and got some responses . >> >> Hello Harish, >> >> The appropriate place to ask questions like this is python-list [1], >> or perhaps Stack Overflow. > > I think this is an OK forum for this question. > If someone isn't sure if something is a bug or not, then why not ask > here before reporting it on the bug tracker? > > This does seem strange behaviour, and the documentation for list.insert > gives no clue as to why this behaviour was chosen. > > Cheers, > Mark. I assume it's based on the concepts of slicing. From the docs "s.insert(i, x) - inserts x into s at the index given by i (same as s[i:i] = [x])". Although shouldn't that read s[i:i+1] = [x] ? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From ethan at stoneleaf.us Tue Sep 16 00:53:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 15 Sep 2014 15:53:41 -0700 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: References: <5417684E.5020002@hotpy.org> Message-ID: <54176DF5.1030304@stoneleaf.us> On 09/15/2014 03:46 PM, Mark Lawrence wrote: > On 15/09/2014 23:29, Mark Shannon wrote: >> >> I think this is an OK forum for this question. It isn't. ;) >> If someone isn't sure if something is a bug or not, then why not ask >> here before reporting it on the bug tracker? The first stop should still be the main Python list, or Python Dev would be inundated with questions about why this or that doesn't work the same way as . If the responses from Python list indicate that it is (or probably is) a bug, then possibly a post here to verify -- but a bug-tracker entry at that point is quite reasonable. >> This does seem strange behaviour, and the documentation for list.insert >> gives no clue as to why this behaviour was chosen. > > I assume it's based on the concepts of slicing. From the docs "s.insert(i, x) - inserts x into s at the index given by > i (same as s[i:i] = [x])". Although shouldn't that read s[i:i+1] = [x] ? No. If it was `s[i:i+1]` then the ith element would be replaced by the inserted object. -- ~Ethan~ From solipsis at pitrou.net Tue Sep 16 00:54:03 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Sep 2014 00:54:03 +0200 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append References: <5417684E.5020002@hotpy.org> Message-ID: <20140916005403.697352df@fsol> On Mon, 15 Sep 2014 23:46:03 +0100 Mark Lawrence wrote: > > I assume it's based on the concepts of slicing. From the docs > "s.insert(i, x) - inserts x into s at the index given by i (same as > s[i:i] = [x])". Although shouldn't that read s[i:i+1] = [x] ? No, the latter would replace the contents at index i, while the former inserts it (formally, it replaces the 0-length slice with a 1-length slice). Regards Antoine. From guido at python.org Tue Sep 16 00:54:57 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Sep 2014 15:54:57 -0700 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: <5417684E.5020002@hotpy.org> References: <5417684E.5020002@hotpy.org> Message-ID: This functionality has existed since the earliest days of Python, and even if we all agreed it was wrong we couldn't change it -- it would just break too much existing code. I can't quite remember why I did it that way but it was definitely a conscious choice; probably some symmetry or edge case. (Note that it works this way at the other end too -- a.insert(-100, x) will insert x at the beginning of a, if a has fewer than 100 elements.) On Mon, Sep 15, 2014 at 3:29 PM, Mark Shannon wrote: > > > On 15/09/14 12:31, Tal Einat wrote: > >> On Mon, Sep 15, 2014 at 6:18 AM, Harish Tech >> wrote: >> >>> I had a list >>> >>> a = [1, 2, 3] >>> >>> when I did >>> >>> a.insert(100, 100) >>> >>> [1, 2, 3, 100] >>> >>> as list was originally of size 4 and I was trying to insert value at >>> index >>> 100 , it behaved like append instead of throwing any errors as I was >>> trying >>> to insert in an index that did not even existed . >>> >>> >>> Should it not throw >>> >>> >>> IndexError: list assignment index out of range >>> >>> >>> exception as it throws when I attempt doing >>> >>> >>> a[100] = 100 >>> >>> Question : 1. Any idea Why has it been designed to silently handle this >>> instead of informing the user with an exception ? >>> >>> >>> Personal Opinion : Lets see how other dynamic languages behave in such a >>> situation : Ruby : >>> >>> >>> > a = [1, 2] >>> >>> > a[100] = 100 >>> >>> > a >>> >>> => [1, 2, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, >>> nil, >>> nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 100] >>> >>> The way ruby handles this is pretty clear and sounds meaningful (and >>> this is >>> how I expected to behave and it behaved as per my expectation) at least >>> to >>> me . So what I felt was either it should throw exception or do the way >>> ruby >>> handles it . >>> >>> >>> Is ruby way of handling not the obvious way ? >>> >>> I even raised it in stackoverflow >>> http://stackoverflow.com/questions/25840177/list- >>> insert-at-index-that-is-well-out-of-range-behaves-like-append >>> >>> and got some responses . >>> >> >> Hello Harish, >> >> The appropriate place to ask questions like this is python-list [1], >> or perhaps Stack Overflow. >> > > I think this is an OK forum for this question. > If someone isn't sure if something is a bug or not, then why not ask here > before reporting it on the bug tracker? > > This does seem strange behaviour, and the documentation for list.insert > gives no clue as to why this behaviour was chosen. > > Cheers, > Mark. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 16 00:57:17 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Sep 2014 15:57:17 -0700 Subject: [Python-Dev] List insert at index that is well out of range - behaves like append In-Reply-To: References: <5417684E.5020002@hotpy.org> Message-ID: On Mon, Sep 15, 2014 at 3:46 PM, Mark Lawrence wrote: > > I assume it's based on the concepts of slicing. From the docs > "s.insert(i, x) - inserts x into s at the index given by i (same as s[i:i] > = [x])". Ah, right. It matches thigs like s[100:] which is the empty string if s is shorter than 100. > Although shouldn't that read s[i:i+1] = [x] ? > Should've stopped while you were ahead. :-) 'Nuff said. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Sep 16 05:34:36 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Sep 2014 12:34:36 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <54173155.087fe00a.7669.4f8e@mx.google.com> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> Message-ID: <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> Jim J. Jewett writes: > In terms of best-effort, it is reasonable to treat the smuggled bytes > as representing a character outside of your unicode repertoire I have to disagree. If you ever end up passing them to something that validates or tries to reencode them without surrogateescape, BOOM! These things are the text equivalent of IEEE NaNs. If all you know (as in the stdlib) is that you have "generic text", the only fairly safe things to do with them are (1) delete them, (2) substitute an appropriate replacement character for them, (3) pass the text containing them verbatim to other code, and (4) reencode them using the same codec they were read with. > -- so it won't ever match entirely valid strings, except perhaps > via a wildcard. And it should still work for .endswith( invalid characters>). Incorrect, I'm pretty sure, unless you know that both texts containing were read with the same codec. Eg, consider two filenames encoded in ISO Cyrillic and ISO Hebrew, read with (encoding='ascii', errors='surrogateescape'). Apps that know the semantics of the text may DWIM/DTRT if they want to, but FWIW-IMHO-YMMV-and-any-other-4-letter-caveat-acronyms-that- may-apply Python and the stdlib shouldn't try to guess. Guessing may be unavoidable, of course. From rosuav at gmail.com Tue Sep 16 05:51:23 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Sep 2014 13:51:23 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull wrote: > Jim J. Jewett writes: > > > In terms of best-effort, it is reasonable to treat the smuggled bytes > > as representing a character outside of your unicode repertoire > > I have to disagree. If you ever end up passing them to something that > validates or tries to reencode them without surrogateescape, BOOM! > These things are the text equivalent of IEEE NaNs. If all you know > (as in the stdlib) is that you have "generic text", the only fairly > safe things to do with them are (1) delete them, (2) substitute an > appropriate replacement character for them, (3) pass the text > containing them verbatim to other code, and (4) reencode them using > the same codec they were read with. Don't forget, these are *errors*. These are bytes that cannot be correctly decoded. That's not something that has any meaning whatsoever in text; so by definition, the only things you can do are the four you list there (as long as "codec" means both the choice of encoding and the use of the surrogateescape flag). It's like dealing with control characters when you need to print something visually, except that they have an official solution [1] and surrogateescape is unofficial. They're not real text, so you have to deal with them somehow. The bytes might each represent one character. Several of them together might represent a single character. Or maybe they don't mean anything at all, and they're just part of a chunked data format... like I was finding in the .cwk files that I was reading this weekend (it's mostly MacRoman encoding, but the text is divided into chunks separated by \0\0 and two more bytes - turns out the bytes are chunk lengths, so they don't mean any sort of characters at all). You can't know. ChrisA [1] http://www.unicode.org/charts/PDF/U2400.pdf From rdmurray at bitdance.com Tue Sep 16 17:00:32 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 16 Sep 2014 11:00:32 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140916150033.3733D250E2B@webabinitio.net> On Tue, 16 Sep 2014 13:51:23 +1000, Chris Angelico wrote: > On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull wrote: > > Jim J. Jewett writes: > > > > > In terms of best-effort, it is reasonable to treat the smuggled bytes > > > as representing a character outside of your unicode repertoire > > > > I have to disagree. If you ever end up passing them to something that > > validates or tries to reencode them without surrogateescape, BOOM! > > These things are the text equivalent of IEEE NaNs. If all you know > > (as in the stdlib) is that you have "generic text", the only fairly > > safe things to do with them are (1) delete them, (2) substitute an > > appropriate replacement character for them, (3) pass the text > > containing them verbatim to other code, and (4) reencode them using > > the same codec they were read with. > > Don't forget, these are *errors*. These are bytes that cannot be > correctly decoded. That's not something that has any meaning > whatsoever in text; so by definition, the only things you can do are > the four you list there (as long as "codec" means both the choice of > encoding and the use of the surrogateescape flag). It's like dealing > with control characters when you need to print something visually, > except that they have an official solution [1] and surrogateescape is > unofficial. They're not real text, so you have to deal with them > somehow. That isn't the case in the email package. The smuggled bytes are not errors[*], they are literally smuggled bytes. But, as Stephen said, the only things email does with them are the last three of the four he listed (if you read (3) as passing it between parts of the email package): the data comes in as text mixed with binary, and the email package parses it until it knows what the binary is supposed to be, turns it back into bytes, and decodes it properly. The goal is to never let the smuggled bytes escape out the email APIs as surrogateescape encoded text; though, in practice, this being consenting-adults Python and code not being bug free, there are places where people have used the knowledge of how surrogateescape is used by email to work around both API and code bugs. --David [*] Some of the encoded bytes *are* errors (non-ascii in headers or undecodable bytes in whatever the CTE/charset is), and in that case email may just turn them back into error bytes in the output, but only *some* of the smuggled bytes are actually errors (and none are if the message is RFC compliant). From rosuav at gmail.com Tue Sep 16 17:27:44 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 17 Sep 2014 01:27:44 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140916150033.3733D250E2B@webabinitio.net> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> Message-ID: On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray wrote: > That isn't the case in the email package. The smuggled bytes are not > errors[*], they are literally smuggled bytes. But they're not characters, which is what Stephen and I were saying - and contrary to what Jim said about treating them as characters. At best, they represent characters but in some encoding other than the one you're using, and you have no idea how many bytes form a character or anything. So you can't, for instance, word-wrap the text, because you can't know how wide these unknown bytes are, whether they represent spaces (wrap points), or newlines, or anything like that. You can't treat them as characters, so while you have them in your string, you can't treat it as a pure Unicode string - it''s a Unicode string with smuggled bytes. ChrisA From rdmurray at bitdance.com Tue Sep 16 19:46:50 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 16 Sep 2014 13:46:50 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> Message-ID: <20140916174650.79DDF250E49@webabinitio.net> On Wed, 17 Sep 2014 01:27:44 +1000, Chris Angelico wrote: > On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray wrote: > > That isn't the case in the email package. The smuggled bytes are not > > errors[*], they are literally smuggled bytes. > > But they're not characters, which is what Stephen and I were saying - > and contrary to what Jim said about treating them as characters. At > best, they represent characters but in some encoding other than the > one you're using, and you have no idea how many bytes form a character > or anything. So you can't, for instance, word-wrap the text, because > you can't know how wide these unknown bytes are, whether they > represent spaces (wrap points), or newlines, or anything like that. > You can't treat them as characters, so while you have them in your > string, you can't treat it as a pure Unicode string - it''s a Unicode > string with smuggled bytes. Well, except that I do. The email header parsing algorithms all work fine if I treat the surrogate escaped bytes as 'unknown junk' and just parse based on the valid unicode. (Unless the header is so garbled that it can't be parsed, of course, at which point it becomes an invalid header). You are right about the wrapping, though. If a header with invalid bytes (and in this scenario we *are* talking about errors) needs to be wrapped, we have to first decode the smuggled bytes and turn it into an 'unknown-8bit' encoded word before we can wrap the header. --David From rosuav at gmail.com Tue Sep 16 20:02:11 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 17 Sep 2014 04:02:11 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140916174650.79DDF250E49@webabinitio.net> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> Message-ID: On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray wrote: >> You can't treat them as characters, so while you have them in your >> string, you can't treat it as a pure Unicode string - it''s a Unicode >> string with smuggled bytes. > > Well, except that I do. The email header parsing algorithms all work > fine if I treat the surrogate escaped bytes as 'unknown junk' and just > parse based on the valid unicode. (Unless the header is so garbled that > it can't be parsed, of course, at which point it becomes an invalid > header). Do what, exactly? As I understand you, you treat the unknown bytes as completely opaque, not representing any characters at all. Which is what I'm saying: those are not characters. If you, instead, represented the header as a list with some str elements and some bytes, it would be just as valid (though much harder to work with); all your manipulations are done on the str parts, and the bytes just tag along for the ride. > You are right about the wrapping, though. If a header with invalid > bytes (and in this scenario we *are* talking about errors) needs to > be wrapped, we have to first decode the smuggled bytes and turn it > into an 'unknown-8bit' encoded word before we can wrap the header. Yeah, and that's going to be a bit messy. If you get 60 characters followed by 30 unknown bytes, where do you wrap it? Dare you wrap in the middle of the smuggled section? ChrisA From jim.baker at python.org Tue Sep 16 19:55:31 2014 From: jim.baker at python.org (Jim Baker) Date: Tue, 16 Sep 2014 11:55:31 -0600 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> Message-ID: Great points here - I especially like the concluding statement "you can't treat it as a pure Unicode string - it's a Unicode string with smuggled bytes" Given that Jython uses UTF-16 as its representation, it is possible to frequently smuggle isolated surrogates in it. A surrogate pair must be a low surrogate in range (D800, DC00), then a high surrogate in range(DC00, E000). So one can likely assign an interpretation that this is in fact the isolated surrogate, and not an actual codepoint. Of course, if you do actually have a smuggled isolated low surrogate FOLLOWED by a smuggled isolated high surrogate - guess what, the only interpretation is a codepoint. Or perhaps more likely garbage. Of course it doesn't happen so often, so maybe we are fine with the occasional bug ;) I personally suspect that we will resolve this by also supporting UCS-4 as a representation in Jython 3.x for such Unicode strings, albeit with the limitation that we have simply moved the problem to when we try to call Java methods taking java.lang.String objects. - Jim On Tue, Sep 16, 2014 at 9:27 AM, Chris Angelico wrote: > On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray > wrote: > > That isn't the case in the email package. The smuggled bytes are not > > errors[*], they are literally smuggled bytes. > > But they're not characters, which is what Stephen and I were saying - > and contrary to what Jim said about treating them as characters. At > best, they represent characters but in some encoding other than the > one you're using, and you have no idea how many bytes form a character > or anything. So you can't, for instance, word-wrap the text, because > you can't know how wide these unknown bytes are, whether they > represent spaces (wrap points), or newlines, or anything like that. > You can't treat them as characters, so while you have them in your > string, you can't treat it as a pure Unicode string - it''s a Unicode > string with smuggled bytes. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com > -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 16 20:05:10 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 17 Sep 2014 04:05:10 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> Message-ID: On Wed, Sep 17, 2014 at 3:55 AM, Jim Baker wrote: > Of course, if you do actually have a smuggled isolated low surrogate > FOLLOWED by a smuggled isolated high surrogate - guess what, the only > interpretation is a codepoint. Or perhaps more likely garbage. Of course it > doesn't happen so often, so maybe we are fine with the occasional bug ;) > > I personally suspect that we will resolve this by also supporting UCS-4 as a > representation in Jython 3.x for such Unicode strings, albeit with the > limitation that we have simply moved the problem to when we try to call Java > methods taking java.lang.String objects. > That'll cost efficiency, of course, but it'll guarantee correctness. And maybe, just maybe, you'll be able to put some pressure on Java itself to start supporting UCS-4 natively... One can dream. ChrisA From rdmurray at bitdance.com Tue Sep 16 21:29:30 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 16 Sep 2014 15:29:30 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> Message-ID: <20140916192930.BB833250E42@webabinitio.net> On Wed, 17 Sep 2014 04:02:11 +1000, Chris Angelico wrote: > On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray wrote: > >> You can't treat them as characters, so while you have them in your > >> string, you can't treat it as a pure Unicode string - it''s a Unicode > >> string with smuggled bytes. > > > > Well, except that I do. The email header parsing algorithms all work > > fine if I treat the surrogate escaped bytes as 'unknown junk' and just > > parse based on the valid unicode. (Unless the header is so garbled that > > it can't be parsed, of course, at which point it becomes an invalid > > header). > > Do what, exactly? As I understand you, you treat the unknown bytes as > completely opaque, not representing any characters at all. Which is > what I'm saying: those are not characters. Yes. I thought you were saying that one could not treat the string with smuggled bytes as if it were a string. (It's a string that can't be encoded unless you use the surrogateescape error handler, but it is still a string from Python's POV, which is the point of the error handler). Or, to put it another way, your implication was that there were no string operations that could be usefully applied to a string containing smuggled bytes, but that is not the case. (I may well have read an implication that was not there; if so I apologize and you can ignore the rest of this :) Basically, we are pretending that the each smuggled byte is single character for string parsing purposes...but they don't match any of our parsing constants. They are all "any character" matches in the regexes and what have you. Of course, this only works in contexts where we can ignore or "carry along" the smuggled bytes as being components of "arbitrary text" portions of the syntax, and we must take care to either replace them with valid unicode error glyphs or turn the string of which the are a part into binary using the same codec and error handler as we used to ingest them to begin with before emitting them. And, of course, we can't *modify* the sections containing the smuggled bytes, only the syntax-matched sections that surround them; and things like line wrapping are just an invitation to ugliness and bugs even if you kept the smuggled bytes sections internally intact. Finally, to explain what I meant by "except that I do": when I added back binary support to the email package in Python3, initially I *did not change the parsing algorithms* in the code. I just smuggled the bytes, and then dealt with the encoding/decoding at the API boundaries. This is the same principle used when dealing with filenames in the API of Python itself. *Except* at that boundary, I do not need to worry about whether a particular string contains smuggled bytes or not.[*] > If you, instead, represented the header as a list with some str > elements and some bytes, it would be just as valid (though much harder > to work with); all your manipulations are done on the str parts, and > the bytes just tag along for the ride. Quite a bit harder, which is why I don't do that. > > You are right about the wrapping, though. If a header with invalid > > bytes (and in this scenario we *are* talking about errors) needs to > > be wrapped, we have to first decode the smuggled bytes and turn it > > into an 'unknown-8bit' encoded word before we can wrap the header. > > Yeah, and that's going to be a bit messy. If you get 60 characters > followed by 30 unknown bytes, where do you wrap it? Dare you wrap in > the middle of the smuggled section? The point of RFC2047 encoded words is that they are an ASCII representation of binary data, so once the bytes are "properly" Content Transfer Encoded (as being in an unknown charset) the string contains no smuggled bytes and can be wrapped. --David [*] I worried a lot that this was re-introducing the bytes/string problem from python2. The difference is that if the smuggled bytes escape from the email API, that's a bug in the email package. So user code using the library is *not* in danger of getting mysterious encoding errors when one day the input is international where before it was all ASCII. (Absent bugs in the library.) From stephen at xemacs.org Wed Sep 17 01:57:21 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Sep 2014 08:57:21 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> Message-ID: <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> Jim Baker writes: > Given that Jython uses UTF-16 as its representation, it is possible to > frequently smuggle isolated surrogates in it. A surrogate pair must be a > low surrogate in range (D800, DC00), then a high surrogate in range(DC00, > E000). > > Of course, if you do actually have a smuggled isolated low > surrogate FOLLOWED by a smuggled isolated high surrogate - guess > what, the only interpretation is a codepoint. Or perhaps more > likely garbage. Of course it doesn't happen so often, so maybe we > are fine with the occasional bug ;) The CPython representation uses trailing surrogates only[1], so it's never possible to interpret them as anything but non-characters -- as soon as you encounter them you know that it's a lone surrogate. Surely you can do the same. As long as the Java string manipulation functions don't check for surrogates, you should be fine with this representation. Of course I suppose your matching functions (etc) don't check for them either, so you will be somewhat vulnerable to bugs due to treating them as characters. But the same is true for CPython, AFAIK. Footnotes: [1] Only 128 bytes are necessary since the 128 ASCII characters are embedded in Unicode as-is. From stephen at xemacs.org Wed Sep 17 02:21:56 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Sep 2014 09:21:56 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140916192930.BB833250E42@webabinitio.net> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> Message-ID: <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > > Do what, exactly? As I understand you, you treat the unknown bytes as > > completely opaque, not representing any characters at all. Which is > > what I'm saying: those are not characters. > > Yes. I thought you were saying that one could not treat the string with > smuggled bytes as if it were a string. Guido's mantra is something like "Python's str doesn't contain characters or even code points[1], it contains code units." Implying that dealing with characters (or the grapheme globs that occasionally raise their ugly heads here) is an issue for higher-level facilities than str to deal with. The point being that > Basically, we are pretending that the each smuggled byte is single > character is something of a misstatement (good enough for present purpose of discussing email, but not good enough for the general case of understanding how this is supposed to work when porting the construct to other Python implementations), while > for string parsing purposes...but they don't match any of our > parsing constants. is precisely Pythonically correct. You might want to add "because all parsing constants contain only valid characters by construction." > [*] I worried a lot that this was re-introducing the bytes/string > problem from python2. It isn't, because the bytes/str problem was that given a str object out of context you could not tell whether it was a binary blob or text, and if text, you couldn't tell if it was external encoded text or internal abstract text. That is not true here because the representations of characters vs. smuggled bytes in str are disjoint sets. Footnotes: [1] In Unicode terminology, a code unit is the smallest computer object that can represent a character (this is uniquely and sanely defined for all real Unicode transformation formats aka UTFs). A code point is an integer 0 - (17*256*256-1) that can represent a character, but many code points such as surrogates and 0xFFFF are defined to be non-characters. Characters are those code points that may be assigned an interpretation as a character, including undefined characters (private space and reserved). From rdmurray at bitdance.com Wed Sep 17 02:25:21 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 16 Sep 2014 20:25:21 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140917002522.36C28250E42@webabinitio.net> On Wed, 17 Sep 2014 08:57:21 +0900, "Stephen J. Turnbull" wrote: > As long as the Java string manipulation functions don't check for > surrogates, you should be fine with this representation. Of course I > suppose your matching functions (etc) don't check for them either, so > you will be somewhat vulnerable to bugs due to treating them as > characters. But the same is true for CPython, AFAIK. >From my point of view, the string function laxness is a feature, not a bug. But I get what you mean. --David From rosuav at gmail.com Wed Sep 17 03:14:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 17 Sep 2014 11:14:15 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140916192930.BB833250E42@webabinitio.net> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> Message-ID: On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote: > Yes. I thought you were saying that one could not treat the string with > smuggled bytes as if it were a string. (It's a string that can't be > encoded unless you use the surrogateescape error handler, but it is > still a string from Python's POV, which is the point of the error > handler). > > Or, to put it another way, your implication was that there were no > string operations that could be usefully applied to a string containing > smuggled bytes, but that is not the case. (I may well have read an > implication that was not there; if so I apologize and you can ignore the > rest of this :) Ahh, I see where we are getting confused. What I said was that you can't treat the string as a *pure* Unicode string. Parts of it are Unicode text, parts of it aren't. > Basically, we are pretending that the each smuggled > byte is single character for string parsing purposes...but they don't > match any of our parsing constants. They are all "any character" matches > in the regexes and what have you. This is slightly iffy, as you can't be sure that one byte represents one character, but as long as you don't much care about that, it's not going to be an issue. I'm fairly sure you're never going to find an encoding in which one unknown byte represents two characters, but there are cases where it takes more than one byte to make up a character (or the bytes are just shift codes or something). Does that ever throw off your regexes? It wouldn't be an issue to a .* between two character markers, but if you ever say .{5} then it might match incorrectly. I think we're in agreement here, just using different words. :) ChrisA From v+python at g.nevcal.com Wed Sep 17 03:30:32 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 16 Sep 2014 18:30:32 -0700 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5418E438.3000901@g.nevcal.com> On 9/16/2014 5:21 PM, Stephen J. Turnbull wrote: > It isn't, because the bytes/str problem was that given a str object > out of context you could not tell whether it was a binary blob or > text, and if text, you couldn't tell if it was external encoded text > or internal abstract text. > > That is not true here because the representations of characters vs. > smuggled bytes in str are disjoint sets. Actually, while it may be true that for the email headers case, all characters are characters, just the encoding is unknown, it is not necessarily true that they are in disjoint sets. Some bytes may decode into characters without needing to be smuggled... maybe not in text-protocols like email, but in the general case. So then some of the bytes that should be interpreted as binary data are not in a disjoint set from characters. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Sep 17 05:28:57 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Sep 2014 12:28:57 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <5418E438.3000901@g.nevcal.com> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> <5418E438.3000901@g.nevcal.com> Message-ID: <8738br2adi.fsf@uwakimon.sk.tsukuba.ac.jp> Glenn Linderman writes: > Some bytes may decode into characters without needing to be > smuggled... maybe not in text-protocols like email, but in the > general case. So then some of the bytes that should be interpreted > as binary data are not in a disjoint set from characters. True, but irrelevant. The point is that whoever chose the codec is responsible for getting it right, not only the right encoding, but for the assumption that the input data was pure encoded text. The rest of the program can now assume that choice was made correctly, and process text as text. The program cannot be blamed for assuming that the person who chose the codec knew what they were about, and so characters can be *assumed* to be decoded from bytes representing characters. This was not true in Python 2, where it was common practice to represent encoded text by itself internally, implicitly assuming that only one encoding would be encountered in each invocation of the program. This was never true, and with the spread of the Internet and then the WWW, it became a major issue. And that's why we invented Python 3, to let text be text without the encumbrance of always being aware of encodings and converting when different encodings collide, etc. From steve at pearwood.info Wed Sep 17 06:42:56 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Sep 2014 14:42:56 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> Message-ID: <20140917044256.GD9293@ando.pearwood.info> On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote: > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote: > > Basically, we are pretending that the each smuggled > > byte is single character for string parsing purposes...but they don't > > match any of our parsing constants. They are all "any character" matches > > in the regexes and what have you. > > This is slightly iffy, as you can't be sure that one byte represents > one character, but as long as you don't much care about that, it's not > going to be an issue. This discussion would probably be a lot more easy to follow, with fewer miscommunications, if there were some examples. Here is my example, perhaps someone can tell me if I'm understanding it correctly. I want to send an email including the header line: 'Subject: ?NOBODY expects the Spanish Inquisition!?' Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I do the right thing and encode it as UTF-8: b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d' but my mail package, not being written in a language as awesome as Python, is just riddled with bugs, and somehow I end up with this corrupted byte-string instead: b'Subject: \x9c\x80\xe2NOBODY expects the Spanish Inquisition!\xe2\x80\x9d' Note that the bytes from the first curly quote bytes are in the wrong order, but the second is okay. (Like I said, it's just *riddled* with bugs.) That means that trying to decode those bytes will fail in Python: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 9: invalid start byte but it's not up to Python's email package to throw those invalid bytes out or permantly replace them with something else. Also, we want to work with Unicode strings, not byte strings, so there has to be a way to smuggle those three bytes into Unicode, without ending up with either the replacement bytes: # using the 'replace' error handler 'Subject: ???NOBODY expects the Spanish Inquisition!?' or incorrectly interpreting them as valid, but wrong, code points. (If we do the second, we end up with two control characters "\x9c\x80" followed by "?".) We want to be able to round-trip back to the same bytes we received. Am I right so far? So the email package uses the surrogate-escape error handler and ends up with this Unicode string: 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!?' which can be encoded back to the bytes we started with. Note that technically those three \u... code points are NOT classified as "noncharacters". They are actually surrogate code points: http://www.unicode.org/faq/private_use.html#nonchar4 http://www.unicode.org/glossary/#surrogate_code_point and they're supposed to be reserved for UTF-16. I'm not sure of the implication of that. > I'm fairly sure you're never going to find an > encoding in which one unknown byte represents two characters, There are encodings which use a "shift" mechanism, whereby a byte X represents one character by default, and a different character after the shift mechanism. But I don't think that matters, since we're not able to interpret those bytes. If we were, we'd just decode them to a text string and be done with it. > but > there are cases where it takes more than one byte to make up a > character (or the bytes are just shift codes or something). Multi-byte encodings are very common. All the Unicode encodings are multi-byte. So are many East Asian encodings. > Does that > ever throw off your regexes? It wouldn't be an issue to a .* between > two character markers, but if you ever say .{5} then it might match > incorrectly. I don't think the idea is to match on these smuggled bytes specifically. I think the idea is to match *around* them. In the example above, we might match everything from "Subject: " to the end of the line. So long as we never end up with a situation where the smuggled bytes are replaced by something else, or shuffled around into different positions, we should be fine. David, is my understanding correct? -- Steven From 4kir4.1i at gmail.com Wed Sep 17 07:10:08 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Wed, 17 Sep 2014 09:10:08 +0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <20140917044256.GD9293@ando.pearwood.info> Message-ID: <871traltn3.fsf@gmail.com> Steven D'Aprano writes: > On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote: >> On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote: > >> > Basically, we are pretending that the each smuggled >> > byte is single character for string parsing purposes...but they don't >> > match any of our parsing constants. They are all "any character" matches >> > in the regexes and what have you. >> >> This is slightly iffy, as you can't be sure that one byte represents >> one character, but as long as you don't much care about that, it's not >> going to be an issue. > > This discussion would probably be a lot more easy to follow, with fewer > miscommunications, if there were some examples. Here is my example, > perhaps someone can tell me if I'm understanding it correctly. > > I want to send an email including the header line: > > 'Subject: ?NOBODY expects the Spanish Inquisition!?' > >>> from email.header import Header >>> h = Header('Subject: ?NOBODY expects the Spanish Inquisition!?') >>> h.encode('utf-8') '=?utf-8?q?Subject=3A_=E2=80=9CNOBODY_expects_the_Spanish_Inquisition!?=\n =?utf-8?q?=E2=80=9D?=' >>> h.encode() '=?utf-8?q?Subject=3A_=E2=80=9CNOBODY_expects_the_Spanish_Inquisition!?=\n =?utf-8?q?=E2=80=9D?=' >>> h.encode('ascii') '=?utf-8?q?Subject=3A_=E2=80=9CNOBODY_expects_the_Spanish_Inquisition!?=\n =?utf-8?q?=E2=80=9D?=' -- Akira From stephen at xemacs.org Wed Sep 17 08:32:00 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Sep 2014 15:32:00 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140917044256.GD9293@ando.pearwood.info> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <20140917044256.GD9293@ando.pearwood.info> Message-ID: <87vbom21wf.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: [long example] > Am I right so far? > > So the email package uses the surrogate-escape error handler and ends up > with this Unicode string: > > 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!?' > > which can be encoded back to the bytes we started with. Yes. > Note that technically those three \u... code points are NOT classified > as "noncharacters". Very unpythonic terminology, easily confusing the nonspecialist. Or the specialist -- I used to know that Unicode gave "noncharacter" a technical definition but it seems I forgot. But then, Unicode isn't a PSF product, so I guess it's OK to be unpythonic. > They are actually surrogate code points: > > http://www.unicode.org/faq/private_use.html#nonchar4 > http://www.unicode.org/glossary/#surrogate_code_point > > and they're supposed to be reserved for UTF-16. I'm not sure of the > implication of that. It means that any Python program that invokes the surrogateescape handler is not a "conforming Unicode process", at least not on the naive interpretation of that definition. A conforming process would interpret them as corrupt characters and raise as soon as detected. A more sophisticated interpretation might argue that Python is multiple processes (in the sense of "process" used by Unicode), and that the Unicode standard only applies to characters. This is especially true of Pythons implementing PEP 393, since no surrogates should ever appear in text[1] at all. Then the smuggled bytes can be treated as noncharacters in practice although technically it's a violation of the Unicode standard to do so. Footnotes: [1] Meaning, no fair using chr() to inject them into str! From rdmurray at bitdance.com Wed Sep 17 09:02:33 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 17 Sep 2014 03:02:33 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140917044256.GD9293@ando.pearwood.info> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <20140917044256.GD9293@ando.pearwood.info> Message-ID: <20140917070234.3578B250E32@webabinitio.net> On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano wrote: > On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote: > > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote: > > > > Basically, we are pretending that the each smuggled > > > byte is single character for string parsing purposes...but they don't > > > match any of our parsing constants. They are all "any character" matches > > > in the regexes and what have you. > > > > This is slightly iffy, as you can't be sure that one byte represents > > one character, but as long as you don't much care about that, it's not > > going to be an issue. > > This discussion would probably be a lot more easy to follow, with fewer > miscommunications, if there were some examples. Here is my example, > perhaps someone can tell me if I'm understanding it correctly. > > I want to send an email including the header line: > > 'Subject: ???NOBODY expects the Spanish Inquisition!???' > > Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I > do the right thing and encode it as UTF-8: > > b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d' That won't work until email supports RFC 6532. Until then, you can only use ascii and encoded words successfully. So just having the curly quotes is a buggy enough program. > but it's not up to Python's email package to throw those invalid bytes > out or permantly replace them with something else. Also, we want to work > with Unicode strings, not byte strings, so there has to be a way to > smuggle those three bytes into Unicode, without ending up with either > the replacement bytes: > > # using the 'replace' error handler > 'Subject: ?????????NOBODY expects the Spanish Inquisition!???' What you'll get if you request a text copy of that header is 'Subject: ?????????NOBODY expects the Spanish Inquisition!?????????' > Am I right so far? > > So the email package uses the surrogate-escape error handler and ends up > with this Unicode string: > > 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!???' Except that it encodes the closing quote, too :) > which can be encoded back to the bytes we started with. Right. If you serialize the message as bytes, the bytes are recovered and output when that header is output. Now, once we support RFC 6532, you will be exactly right, as we will then have the option of handling utf-8 encoded headers, and we will do that using the utf-8 codec to ingest headers, and the surrogateescape error handler to handle exactly the kind of bad data you postulate. --David From rdmurray at bitdance.com Wed Sep 17 09:20:50 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 17 Sep 2014 03:20:50 -0400 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140917070234.3578B250E32@webabinitio.net> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <20140917044256.GD9293@ando.pearwood.info> <20140917070234.3578B250E32@webabinitio.net> Message-ID: <20140917072051.BA663250E29@webabinitio.net> Sorry for the mojibake. I've not yet gotten around to actually using the email package to write a smarter replacement for nmh, which is what I use for email, and I always forget that I need to manually tell nmh when there non-ascii in the message... On Wed, 17 Sep 2014 03:02:33 -0400, "R. David Murray" wrote: > On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano wrote: > > On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote: > > > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote: > > > > > > Basically, we are pretending that the each smuggled > > > > byte is single character for string parsing purposes...but they don't > > > > match any of our parsing constants. They are all "any character" matches > > > > in the regexes and what have you. > > > > > > This is slightly iffy, as you can't be sure that one byte represents > > > one character, but as long as you don't much care about that, it's not > > > going to be an issue. > > > > This discussion would probably be a lot more easy to follow, with fewer > > miscommunications, if there were some examples. Here is my example, > > perhaps someone can tell me if I'm understanding it correctly. > > > > I want to send an email including the header line: > > > > 'Subject: ???NOBODY expects the Spanish Inquisition!???' > > > > Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I > > do the right thing and encode it as UTF-8: > > > > b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d' > > That won't work until email supports RFC 6532. Until then, you can only > use ascii and encoded words successfully. So just having the curly > quotes is a buggy enough program. > > > but it's not up to Python's email package to throw those invalid bytes > > out or permantly replace them with something else. Also, we want to work > > with Unicode strings, not byte strings, so there has to be a way to > > smuggle those three bytes into Unicode, without ending up with either > > the replacement bytes: > > > > # using the 'replace' error handler > > 'Subject: ?????????NOBODY expects the Spanish Inquisition!???' > > What you'll get if you request a text copy of that header is > > 'Subject: ?????????NOBODY expects the Spanish Inquisition!?????????' > > > Am I right so far? > > > > So the email package uses the surrogate-escape error handler and ends up > > with this Unicode string: > > > > 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!???' > > Except that it encodes the closing quote, too :) > > > which can be encoded back to the bytes we started with. > > Right. If you serialize the message as bytes, the bytes are recovered > and output when that header is output. > > Now, once we support RFC 6532, you will be exactly right, as we will > then have the option of handling utf-8 encoded headers, and we will do > that using the utf-8 codec to ingest headers, and the surrogateescape > error handler to handle exactly the kind of bad data you postulate. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From ja.py at farowl.co.uk Wed Sep 17 09:29:20 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Wed, 17 Sep 2014 08:29:20 +0100 Subject: [Python-Dev] Smuggling bytes in a UTF-16 implementation of str/unicode (was: Multilingual programming article on the Red Hat Developer blog) In-Reply-To: <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <54193850.5020105@farowl.co.uk> This feels like a jython-dev discussion. But anyway ... On 17/09/2014 00:57, Stephen J. Turnbull wrote: > The CPython representation uses trailing surrogates only[1], so it's > never possible to interpret them as anything but non-characters -- as > soon as you encounter them you know that it's a lone surrogate. > Surely you can do the same. > > As long as the Java string manipulation functions don't check for > surrogates, you should be fine with this representation. Of course I > suppose your matching functions (etc) don't check for them either, so > you will be somewhat vulnerable to bugs due to treating them as > characters. But the same is true for CPython, AFAIK. They don't check. I agree that since only the trailing surrogate code points are allowed, you can tell that you have one, even in the UTF-16 form. The problem is that, if strings containing lone trailing surrogates are allowed, then: u'\udc83' in u'abc\U00010083xyz' u'abc\U00010083xyz'.endswith(u'\udc83xyz') are both True, if implemented in the obvious way on the UTF-16 representation. And this should not be so in Jython, which claims to be a wide build. (I can't actually type the second one, but I can get the same effect in Jython 2.7b3 via a java.lang.StringBuilder.) I believe that the usual string operations work correctly on the UTF-16 version of the string, as long as indexes are adjusted correctly. If we think it is ok that code using such methods give the wrong answer when fed strings containing smuggled bytes, then isolated (trailing) surrogates could be allowed. It's the user's fault for calling the method on that data. But I think it kinder that our implementation defend users from these wrong answers. In the latest state of Jython, we do this by rigorously preventing the construction of a PyUnicode containing a lone surrogate, so we can just use UTF-16 operations without further checks. I'm not sure that rigour will be universally welcomed, and clearly it precludes PEP-383 byte smuggling. Jeff From steve at pearwood.info Wed Sep 17 10:56:02 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Sep 2014 18:56:02 +1000 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140917085602.GE9293@ando.pearwood.info> On Wed, Sep 17, 2014 at 09:21:56AM +0900, Stephen J. Turnbull wrote: > Guido's mantra is something like "Python's str doesn't contain > characters or even code points[1], it contains code units." But is that true? If it were true, I would expect to be able to make Python text strings containing code units that aren't code points, e.g. something like "\U12340000" or chr(0x12340000) should work, but neither do. As far as I can tell, there is no way to build a string containing items which aren't code points. I don't think it is useful to say that strings *contain* code units, more that they *are made up from* code units. Code units are the implementation: 16-bit code units in narrow builds, 32-bit code units in wide builds, and either 8-, 16- or 32-bit code units in Python 3.3 and beyond. (I don't know of any Python implementation which uses UTF-8 internally, but if there was one, it would use 8-bit code units.) It isn't very useful to say that in Python 3.3 the string "A" *contains* the 8-bit code unit 0x41. That's conflating two different levels of explanation (the high-level interface and the underlying implemention) and potentially leads to user confusion like # 8-bit code units are bytes, right? assert b'\41' in "A" which is Not Even Wrong. http://rationalwiki.org/wiki/Not_even_wrong I think it is correct to say that Python strings are sequences of Unicode code points U+0000 through U+10FFFF. There are no other restrictions, e.g. strings can contain surrogates, noncharacters, or nonsensical combinations of code points such as a U+0300 COMBINING GRAVE ACCENT combined with U+000A (newline). > Implying > that dealing with characters (or the grapheme globs that occasionally > raise their ugly heads here) is an issue for higher-level facilities > than str to deal with. Agreed that Python doesn't offer a string type based on graphemes, and that such a facility belongs as a high-level library, not a built-in type. Also agreed that talking about characters is sloppy. Nevertheless, for English speakers at least, "code point = character" isn't too awful a first approximation. > The point being that > > > Basically, we are pretending that the each smuggled byte is single > > character > > is something of a misstatement (good enough for present purpose of > discussing email, but not good enough for the general case of > understanding how this is supposed to work when porting the construct > to other Python implementations), while > > > for string parsing purposes...but they don't match any of our > > parsing constants. > > is precisely Pythonically correct. You might want to add "because all > parsing constants contain only valid characters by construction." I don't understand what you are trying to say here. > > [*] I worried a lot that this was re-introducing the bytes/string > > problem from python2. > > It isn't, because the bytes/str problem was that given a str object > out of context you could not tell whether it was a binary blob or > text, and if text, you couldn't tell if it was external encoded text > or internal abstract text. > > That is not true here because the representations of characters vs. > smuggled bytes in str are disjoint sets. Nor am I sure what you are trying to say here either. > Footnotes: > [1] In Unicode terminology, a code unit is the smallest computer > object that can represent a character (this is uniquely and sanely > defined for all real Unicode transformation formats aka UTFs). A code > point is an integer 0 - (17*256*256-1) that can represent a character, > but many code points such as surrogates and 0xFFFF are defined to be > non-characters. Actually not quite. "Noncharacter" is concretely defined in Unicode, and there are only 66 of them, many fewer than the surrogate code points alone. Surrogates are reserved, not noncharacters. http://www.unicode.org/glossary/#surrogate_code_point http://www.unicode.org/faq/private_use.html#nonchar1 It is wrong to talk about "surrogate characters", but perhaps you mean to say that surrogates (by which I understand you to mean surrogate code points) are "not human-meaningful characters", which is not the same thing as a Unicode noncharacter. > Characters are those code points that may be assigned > an interpretation as a character, including undefined characters > (private space and reserved). So characters are code points which are characters, including undefined characters? :-) http://www.unicode.org/glossary/#character -- Steven From solipsis at pitrou.net Wed Sep 17 11:37:43 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 17 Sep 2014 11:37:43 +0200 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> <20140917085602.GE9293@ando.pearwood.info> Message-ID: <20140917113743.1059bd8c@fsol> Seriously, can this discussion move somewhere else? This has nothing to do on python-dev. Thank you Antoine. On Wed, 17 Sep 2014 18:56:02 +1000 Steven D'Aprano wrote: > On Wed, Sep 17, 2014 at 09:21:56AM +0900, Stephen J. Turnbull wrote: > > > Guido's mantra is something like "Python's str doesn't contain > > characters or even code points[1], it contains code units." > > But is that true? If it were true, I would expect to be able to make > Python text strings containing code units that aren't code points, e.g. > something like "\U12340000" or chr(0x12340000) should work, but neither > do. As far as I can tell, there is no way to build a string containing > items which aren't code points. > > I don't think it is useful to say that strings *contain* code units, > more that they *are made up from* code units. Code units are the > implementation: 16-bit code units in narrow builds, 32-bit code units > in wide builds, and either 8-, 16- or 32-bit code units in Python 3.3 and > beyond. (I don't know of any Python implementation which uses UTF-8 > internally, but if there was one, it would use 8-bit code units.) > > It isn't very useful to say that in Python 3.3 the string "A" *contains* > the 8-bit code unit 0x41. That's conflating two different levels of > explanation (the high-level interface and the underlying implemention) > and potentially leads to user confusion like > > # 8-bit code units are bytes, right? > assert b'\41' in "A" > > which is Not Even Wrong. > http://rationalwiki.org/wiki/Not_even_wrong > > I think it is correct to say that Python strings are sequences of > Unicode code points U+0000 through U+10FFFF. There are no other > restrictions, e.g. strings can contain surrogates, noncharacters, or > nonsensical combinations of code points such as a U+0300 COMBINING GRAVE > ACCENT combined with U+000A (newline). > > > > Implying > > that dealing with characters (or the grapheme globs that occasionally > > raise their ugly heads here) is an issue for higher-level facilities > > than str to deal with. > > Agreed that Python doesn't offer a string type based on graphemes, and > that such a facility belongs as a high-level library, not a built-in > type. > > Also agreed that talking about characters is sloppy. Nevertheless, for > English speakers at least, "code point = character" isn't too awful a > first approximation. > > > > The point being that > > > > > Basically, we are pretending that the each smuggled byte is single > > > character > > > > is something of a misstatement (good enough for present purpose of > > discussing email, but not good enough for the general case of > > understanding how this is supposed to work when porting the construct > > to other Python implementations), while > > > > > for string parsing purposes...but they don't match any of our > > > parsing constants. > > > > is precisely Pythonically correct. You might want to add "because all > > parsing constants contain only valid characters by construction." > > I don't understand what you are trying to say here. > > > > > [*] I worried a lot that this was re-introducing the bytes/string > > > problem from python2. > > > > It isn't, because the bytes/str problem was that given a str object > > out of context you could not tell whether it was a binary blob or > > text, and if text, you couldn't tell if it was external encoded text > > or internal abstract text. > > > > That is not true here because the representations of characters vs. > > smuggled bytes in str are disjoint sets. > > Nor am I sure what you are trying to say here either. > > > > Footnotes: > > [1] In Unicode terminology, a code unit is the smallest computer > > object that can represent a character (this is uniquely and sanely > > defined for all real Unicode transformation formats aka UTFs). A code > > point is an integer 0 - (17*256*256-1) that can represent a character, > > but many code points such as surrogates and 0xFFFF are defined to be > > non-characters. > > Actually not quite. "Noncharacter" is concretely defined in Unicode, and > there are only 66 of them, many fewer than the surrogate code points > alone. Surrogates are reserved, not noncharacters. > > http://www.unicode.org/glossary/#surrogate_code_point > http://www.unicode.org/faq/private_use.html#nonchar1 > > It is wrong to talk about "surrogate characters", but perhaps you mean > to say that surrogates (by which I understand you to mean surrogate code > points) are "not human-meaningful characters", which is not the same > thing as a Unicode noncharacter. > > > > Characters are those code points that may be assigned > > an interpretation as a character, including undefined characters > > (private space and reserved). > > So characters are code points which are characters, including undefined > characters? :-) > > http://www.unicode.org/glossary/#character > > > From martin at v.loewis.de Wed Sep 17 14:06:06 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 17 Sep 2014 14:06:06 +0200 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140917085602.GE9293@ando.pearwood.info> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> <20140917085602.GE9293@ando.pearwood.info> Message-ID: <5419792E.10507@v.loewis.de> Am 17.09.14 10:56, schrieb Steven D'Aprano: > On Wed, Sep 17, 2014 at 09:21:56AM +0900, Stephen J. Turnbull wrote: > >> Guido's mantra is something like "Python's str doesn't contain >> characters or even code points[1], it contains code units." > > But is that true? It used to be true, and stopped being so with PEP 393. In particular, Python 3.2 and before would expose UTF-16 in the narrow build, so the elements of a string would be code units. Since Python 3.3, the surrogate code points are not longer interpreted as UTF-16 code units. Regards, Martin From stephen at xemacs.org Thu Sep 18 06:45:36 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 18 Sep 2014 13:45:36 +0900 Subject: [Python-Dev] Smuggling bytes in a UTF-16 implementation of str/unicode (was: Multilingual programming article on the Red Hat Developer blog) In-Reply-To: <54193850.5020105@farowl.co.uk> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <87bnqf2k66.fsf@uwakimon.sk.tsukuba.ac.jp> <54193850.5020105@farowl.co.uk> Message-ID: <87lhph1qq7.fsf@uwakimon.sk.tsukuba.ac.jp> Jeff Allen writes: > This feels like a jython-dev discussion. But anyway ... Well, if the same representation could be used in Jython you could just point to PEP 383 and be done with it. > u'\udc83' in u'abc\U00010083xyz' IMHO being able to type that is a bug. There should be no literal notation for surrogates in Python (that is, if you type a literal that looks like it refers to a surrogate, you should get a UnicodeError). The "right way" (IMHO) to spell that is chr(0xdc83) in u'abc\U00010083xyz' I'm not Guido, and don't claim to channel him on this. But it seems reasonable to me that Unicode literals should conform to Unicode in this way. I might even extend that to noncharacters (the last two code points in each plane and the 32-point "hole" in Arabic). I'll grant that chr() is an unfortunate spelling, but I would imagine we could live with that since chr() goes back forever with these semantics. > u'abc\U00010083xyz'.endswith(u'\udc83xyz') > > are both True, if implemented in the obvious way on the UTF-16 > representation. And this should not be so in Jython, which claims to be > a wide build. (I can't actually type the second one, but I can get the > same effect in Jython 2.7b3 via a java.lang.StringBuilder.) I agree that's very ugly, but AFAIK that's how things would work in narrow CPython (which uses UTF-16 internally for the astral planes). Personally I would document that explicit smuggled bytes are not supported for comparison operations, and leave it at that. > If we think it is ok that code using such methods give the wrong answer > when fed strings containing smuggled bytes, then isolated (trailing) > surrogates could be allowed. It's the user's fault for calling the > method on that data. But I think it kinder that our implementation > defend users from these wrong answers. In the latest state of Jython, we > do this by rigorously preventing the construction of a PyUnicode > containing a lone surrogate, so we can just use UTF-16 operations > without further checks. That seems like a reasonable approach. From stephen at xemacs.org Thu Sep 18 06:57:40 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 18 Sep 2014 13:57:40 +0900 Subject: [Python-Dev] Multilingual programming article on the Red Hat Developer blog In-Reply-To: <20140917085602.GE9293@ando.pearwood.info> References: <541370BE.8030700@farowl.co.uk> <54173155.087fe00a.7669.4f8e@mx.google.com> <87sijs2q7n.fsf@uwakimon.sk.tsukuba.ac.jp> <20140916150033.3733D250E2B@webabinitio.net> <20140916174650.79DDF250E49@webabinitio.net> <20140916192930.BB833250E42@webabinitio.net> <87a95z2j17.fsf@uwakimon.sk.tsukuba.ac.jp> <20140917085602.GE9293@ando.pearwood.info> Message-ID: <87k3511q63.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > On Wed, Sep 17, 2014 at 09:21:56AM +0900, Stephen J. Turnbull wrote: > > > Guido's mantra is something like "Python's str doesn't contain > > characters or even code points[1], it contains code units." > > But is that true? It's not. That's why I wrote the slightly pejorative "mantra" and qualified it with "something like". The precise statement is "something like" the array property is more important than preserving character boundaries, so slices etc are allowed to do unexpected or even evil things in the presence of astral characters in UTF-16 representations. > I don't understand what you are trying to say here. > Nor am I sure what you are trying to say here either. We can discuss this off-list if you would like. The natives are getting restless. > > non-characters. > > Actually not quite. "Noncharacter" Note the hyphen! (Just kidding, I will avoid that terminology in the future. I knew, but forgot.) > > Characters are those code points that may be assigned > > an interpretation as a character, including undefined characters > > (private space and reserved). > > So characters are code points which are characters, including undefined > characters? :-) No, there's a clear hierarchy here. From benhoyt at gmail.com Fri Sep 19 03:30:17 2014 From: benhoyt at gmail.com (Ben Hoyt) Date: Thu, 18 Sep 2014 21:30:17 -0400 Subject: [Python-Dev] Adding numbering to PEP 20, the Zen of Python Message-ID: I was emailing someone today about implementing something (for PEP 471, as it happens) and wanted to link to the Zen of Python [1] and note a particular clause (in this case "If the implementation is hard to explain, it's a bad idea."). However, there are no clause numbers, so you can't refer to specific phrases. I know it's a short enough document that it probably doesn't matter. And maybe numbering them would make it less Zen. Would be handy in code reviews and the like, for example: "Not very Pythonic. See PEP 20 point 5." Is it just my pedantic self, or have others wanted to do this too? [1] http://legacy.python.org/dev/peps/pep-0020/ -Ben From graffatcolmingov at gmail.com Fri Sep 19 03:40:34 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 18 Sep 2014 20:40:34 -0500 Subject: [Python-Dev] Adding numbering to PEP 20, the Zen of Python In-Reply-To: References: Message-ID: On Thu, Sep 18, 2014 at 8:30 PM, Ben Hoyt wrote: > I was emailing someone today about implementing something (for PEP > 471, as it happens) and wanted to link to the Zen of Python [1] and > note a particular clause (in this case "If the implementation is hard > to explain, it's a bad idea."). However, there are no clause numbers, > so you can't refer to specific phrases. > > I know it's a short enough document that it probably doesn't matter. > And maybe numbering them would make it less Zen. Would be handy in > code reviews and the like, for example: "Not very Pythonic. See PEP 20 > point 5." Is it just my pedantic self, or have others wanted to do > this too? > > [1] http://legacy.python.org/dev/peps/pep-0020/ > > -Ben > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com It's just you I think. Also, isn't this better suited for python-ideas? From tim.peters at gmail.com Fri Sep 19 03:46:23 2014 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 18 Sep 2014 20:46:23 -0500 Subject: [Python-Dev] Adding numbering to PEP 20, the Zen of Python In-Reply-To: References: Message-ID: [Ben Hoyt] > I was emailing someone today about implementing something (for PEP > 471, as it happens) and wanted to link to the Zen of Python [1] and > note a particular clause (in this case "If the implementation is hard > to explain, it's a bad idea."). However, there are no clause numbers, > so you can't refer to specific phrases. > > I know it's a short enough document that it probably doesn't matter. > And maybe numbering them would make it less Zen. Would be handy in > code reviews and the like, for example: "Not very Pythonic. See PEP 20 > point 5." Is it just my pedantic self, or have others wanted to do > this too? > > [1] http://legacy.python.org/dev/peps/pep-0020/ See Revelation 22:18. Adding numbers is how heresy begins ;-) http://biblehub.com/revelation/22-18.htm From barry at python.org Fri Sep 19 04:34:47 2014 From: barry at python.org (Barry Warsaw) Date: Thu, 18 Sep 2014 22:34:47 -0400 Subject: [Python-Dev] Adding numbering to PEP 20, the Zen of Python In-Reply-To: References: Message-ID: <20140918223447.6d0e42d6@anarchist.wooz.org> On Sep 18, 2014, at 08:46 PM, Tim Peters wrote: >See Revelation 22:18. Adding numbers is how heresy begins ;-) Thus Spake Timbot -Barry From victor.stinner at gmail.com Fri Sep 19 08:11:40 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 19 Sep 2014 08:11:40 +0200 Subject: [Python-Dev] Adding numbering to PEP 20, the Zen of Python In-Reply-To: References: Message-ID: Why not quoting the whole sentence? Victor Le 19 sept. 2014 03:31, "Ben Hoyt" a ?crit : > I was emailing someone today about implementing something (for PEP > 471, as it happens) and wanted to link to the Zen of Python [1] and > note a particular clause (in this case "If the implementation is hard > to explain, it's a bad idea."). However, there are no clause numbers, > so you can't refer to specific phrases. > > I know it's a short enough document that it probably doesn't matter. > And maybe numbering them would make it less Zen. Would be handy in > code reviews and the like, for example: "Not very Pythonic. See PEP 20 > point 5." Is it just my pedantic self, or have others wanted to do > this too? > > [1] http://legacy.python.org/dev/peps/pep-0020/ > > -Ben > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkabrda at redhat.com Fri Sep 19 09:31:39 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Fri, 19 Sep 2014 03:31:39 -0400 (EDT) Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> Message-ID: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> Hi, as Fedora is getting closer to having python3 as a default, I'm being more and more asked by Fedora users/contributors what'll "/usr/bin/python" invoke when we achieve this (Fedora 22 hopefully). So I was rereading PEP 394 and I think I need a small clarification regarding two points in the PEP: - "for the time being, all distributions should ensure that python refers to the same target as python2." - "Similarly, the more general python command should be installed whenever any version of Python is installed and should invoke the same version of Python as either python2 or python3." The important word in the second point is, I think, *whenever*. Trying to apply these two points to Fedora 22 situation, I can think of several approaches: - /usr/bin/python will always point to python3 (seems to go against the first mentioned PEP recommendation) - /usr/bin/python will always point to python2 (seems to go against the second mentioned PEP recommendation, there is no /usr/bin/python if python2 is not installed) - /usr/bin/python will point to python3 if python2 is not installed, else it will point to python2 (inconsistent; also the user doesn't know he's running and what libraries he'll be able to import - the system can have different sets of python2-* and python3-* extension modules installed) - there will be no /usr/bin/python (goes against PEP and seems just wrong) I'd really appreciate upstream guidance and perhaps a PEP clarification for distributions that ship both python2 and python3, but can live without python2 (and are not Arch :)). Thanks a lot! -- Regards, Slavek Kabrda From donald at stufft.io Fri Sep 19 10:44:26 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 19 Sep 2014 04:44:26 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> References: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> Message-ID: <92F5642C-39F7-42DA-AFC7-8067BEAB3130@stufft.io> > On Sep 19, 2014, at 3:31 AM, Bohuslav Kabrda wrote: > > Hi, > as Fedora is getting closer to having python3 as a default, I'm being more and more asked by Fedora users/contributors what'll "/usr/bin/python" invoke when we achieve this (Fedora 22 hopefully). So I was rereading PEP 394 and I think I need a small clarification regarding two points in the PEP: > - "for the time being, all distributions should ensure that python refers to the same target as python2." > - "Similarly, the more general python command should be installed whenever any version of Python is installed and should invoke the same version of Python as either python2 or python3." > > The important word in the second point is, I think, *whenever*. Trying to apply these two points to Fedora 22 situation, I can think of several approaches: > - /usr/bin/python will always point to python3 (seems to go against the first mentioned PEP recommendation) > - /usr/bin/python will always point to python2 (seems to go against the second mentioned PEP recommendation, there is no /usr/bin/python if python2 is not installed) > - /usr/bin/python will point to python3 if python2 is not installed, else it will point to python2 (inconsistent; also the user doesn't know he's running and what libraries he'll be able to import - the system can have different sets of python2-* and python3-* extension modules installed) > - there will be no /usr/bin/python (goes against PEP and seems just wrong) > > I'd really appreciate upstream guidance and perhaps a PEP clarification for distributions that ship both python2 and python3, but can live without python2 (and are not Arch :)). > > Thanks a lot! I don?t know for a fact, but I assume that as long as Python 2.x is installed by default than ``python`` should point to ``python2``. If Python 3.x is the default version and Python 2.x is the ?optional? version than I think personally it makes sense to switch eventually. Maybe not immediately to give people time to update though? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Sep 19 11:31:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Sep 2014 19:31:17 +1000 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <92F5642C-39F7-42DA-AFC7-8067BEAB3130@stufft.io> References: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <92F5642C-39F7-42DA-AFC7-8067BEAB3130@stufft.io> Message-ID: <20140919093117.GY9293@ando.pearwood.info> On Fri, Sep 19, 2014 at 04:44:26AM -0400, Donald Stufft wrote: > > > On Sep 19, 2014, at 3:31 AM, Bohuslav Kabrda wrote: > > > > Hi, as Fedora is getting closer to having python3 as a default, I'm > > being more and more asked by Fedora users/contributors what'll > > "/usr/bin/python" invoke when we achieve this (Fedora 22 hopefully). > > So I was rereading PEP 394 and I think I need a small clarification > > regarding two points in the PEP: - "for the time being, all > > distributions should ensure that python refers to the same target as > > python2." - "Similarly, the more general python command should be > > installed whenever any version of Python is installed and should > > invoke the same version of Python as either python2 or python3." > > > > The important word in the second point is, I think, *whenever*. > > Trying to apply these two points to Fedora 22 situation, I can think > > of several approaches: > > - /usr/bin/python will always point to python3 (seems to go against > > the first mentioned PEP recommendation) Definitely not that. Arch Linux pointed /usr/bin/python at Python 3 some years ago, and I understand that this has caused no end of trouble for the folks on #python. I haven't seen any sign of this being an issue on the tutor@ or python-list at python.org mailing lists, but the demographics are quite different so that's not surprising. > > - /usr/bin/python will always point to python2 (seems to go against > > the second mentioned PEP recommendation, there is no /usr/bin/python > > if python2 is not installed) My understanding is that this is the intention of the PEP, at least until such time as Python 2 is end-of-lifed. My interpretion would be that the second recommendation in the PEP is just confused :-) Perhaps the PEP author could clarify what the intention is. > > - /usr/bin/python will point to python3 if python2 is not installed, > > else it will point to python2 (inconsistent; also the user doesn't > > know he's running and what libraries he'll be able to import - the > > system can have different sets of python2-* and python3-* extension > > modules installed) Likely to cause all sorts of problems, and I understood that this was not the intention. Perhaps it was added *only* as a "grand-father clause" so that people don't yell at Arch Linux "See, the PEP says you're doing it wrong!". > > - there will be no /usr/bin/python (goes against PEP and seems just wrong) Seems like the least-worst to me. If you think of "python == Python 2.x" (at least for the next few years), then if Python 2.x isn't installed, there should be no /usr/bin/python either. > I don?t know for a fact, but I assume that as long as Python 2.x is > installed by default than ``python`` should point to ``python2``. If > Python 3.x is the default version and Python 2.x is the ?optional? > version than I think personally it makes sense to switch eventually. > Maybe not immediately to give people time to update though? Agreed. Once Python 2 is finally end-of-lifed in 2023 or thereabouts, then we can reconsider pointing /usr/bin/python at Python 3 (or 4, whatever is current by then). If Arch Linux jumped the gun by a decade or so, that's their problem :-) -- Steven From ncoghlan at gmail.com Fri Sep 19 12:07:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Sep 2014 20:07:02 +1000 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> Message-ID: On 19 Sep 2014 17:38, "Bohuslav Kabrda" wrote: > - "Similarly, the more general python command should be installed whenever any version of Python is installed and should invoke the same version of Python as either python2 or python3." > > The important word in the second point is, I think, *whenever*. Trying to apply these two points to Fedora 22 situation, I can think of several approaches: > - /usr/bin/python will always point to python3 (seems to go against the first mentioned PEP recommendation) > - /usr/bin/python will always point to python2 (seems to go against the second mentioned PEP recommendation, there is no /usr/bin/python if python2 is not installed) I think this is what should happen, and the PEP is currently wrong. When writing the PEP, I don't think we accounted properly for the case where the "system Python" has migrated to Python 3, but the "default Python for end user scripts that don't specify otherwise" is still Python 2 (which is the migration strategy both Fedora and Ubuntu are adopting). How does this sound as a possible revised recommendation (keep in mind I haven't checked this against the larger context yet): "The more general python command should only be installed whenever the corresponding version of Python is installed (whether python2 or python3)." Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkabrda at redhat.com Fri Sep 19 12:12:38 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Fri, 19 Sep 2014 06:12:38 -0400 (EDT) Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> Message-ID: <337736447.19208726.1411121558311.JavaMail.zimbra@redhat.com> ----- Original Message ----- > On 19 Sep 2014 17:38, "Bohuslav Kabrda" < bkabrda at redhat.com > wrote: > > - "Similarly, the more general python command should be installed whenever > > any version of Python is installed and should invoke the same version of > > Python as either python2 or python3." > > > > The important word in the second point is, I think, *whenever*. Trying to > > apply these two points to Fedora 22 situation, I can think of several > > approaches: > > - /usr/bin/python will always point to python3 (seems to go against the > > first mentioned PEP recommendation) > > - /usr/bin/python will always point to python2 (seems to go against the > > second mentioned PEP recommendation, there is no /usr/bin/python if > > python2 is not installed) > I think this is what should happen, and the PEP is currently wrong. When > writing the PEP, I don't think we accounted properly for the case where the > "system Python" has migrated to Python 3, but the "default Python for end > user scripts that don't specify otherwise" is still Python 2 (which is the > migration strategy both Fedora and Ubuntu are adopting). Thanks, that was my thinking, too. > How does this sound as a possible revised recommendation (keep in mind I > haven't checked this against the larger context yet): > "The more general python command should only be installed whenever the > corresponding version of Python is installed (whether python2 or python3)." It seems to me that it is a bit unclear what "corresponding" is. Would it make sense to explicitly say that "python" command should be installed whenever the distro-chosen default system Python is installed? > Regards, > Nick. Thanks a lot -- Regards, Slavek Kabrda -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Sep 19 13:13:44 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 19 Sep 2014 20:13:44 +0900 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <337736447.19208726.1411121558311.JavaMail.zimbra@redhat.com> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <337736447.19208726.1411121558311.JavaMail.zimbra@redhat.com> Message-ID: There are many python2 only scripts with "#!/usr/bin/python" or "#!/usr/bin/env python" shebang in the world. I think Ubuntu and Fedora's strategy is better for now. On Fri, Sep 19, 2014 at 7:12 PM, Bohuslav Kabrda wrote: > > ________________________________ > > > On 19 Sep 2014 17:38, "Bohuslav Kabrda" wrote: >> - "Similarly, the more general python command should be installed whenever >> any version of Python is installed and should invoke the same version of >> Python as either python2 or python3." >> >> The important word in the second point is, I think, *whenever*. Trying to >> apply these two points to Fedora 22 situation, I can think of several >> approaches: >> - /usr/bin/python will always point to python3 (seems to go against the >> first mentioned PEP recommendation) >> - /usr/bin/python will always point to python2 (seems to go against the >> second mentioned PEP recommendation, there is no /usr/bin/python if python2 >> is not installed) > > I think this is what should happen, and the PEP is currently wrong. When > writing the PEP, I don't think we accounted properly for the case where the > "system Python" has migrated to Python 3, but the "default Python for end > user scripts that don't specify otherwise" is still Python 2 (which is the > migration strategy both Fedora and Ubuntu are adopting). > > Thanks, that was my thinking, too. > > How does this sound as a possible revised recommendation (keep in mind I > haven't checked this against the larger context yet): > > "The more general python command should only be installed whenever the > corresponding version of Python is installed (whether python2 or python3)." > > It seems to me that it is a bit unclear what "corresponding" is. Would it > make sense to explicitly say that "python" command should be installed > whenever the distro-chosen default system Python is installed? > > Regards, > Nick. > > > Thanks a lot > > -- > Regards, > Slavek Kabrda > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > -- INADA Naoki From barry at python.org Fri Sep 19 16:16:20 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Sep 2014 10:16:20 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> Message-ID: <20140919101620.01189b45@limelight.wooz.org> On Sep 19, 2014, at 03:31 AM, Bohuslav Kabrda wrote: >as Fedora is getting closer to having python3 as a default, I'm being more >and more asked by Fedora users/contributors what'll "/usr/bin/python" invoke >when we achieve this (Fedora 22 hopefully). So I was rereading PEP 394 and I >think I need a small clarification regarding two points in the PEP: As Nick points out, I think that the Fedora and Debian ecosystems are aligned on what we *should* do, even if the PEP text needs updating. The way I look at it is that "/usr/bin/python" is user interface. Distributions are completely free to choose whichever Python they want for system scripts, and it's great to see that Fedora is well on their way to making Python 3 the default for system scripts. Debian is also making good progress, though we likely won't complete the transition until Jessie+1. But none of that matters for /usr/bin/python because system scripts will just shebang /usr/bin/python3 and nobody cares. Users almost certainly don't care as long as the script just keeps working, although if there are plugins, it will indirectly matter for them. Only the maintainers of those system scripts really care what the shebang is. For those system scripts, or other /usr/bin entries, that still require Python 2, I think they should already be shebanged /usr/bin/python2. OTOH, users have /usr/bin on their $PATH and they're just going to type 'python' so for them, it's ui, and it *does* matter. In that case I think /usr/bin/python should always point to /usr/bin/python2 which always points to the default Python 2 version (these days, for modern OS versions, almost always 2.7). I don't care how that "pointing" actually works. TL;DR: use #!/usr/bin/python3 for Python 3 system scripts, #!/usr/bin/python2 for Python 2 system scripts, and leave /usr/bin/python to invoke the default Python 2 version. >The important word in the second point is, I think, *whenever*. Trying to >apply these two points to Fedora 22 situation, I can think of several >approaches: >- /usr/bin/python will always point to python3 (seems to go against the first >mentioned PEP recommendation) >- /usr/bin/python will always point to python2 (seems to go against the >second mentioned PEP recommendation, there is no /usr/bin/python if python2 >is not installed) >- /usr/bin/python will point to python3 if python2 is not installed, else it >will point to python2 (inconsistent; also the user doesn't know he's >running and what libraries he'll be able to import - the system can have >different sets of python2-* and python3-* extension modules installed) >- there will be no /usr/bin/python (goes against PEP and seems just wrong) I think the second point is right. If /usr/bin/python2 is not installed, neither should there be a /usr/bin/python. The reasoning is aligned with my views above. Python 2 won't go away, so even if it is not installed *by default* (and I hope we get there in both Debian and Ubuntu), it is entirely reasonable to assume it will eventually get installed. Someday a user is going to install a script that is Python 2, and then they'll get /usr/bin/python2 installed for that script's shebang. At that point, they'll *also* get /usr/bin/python which of course points to /usr/bin/python2. If the user wants to invoke Python 3, it's not hard to type 'python3' and I think that's the message we should be spreading. That already seems pretty ingrained in user habits afaict. If there's general agreement on that, we can work on improving the PEP's wording to make that policy more clear. Cheers, -Barry From donald at stufft.io Fri Sep 19 16:23:51 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 19 Sep 2014 10:23:51 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919101620.01189b45@limelight.wooz.org> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> Message-ID: <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> > On Sep 19, 2014, at 10:16 AM, Barry Warsaw wrote: > > If the user wants to invoke Python 3, it's not hard to type 'python3' and I > think that's the message we should be spreading. That already seems pretty > ingrained in user habits afaict. My biggest problem with ``python3``, is what happens after 3.9. I know Guido doesn?t particularly like two digit version numbers and it?s been suggested on this list that instead of 3.10 we?re likely to move directly into 4.0 regardless of if it?s a ?big? change or not. If that is the case, then all of the user education, ui, etc around ``python3`` would then need to be again updated to ``python4`` *OR* we?d need a ``python3`` bin which points to ``python4``. If there?s a call to action for at some point moving ``python`` to invoke Python 3.x at some point then hopefully at that point Python 4.x could just be ``python``. All of this assuming of course that 4.0 isn?t a major break like 3.0 and that we do 4.0 instead of 3.10 as has been suggested. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Sep 19 16:31:46 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Sep 2014 10:31:46 -0400 Subject: [Python-Dev] 3.5 release schedule PEP Message-ID: <20140919103146.0d763a24@limelight.wooz.org> Hi Larry, I think we need a Python 3.5 Release Schedule PEP. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Fri Sep 19 16:41:58 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Sep 2014 10:41:58 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> Message-ID: <20140919104158.678e2234@limelight.wooz.org> On Sep 19, 2014, at 10:23 AM, Donald Stufft wrote: >My biggest problem with ``python3``, is what happens after 3.9. FWIW, 3.9 by my rough calculation is 7 years away. >I know Guido doesn?t particularly like two digit version numbers and it?s >been suggested on this list that instead of 3.10 we?re likely to move >directly into 4.0 regardless of if it?s a ?big? change or not. > >If that is the case, then all of the user education, ui, etc around >``python3`` would then need to be again updated to ``python4`` *OR* we?d need >a ``python3`` bin which points to ``python4``. If there?s a call to action >for at some point moving ``python`` to invoke Python 3.x at some point then >hopefully at that point Python 4.x could just be ``python``. > >All of this assuming of course that 4.0 isn?t a major break like 3.0 and that >we do 4.0 instead of 3.10 as has been suggested. I seem to recall Guido saying that *if* there's a 4.0, it won't be a major break like Python 3, whatever that says about the numbering scheme after 3.9. Is 7 years enough to eradicate Python 2 the way we did for Python 1? Then maybe Python 4 can reclaim /usr/bin/python. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From steve at pearwood.info Fri Sep 19 17:03:30 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Sep 2014 01:03:30 +1000 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919104158.678e2234@limelight.wooz.org> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> Message-ID: <20140919150330.GR9293@ando.pearwood.info> On Fri, Sep 19, 2014 at 10:41:58AM -0400, Barry Warsaw wrote: > On Sep 19, 2014, at 10:23 AM, Donald Stufft wrote: > > >My biggest problem with ``python3``, is what happens after 3.9. > > FWIW, 3.9 by my rough calculation is 7 years away. That makes it 2021, one year after Python 2.7 free support ends, but two years before Red Hat commercial support for it ends. > I seem to recall Guido saying that *if* there's a 4.0, it won't be a major > break like Python 3, whatever that says about the numbering scheme after 3.9. > > Is 7 years enough to eradicate Python 2 the way we did for Python 1? Then > maybe Python 4 can reclaim /usr/bin/python. I expect not quite. Perhaps 10 years though. -- Steven From rdmurray at bitdance.com Fri Sep 19 17:17:27 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 19 Sep 2014 11:17:27 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919101620.01189b45@limelight.wooz.org> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> Message-ID: <20140919151731.21137250E20@webabinitio.net> On Fri, 19 Sep 2014 10:16:20 -0400, Barry Warsaw wrote: > The way I look at it is that "/usr/bin/python" is user interface. > Distributions are completely free to choose whichever Python they want for > system scripts, and it's great to see that Fedora is well on their way to > making Python 3 the default for system scripts. Debian is also making good > progress, though we likely won't complete the transition until Jessie+1. Yep, user interface. Therefore I think the distribution should let the user choose :) Most users will chose python2, because that's what other scripts they might download and install will expect. So that should be the default. But I'll choose python3 (and have, on most of my gentoo linux systems, which supports having both quite well, for system installed packages). On the gripping hand, given what Fedora is doing, it is not that hard to change (or create) the symlinks as an end user to point to the python3 programs even if Fedora doesn't support it directly, so I don't see a problem with the proposed strategy. > If the user wants to invoke Python 3, it's not hard to type 'python3' and I > think that's the message we should be spreading. That already seems pretty > ingrained in user habits afaict. As Donald pointed out, that presents a problem for the future. But since there is no good alternative to putting /usr/bin/python3 in the shebang of scripts that require python3, I don't think there is a solution and we're just going to have to cross that further bridge when we come to it. --David From marko at pacujo.net Fri Sep 19 17:05:02 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 19 Sep 2014 18:05:02 +0300 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> (Donald Stufft's message of "Fri, 19 Sep 2014 10:23:51 -0400") References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> Message-ID: <8738bn6481.fsf@elektro.pacujo.net> Donald Stufft : > My biggest problem with ``python3``, is what happens after 3.9. I know > Guido doesn?t particularly like two digit version numbers and it?s > been suggested on this list that instead of 3.10 we?re likely to move > directly into 4.0 regardless of if it?s a ?big? change or not. python3 should be the name of the programming language specification. If CPython-4.0 supports all of the python3 programming language, /usr/bin/python3 should be a symbolic link to CPython-4.0. Or, if I reimplemented python3 with cyphon0.7, python3 could be a link to that. If CPython-4.x dropped support for some python3 features, you would no longer link python3 to it. Now, what would plain "python" be? It would make life easier for a lot of people if it were "python2" for all eternity. By analogy, look at #!/bin/sh, which used to invoke the Bourne shell, later /bin/bash and on modern Debian, /bin/dash. Marko From guido at python.org Fri Sep 19 17:20:48 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Sep 2014 08:20:48 -0700 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919150330.GR9293@ando.pearwood.info> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> <20140919150330.GR9293@ando.pearwood.info> Message-ID: "python" should always be the same as "python2". On Fri, Sep 19, 2014 at 8:03 AM, Steven D'Aprano wrote: > On Fri, Sep 19, 2014 at 10:41:58AM -0400, Barry Warsaw wrote: > > On Sep 19, 2014, at 10:23 AM, Donald Stufft wrote: > > > > >My biggest problem with ``python3``, is what happens after 3.9. > > > > FWIW, 3.9 by my rough calculation is 7 years away. > > That makes it 2021, one year after Python 2.7 free support ends, but two > years before Red Hat commercial support for it ends. > > > I seem to recall Guido saying that *if* there's a 4.0, it won't be a > major > > break like Python 3, whatever that says about the numbering scheme after > 3.9. > > > > Is 7 years enough to eradicate Python 2 the way we did for Python 1? > Then > > maybe Python 4 can reclaim /usr/bin/python. > > I expect not quite. Perhaps 10 years though. > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Sep 19 17:35:19 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Sep 2014 17:35:19 +0200 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> <20140919150330.GR9293@ando.pearwood.info> Message-ID: <20140919173519.1b9cc88e@fsol> On Fri, 19 Sep 2014 08:20:48 -0700 Guido van Rossum wrote: > "python" should always be the same as "python2". "Always" as in "eternally"? From guido at python.org Fri Sep 19 17:40:50 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Sep 2014 08:40:50 -0700 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919173519.1b9cc88e@fsol> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> <20140919150330.GR9293@ando.pearwood.info> <20140919173519.1b9cc88e@fsol> Message-ID: On Sep 19, 2014 8:36 AM, "Antoine Pitrou" wrote: > > On Fri, 19 Sep 2014 08:20:48 -0700 > Guido van Rossum wrote: > > "python" should always be the same as "python2". > > "Always" as in "eternally"? Until I say so. Which will happen in the distant future. -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Sep 19 18:07:41 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 19 Sep 2014 18:07:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140919160741.B2F5F5613E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-09-12 - 2014-09-19) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4662 (+10) closed 29544 (+35) total 34206 (+45) Open issues with patches: 2197 Issues opened (33) ================== #22166: test_codecs leaks references http://bugs.python.org/issue22166 reopened by haypo #22395: test_pathlib error for complex symlinks on Windows http://bugs.python.org/issue22395 opened by jfoo #22396: AIX posix_fadvise and posix_fallocate http://bugs.python.org/issue22396 opened by David.Edelsohn #22397: test_socket failure on AIX http://bugs.python.org/issue22397 opened by David.Edelsohn #22401: argparse: 'resolve' conflict handler damages the actions of th http://bugs.python.org/issue22401 opened by paul.j3 #22406: uu-codec trailing garbage workaround is Python 2 code http://bugs.python.org/issue22406 opened by vadmium #22407: re.LOCALE is nonsensical for Unicode http://bugs.python.org/issue22407 opened by serhiy.storchaka #22408: Tkinter doesn't handle Unicode key events on Windows http://bugs.python.org/issue22408 opened by Drekin #22410: Locale dependent regexps on different locales http://bugs.python.org/issue22410 opened by serhiy.storchaka #22411: Embedding Python on Windows http://bugs.python.org/issue22411 opened by Joakim.Karlsson #22413: Bizarre StringIO(newline="\r\n") translation http://bugs.python.org/issue22413 opened by vadmium #22415: Fix re debugging output http://bugs.python.org/issue22415 opened by serhiy.storchaka #22417: PEP 476: verify HTTPS certificates by default http://bugs.python.org/issue22417 opened by ncoghlan #22418: ipaddress.py new IPv6 Method for Solicited Multicast Address http://bugs.python.org/issue22418 opened by Jason.Nadeau #22420: Use print(file=sys.stderr) instead of sys.stderr.write() in ID http://bugs.python.org/issue22420 opened by serhiy.storchaka #22422: IDLE closes all when in dropdown menu http://bugs.python.org/issue22422 opened by mandolout #22423: Errors in printing exceptions raised in a thread http://bugs.python.org/issue22423 opened by serhiy.storchaka #22425: 2to3 import fixer writes dotted_as_names into import_as_names http://bugs.python.org/issue22425 opened by simonmweber #22426: strptime accepts the wrong '2010-06-01 MSK' string but rejects http://bugs.python.org/issue22426 opened by akira #22427: TemporaryDirectory attempts to clean up twice http://bugs.python.org/issue22427 opened by oconnor663 #22428: asyncio: KeyboardInterrupt inside a coroutine causes Attribute http://bugs.python.org/issue22428 opened by oconnor663 #22429: asyncio: pending call to loop.stop() if a coroutine raises a B http://bugs.python.org/issue22429 opened by haypo #22430: Build failure if configure flags --prefix or --exec-prefix is http://bugs.python.org/issue22430 opened by diff.812 #22431: Change format of test runner output http://bugs.python.org/issue22431 opened by googol #22433: Argparse considers unknown optional arguments with spaces as a http://bugs.python.org/issue22433 opened by DenKoren #22434: Use named constants internally in the re module http://bugs.python.org/issue22434 opened by serhiy.storchaka #22435: socketserver.TCPSocket leaks socket to garbage collector if se http://bugs.python.org/issue22435 opened by vadmium #22437: re module: number of named groups is limited to 100 max http://bugs.python.org/issue22437 opened by yselivanov #22438: eventlet broke by python 2.7.x http://bugs.python.org/issue22438 opened by alex #22440: Setting SSLContext object's check_hostname manually might acci http://bugs.python.org/issue22440 opened by orsenthil #22441: Not all attributes of the console for a subprocess with creati http://bugs.python.org/issue22441 opened by Sworddragon #22442: subprocess.check_call hangs on large PIPEd data. http://bugs.python.org/issue22442 opened by juj #22443: read(1) blocks on unflushed output http://bugs.python.org/issue22443 opened by Sworddragon Most recent 15 issues with no replies (15) ========================================== #22441: Not all attributes of the console for a subprocess with creati http://bugs.python.org/issue22441 #22435: socketserver.TCPSocket leaks socket to garbage collector if se http://bugs.python.org/issue22435 #22429: asyncio: pending call to loop.stop() if a coroutine raises a B http://bugs.python.org/issue22429 #22425: 2to3 import fixer writes dotted_as_names into import_as_names http://bugs.python.org/issue22425 #22423: Errors in printing exceptions raised in a thread http://bugs.python.org/issue22423 #22422: IDLE closes all when in dropdown menu http://bugs.python.org/issue22422 #22411: Embedding Python on Windows http://bugs.python.org/issue22411 #22397: test_socket failure on AIX http://bugs.python.org/issue22397 #22394: Update documentation building to use venv and pip http://bugs.python.org/issue22394 #22393: multiprocessing.Pool shouldn't hang forever if a worker proces http://bugs.python.org/issue22393 #22392: Clarify documentation of __getinitargs__ http://bugs.python.org/issue22392 #22370: pathlib OS detection http://bugs.python.org/issue22370 #22367: Please add F_OFD_SETLK, etc support to fcntl.lockf http://bugs.python.org/issue22367 #22357: inspect module documentation makes no reference to __qualname_ http://bugs.python.org/issue22357 #22356: mention explicitly that stdlib assumes gmtime(0) epoch is 1970 http://bugs.python.org/issue22356 Most recent 15 issues waiting for review (15) ============================================= #22437: re module: number of named groups is limited to 100 max http://bugs.python.org/issue22437 #22434: Use named constants internally in the re module http://bugs.python.org/issue22434 #22433: Argparse considers unknown optional arguments with spaces as a http://bugs.python.org/issue22433 #22427: TemporaryDirectory attempts to clean up twice http://bugs.python.org/issue22427 #22423: Errors in printing exceptions raised in a thread http://bugs.python.org/issue22423 #22420: Use print(file=sys.stderr) instead of sys.stderr.write() in ID http://bugs.python.org/issue22420 #22418: ipaddress.py new IPv6 Method for Solicited Multicast Address http://bugs.python.org/issue22418 #22417: PEP 476: verify HTTPS certificates by default http://bugs.python.org/issue22417 #22415: Fix re debugging output http://bugs.python.org/issue22415 #22410: Locale dependent regexps on different locales http://bugs.python.org/issue22410 #22407: re.LOCALE is nonsensical for Unicode http://bugs.python.org/issue22407 #22397: test_socket failure on AIX http://bugs.python.org/issue22397 #22396: AIX posix_fadvise and posix_fallocate http://bugs.python.org/issue22396 #22395: test_pathlib error for complex symlinks on Windows http://bugs.python.org/issue22395 #22393: multiprocessing.Pool shouldn't hang forever if a worker proces http://bugs.python.org/issue22393 Top 10 most discussed issues (10) ================================= #22410: Locale dependent regexps on different locales http://bugs.python.org/issue22410 8 msgs #22427: TemporaryDirectory attempts to clean up twice http://bugs.python.org/issue22427 8 msgs #22377: %Z in strptime doesn't match EST and others http://bugs.python.org/issue22377 6 msgs #22426: strptime accepts the wrong '2010-06-01 MSK' string but rejects http://bugs.python.org/issue22426 6 msgs #22434: Use named constants internally in the re module http://bugs.python.org/issue22434 6 msgs #21998: asyncio: a new self-pipe should be created in the child proces http://bugs.python.org/issue21998 5 msgs #22379: Empty exception message of str.join http://bugs.python.org/issue22379 5 msgs #22430: Build failure if configure flags --prefix or --exec-prefix is http://bugs.python.org/issue22430 5 msgs #22437: re module: number of named groups is limited to 100 max http://bugs.python.org/issue22437 5 msgs #22253: ConfigParser does not handle files without sections http://bugs.python.org/issue22253 4 msgs Issues closed (32) ================== #4180: warnings.simplefilter("always") does not make warnings always http://bugs.python.org/issue4180 closed by pitrou #11471: If without else generates redundant jump http://bugs.python.org/issue11471 closed by python-dev #14824: reprlib documentation references string module http://bugs.python.org/issue14824 closed by berker.peksag #16827: Remove the relatively advanced content from section 2 in tutor http://bugs.python.org/issue16827 closed by orsenthil #18159: ConfigParser getters not available on SectionProxy http://bugs.python.org/issue18159 closed by python-dev #20537: logging exc_info parameter should accept exception instances http://bugs.python.org/issue20537 closed by python-dev #20678: re does not allow back references in {} matching operator http://bugs.python.org/issue20678 closed by serhiy.storchaka #21270: unittest.mock.call object has inherited count method http://bugs.python.org/issue21270 closed by python-dev #21391: shutil uses both os.path.abspath and an 'import from' of abspa http://bugs.python.org/issue21391 closed by berker.peksag #21706: Add base for enumerations (Functional API) http://bugs.python.org/issue21706 closed by python-dev #21738: Enum docs claim replacing __new__ is not possible http://bugs.python.org/issue21738 closed by ethan.furman #22043: Use a monotonic clock to compute timeouts http://bugs.python.org/issue22043 closed by haypo #22168: Turtle Graphics RawTurtle problem http://bugs.python.org/issue22168 closed by ned.deily #22220: Ttk extensions test failure http://bugs.python.org/issue22220 closed by serhiy.storchaka #22326: tempfile.TemporaryFile fails on NFS v4 filesystems http://bugs.python.org/issue22326 closed by haypo #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 closed by ethan.furman #22366: urllib.request.urlopen shoudl take a "context" (SSLContext) ar http://bugs.python.org/issue22366 closed by orsenthil #22378: SO_MARK support for Linux http://bugs.python.org/issue22378 closed by neologix #22384: Tk.report_callback_exception kills process when run with pytho http://bugs.python.org/issue22384 closed by serhiy.storchaka #22398: Tools/msi enhancements for 2.7 http://bugs.python.org/issue22398 closed by steve.dower #22399: Doc: missing anchor for dict in library/functions.html http://bugs.python.org/issue22399 closed by georg.brandl #22400: Stable API broken on Windows for PyUnicode_* http://bugs.python.org/issue22400 closed by jkloth #22409: namedtuples bug between 3.3.2 and 3.4.1 http://bugs.python.org/issue22409 closed by binnisb #22412: Towards an asyncio-enabled command line http://bugs.python.org/issue22412 closed by gvanrossum #22414: I'd like to add you to my professional network on LinkedIn http://bugs.python.org/issue22414 closed by georg.brandl #22416: Pickling compiled re patterns http://bugs.python.org/issue22416 closed by gvanrossum #22419: wsgiref request length http://bugs.python.org/issue22419 closed by orsenthil #22421: securing pydoc server http://bugs.python.org/issue22421 closed by orsenthil #22424: make: *** [Objects/unicodeobject.o] Error 1 http://bugs.python.org/issue22424 closed by haypo #22432: function crypt not working on OSX http://bugs.python.org/issue22432 closed by r.david.murray #22436: logging geteffectivelevel does not document its return value http://bugs.python.org/issue22436 closed by python-dev #22439: subprocess.PIPE.stdin.flush() causes to hang while subprocess. http://bugs.python.org/issue22439 closed by r.david.murray From barry at python.org Fri Sep 19 18:09:11 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 19 Sep 2014 12:09:11 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> <20140919150330.GR9293@ando.pearwood.info> <20140919173519.1b9cc88e@fsol> Message-ID: <20140919120911.77f25958@anarchist.wooz.org> On Sep 19, 2014, at 08:40 AM, Guido van Rossum wrote: >Until I say so. Which will happen in the distant future. I'm gonna hid your time machine keys so you didn't find them. -Barry From alex.gaynor at gmail.com Fri Sep 19 18:53:20 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 19 Sep 2014 16:53:20 +0000 (UTC) Subject: [Python-Dev] PEP476: Enabling certificate validation by default Message-ID: Hi all, I've just updated the PEP to reflect the API suggestions from Nick, and the fact that the necessary changes to urllib were landed. I think this is ready for pronouncement, Guido? Cheers, Alex From larry at hastings.org Fri Sep 19 21:24:38 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 19 Sep 2014 20:24:38 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <20140919103146.0d763a24@limelight.wooz.org> References: <20140919103146.0d763a24@limelight.wooz.org> Message-ID: <541C82F6.4040706@hastings.org> Yep. I plan to write it on Monday, at the PyCon UK sprints, right after 3.4.2rc1 goes out. FWIW it'll be 3.4 + 18 months. //arry/ On 09/19/2014 03:31 PM, Barry Warsaw wrote: > Hi Larry, > > I think we need a Python 3.5 Release Schedule PEP. > > Cheers, > -Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 20 00:06:27 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Sep 2014 15:06:27 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: The PEP doesn't specify any of the API changes for Python 2.7. I feel it is necessary for the PEP to show a few typical code snippets using urllib in Python 2.7 and how one would modify these in order to disable the cert checking. There are also a few typos; especially this paragraph puzzled me: This will be acheived by adding a new ``ssl._create_default_https_context`` function, which is the same as ``ssl.create_default``. ``http.client`` can then replace it's usage of ``ssl._create_stdlib_context`` with the new ``ssl._create_default_https_context``. (1) spelling: it's achieved, not achieved (2) method name: it's ssl.create_default_context, not ssl.create_default (3) There's not enough whitespace (in the rendered HTML on legacy.python.org) before http.client -- I kept reading it as "... which is the same as ssl.create_default.http.client ..." (4) There's no mention of the Python 2 equivalent of http.client. Finally, it's kind of non-obvious in the PEP that this affects Python 2.7.X (I guess the one after the next) as well as 3.4 and 3.5. On Fri, Sep 19, 2014 at 9:53 AM, Alex Gaynor wrote: > Hi all, > > I've just updated the PEP to reflect the API suggestions from Nick, and the > fact that the necessary changes to urllib were landed. > > I think this is ready for pronouncement, Guido? > > Cheers, > Alex > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Sat Sep 20 00:34:08 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 19 Sep 2014 15:34:08 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with. On Fri, Sep 19, 2014 at 3:06 PM, Guido van Rossum wrote: > The PEP doesn't specify any of the API changes for Python 2.7. I feel it > is necessary for the PEP to show a few typical code snippets using urllib > in Python 2.7 and how one would modify these in order to disable the cert > checking. > > There are also a few typos; especially this paragraph puzzled me: > > This will be acheived by adding a new ``ssl._create_default_https_context`` > function, which is the same as ``ssl.create_default``. ``http.client`` can > then > replace it's usage of ``ssl._create_stdlib_context`` with the new > ``ssl._create_default_https_context``. > > (1) spelling: it's achieved, not achieved > > (2) method name: it's ssl.create_default_context, not ssl.create_default > > (3) There's not enough whitespace (in the rendered HTML on > legacy.python.org) before http.client -- I kept reading it as "... which > is the same as ssl.create_default.http.client ..." > > (4) There's no mention of the Python 2 equivalent of http.client. > > Finally, it's kind of non-obvious in the PEP that this affects Python > 2.7.X (I guess the one after the next) as well as 3.4 and 3.5. > > On Fri, Sep 19, 2014 at 9:53 AM, Alex Gaynor > wrote: > >> Hi all, >> >> I've just updated the PEP to reflect the API suggestions from Nick, and >> the >> fact that the necessary changes to urllib were landed. >> >> I think this is ready for pronouncement, Guido? >> >> Cheers, >> Alex >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Sep 20 00:44:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Sep 2014 08:44:39 +1000 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> Message-ID: On 20 September 2014 00:23, Donald Stufft wrote: > > On Sep 19, 2014, at 10:16 AM, Barry Warsaw wrote: > > If the user wants to invoke Python 3, it's not hard to type 'python3' and I > think that's the message we should be spreading. That already seems pretty > ingrained in user habits afaict. > > > My biggest problem with ``python3``, is what happens after 3.9. I know Guido > doesn?t particularly like two digit version numbers and it?s been suggested > on > this list that instead of 3.10 we?re likely to move directly into 4.0 > regardless of > if it?s a ?big? change or not. FWIW, I think we actually do this better on Windows these days, where PEP 397 made "py" a switchable entry point. I'd like to bring the same scheme to POSIX systems at some point, but that's a *long* way down the todo list (like, so far down I can't even see it any more). I'd be willing to review proposals, though :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Sep 20 00:52:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Sep 2014 08:52:32 +1000 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: On 20 September 2014 08:34, Alex Gaynor wrote: > Pushed a new version which I believe adresses all of these. I added an > example of opting-out with urllib.urlopen, let me know if there's any other > APIs you think I should show an example with. It would be worth explicitly stating the process global monkeypatching hack: import ssl ssl._create_default_https_context = ssl._create_unverified_context Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts. It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Sep 20 01:13:30 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 19 Sep 2014 16:13:30 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: +1 on Nick's suggestion. (Might also mention that this is the reason why both functions should exist and have compatible signatures.) Also please, please, please add explicit mention of Python 2.7, 3.4 and 3.5 in the Abstract (for example in the 3rd paragraph of the abstract). On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan wrote: > On 20 September 2014 08:34, Alex Gaynor wrote: > > Pushed a new version which I believe adresses all of these. I added an > > example of opting-out with urllib.urlopen, let me know if there's any > other > > APIs you think I should show an example with. > > It would be worth explicitly stating the process global monkeypatching > hack: > > import ssl > ssl._create_default_https_context = ssl._create_unverified_context > > Adding that hack to sitecustomize allows corporate sysadmins that can > update their standard operating environment more easily than they can > fix invalid certificate infrastructure to work around the problem on > behalf of their users. It also helps out users that will be able to > deal with such broken infrastructure without updating each and every > one of their scripts. > > It's deliberately ugly because it's a genuinely bad idea that folks > should want to avoid using, but as a matter of practical reality, > corporate IT departments are chronically understaffed, and often fully > committed to fighting the crisis du jour, without sufficient time > being available for regular infrastructure maintenance tasks. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Sep 20 02:02:22 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Sep 2014 12:02:22 +1200 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> Message-ID: <541CC40E.3060804@canterbury.ac.nz> Donald Stufft wrote: > My biggest problem with ``python3``, is what happens after 3.9. Python2 technically includes 1.x versions as well, so it wouldn't be unprecedented for python3 to imply versions beyond 3.x. It would be a bit confusing, though. -- Greg From donald at stufft.io Sat Sep 20 02:06:26 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 19 Sep 2014 20:06:26 -0400 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <541CC40E.3060804@canterbury.ac.nz> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <541CC40E.3060804@canterbury.ac.nz> Message-ID: <74AA562C-7BCC-472B-A781-54943E1C0710@stufft.io> > > On Sep 19, 2014, at 8:02 PM, Greg Ewing wrote: > > Donald Stufft wrote: > >> My biggest problem with ``python3``, is what happens after 3.9. > > Python2 technically includes 1.x versions as well, so it > wouldn't be unprecedented for python3 to imply versions > beyond 3.x. It would be a bit confusing, though. My problem isn?t that it *includes* it, it?s that either people have to go through the mess to update all of their things to ``python4`` at some point in the future, or Python 3.x will need to eventually mean ``python``. Well that or Python 4.x has a ``python3`` binary. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Sep 20 02:11:53 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Sep 2014 12:11:53 +1200 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <20140919120911.77f25958@anarchist.wooz.org> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <20140919104158.678e2234@limelight.wooz.org> <20140919150330.GR9293@ando.pearwood.info> <20140919173519.1b9cc88e@fsol> <20140919120911.77f25958@anarchist.wooz.org> Message-ID: <541CC649.1070204@canterbury.ac.nz> Barry Warsaw wrote: > On Sep 19, 2014, at 08:40 AM, Guido van Rossum wrote: > >>Until I say so. Which will happen in the distant future. > > I'm gonna hid your time machine keys so you didn't find them. Hiding someone's time machine keys never works. Chances are he's already taken a trip to the future in which you get kidnapped and tortured until you reveal where you hid them, and then nipped over there to take them back. Which means he *might* be able to avoid carrying out the actual torture now, as long as it doesn't create too big a temporal paradox. -- Greg From christian at python.org Sat Sep 20 12:40:06 2014 From: christian at python.org (Christian Heimes) Date: Sat, 20 Sep 2014 12:40:06 +0200 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: <541D5986.7070107@python.org> On 19.09.2014 18:53, Alex Gaynor wrote: > Hi all, > > I've just updated the PEP to reflect the API suggestions from Nick, and the > fact that the necessary changes to urllib were landed. > > I think this is ready for pronouncement, Guido? There is still the issue with SSL_CERT_DIR and SSL_CERT_FILE on Windows and Apple's OpenSSL builds on OSX. I've opened a bug report http://bugs.python.org/issue22449 tl;dr On Windows SSL_CERT_DIR and SSL_CERT_FILE are simply ignored by SSLContext.load_verify_locations. On OSX Apple's Trust Evaluation Agent adds certs behind the scene. From alex.gaynor at gmail.com Sat Sep 20 17:54:56 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 20 Sep 2014 08:54:56 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: Done and done. Alex On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum wrote: > +1 on Nick's suggestion. (Might also mention that this is the reason why > both functions should exist and have compatible signatures.) > > Also please, please, please add explicit mention of Python 2.7, 3.4 and > 3.5 in the Abstract (for example in the 3rd paragraph of the abstract). > > On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan wrote: > >> On 20 September 2014 08:34, Alex Gaynor wrote: >> > Pushed a new version which I believe adresses all of these. I added an >> > example of opting-out with urllib.urlopen, let me know if there's any >> other >> > APIs you think I should show an example with. >> >> It would be worth explicitly stating the process global monkeypatching >> hack: >> >> import ssl >> ssl._create_default_https_context = ssl._create_unverified_context >> >> Adding that hack to sitecustomize allows corporate sysadmins that can >> update their standard operating environment more easily than they can >> fix invalid certificate infrastructure to work around the problem on >> behalf of their users. It also helps out users that will be able to >> deal with such broken infrastructure without updating each and every >> one of their scripts. >> >> It's deliberately ugly because it's a genuinely bad idea that folks >> should want to avoid using, but as a matter of practical reality, >> corporate IT departments are chronically understaffed, and often fully >> committed to fighting the crisis du jour, without sufficient time >> being available for regular infrastructure maintenance tasks. >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 20 18:46:56 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Sep 2014 09:46:56 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: Nice. I just realized the release candidate for 3.4.2 is really close (RC1 Monday, final Oct 6, see PEP 429). What's your schedule for 3.4? I see no date for 2.7.9 yet (but that could just be that PEP 373 hasn't been updated). What about the Apple and Microsoft issues Christian pointed out? Regarding the approval process, I want to get this into 2.7 and 3.4, but I want it done right, and I'm not convinced that the implementation is sufficiently worked out. I don't want you to feel rushed, and I don't want you to feel that you can't start coding until the PEP is approved, but I also feel that I want to see more working code and some beta testing before it goes live. Perhaps I should just approve the PEP but separately get to approve the code? (Others will have to review it for correctness -- but I want to understand and review the API.) On Sat, Sep 20, 2014 at 8:54 AM, Alex Gaynor wrote: > Done and done. > > Alex > > On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum > wrote: > >> +1 on Nick's suggestion. (Might also mention that this is the reason why >> both functions should exist and have compatible signatures.) >> >> Also please, please, please add explicit mention of Python 2.7, 3.4 and >> 3.5 in the Abstract (for example in the 3rd paragraph of the abstract). >> >> On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan wrote: >> >>> On 20 September 2014 08:34, Alex Gaynor wrote: >>> > Pushed a new version which I believe adresses all of these. I added an >>> > example of opting-out with urllib.urlopen, let me know if there's any >>> other >>> > APIs you think I should show an example with. >>> >>> It would be worth explicitly stating the process global monkeypatching >>> hack: >>> >>> import ssl >>> ssl._create_default_https_context = ssl._create_unverified_context >>> >>> Adding that hack to sitecustomize allows corporate sysadmins that can >>> update their standard operating environment more easily than they can >>> fix invalid certificate infrastructure to work around the problem on >>> behalf of their users. It also helps out users that will be able to >>> deal with such broken infrastructure without updating each and every >>> one of their scripts. >>> >>> It's deliberately ugly because it's a genuinely bad idea that folks >>> should want to avoid using, but as a matter of practical reality, >>> corporate IT departments are chronically understaffed, and often fully >>> committed to fighting the crisis du jour, without sufficient time >>> being available for regular infrastructure maintenance tasks. >>> >>> Regards, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Sat Sep 20 19:05:45 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 20 Sep 2014 10:05:45 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: That sounds reasonable to me -- at this point I don't expect this to make it into 3.4.2; Nick has some working code on the ticket: http://bugs.python.org/issue22417 it's mostly missing documentation. Alex On Sat, Sep 20, 2014 at 9:46 AM, Guido van Rossum wrote: > Nice. I just realized the release candidate for 3.4.2 is really close (RC1 > Monday, final Oct 6, see PEP 429). What's your schedule for 3.4? I see no > date for 2.7.9 yet (but that could just be that PEP 373 hasn't been > updated). What about the Apple and Microsoft issues Christian pointed out? > > Regarding the approval process, I want to get this into 2.7 and 3.4, but I > want it done right, and I'm not convinced that the implementation is > sufficiently worked out. I don't want you to feel rushed, and I don't want > you to feel that you can't start coding until the PEP is approved, but I > also feel that I want to see more working code and some beta testing before > it goes live. Perhaps I should just approve the PEP but separately get to > approve the code? (Others will have to review it for correctness -- but I > want to understand and review the API.) > > On Sat, Sep 20, 2014 at 8:54 AM, Alex Gaynor > wrote: > >> Done and done. >> >> Alex >> >> On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum >> wrote: >> >>> +1 on Nick's suggestion. (Might also mention that this is the reason why >>> both functions should exist and have compatible signatures.) >>> >>> Also please, please, please add explicit mention of Python 2.7, 3.4 and >>> 3.5 in the Abstract (for example in the 3rd paragraph of the abstract). >>> >>> On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan >>> wrote: >>> >>>> On 20 September 2014 08:34, Alex Gaynor wrote: >>>> > Pushed a new version which I believe adresses all of these. I added an >>>> > example of opting-out with urllib.urlopen, let me know if there's any >>>> other >>>> > APIs you think I should show an example with. >>>> >>>> It would be worth explicitly stating the process global monkeypatching >>>> hack: >>>> >>>> import ssl >>>> ssl._create_default_https_context = ssl._create_unverified_context >>>> >>>> Adding that hack to sitecustomize allows corporate sysadmins that can >>>> update their standard operating environment more easily than they can >>>> fix invalid certificate infrastructure to work around the problem on >>>> behalf of their users. It also helps out users that will be able to >>>> deal with such broken infrastructure without updating each and every >>>> one of their scripts. >>>> >>>> It's deliberately ugly because it's a genuinely bad idea that folks >>>> should want to avoid using, but as a matter of practical reality, >>>> corporate IT departments are chronically understaffed, and often fully >>>> committed to fighting the crisis du jour, without sufficient time >>>> being available for regular infrastructure maintenance tasks. >>>> >>>> Regards, >>>> Nick. >>>> >>>> -- >>>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> >> >> -- >> "I disapprove of what you say, but I will defend to the death your right >> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) >> "The people's good is the highest law." -- Cicero >> GPG Key fingerprint: 125F 5C67 DFE9 4084 >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Sep 21 00:05:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Sep 2014 08:05:01 +1000 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: On 21 September 2014 03:05, Alex Gaynor wrote: > That sounds reasonable to me -- at this point I don't expect this to make it > into 3.4.2; Nick has some working code on the ticket: > http://bugs.python.org/issue22417 it's mostly missing documentation. I also think it's more sensible to target 2.7.9 & 3.4.3 for this change, especially given the remaining rough edges in custom trust database configuration on WIndows and Mac OS X that Christian pointed out in http://bugs.python.org/issue22449 I don't believe Benjamin has picked a specific date for 2.7.9 yet, but the regular maintenance release cadence (ignoring security releases) would put it some time in November, which should be sufficient time to get the remaining issues ironed out for 3.5 under the normal development process, and then included under the banner of PEP 476 for backporting to the maintenance branches. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robertc at robertcollins.net Sun Sep 21 00:21:47 2014 From: robertc at robertcollins.net (Robert Collins) Date: Sun, 21 Sep 2014 10:21:47 +1200 Subject: [Python-Dev] web-sig mailing list moderating every post? In-Reply-To: References: Message-ID: Ugh - this was in my mailbox shortly after the moderator action email from mailman: "No, this looks like the spam filter. Don't know what triggered it. Or why it went to you. But the list moderation is turned off (except for non-members posting to the list), and you yourself are not moderated, so... Bill" - nothing to see here, move right along, and sorry for the noise. -Rob On 21 September 2014 10:19, Robert Collins wrote: > I'm not sure of the right place to bring this up - I tried to on the > web-sig list itself, but the moderator rejected the post. > > What I tried to post there was > > """Looks like *every* post to web-sig gets manually moderated. That seems > like it will make discussion rather hard: can we get that changed (or > is there some historical need for it - if so, perhaps we should use > python-dev or some other list) ?""" > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -- Robert Collins Distinguished Technologist HP Converged Cloud From guido at python.org Sun Sep 21 00:22:57 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Sep 2014 15:22:57 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: Sounds good. Maybe we should put the specifically targeted releases in PEP 476? Nick, do Christian's issues need to be mentioned in the PEP or should we just keep those in the corresponding tracker items? On Sat, Sep 20, 2014 at 3:05 PM, Nick Coghlan wrote: > On 21 September 2014 03:05, Alex Gaynor wrote: > > That sounds reasonable to me -- at this point I don't expect this to > make it > > into 3.4.2; Nick has some working code on the ticket: > > http://bugs.python.org/issue22417 it's mostly missing documentation. > > I also think it's more sensible to target 2.7.9 & 3.4.3 for this > change, especially given the remaining rough edges in custom trust > database configuration on WIndows and Mac OS X that Christian pointed > out in http://bugs.python.org/issue22449 > > I don't believe Benjamin has picked a specific date for 2.7.9 yet, but > the regular maintenance release cadence (ignoring security releases) > would put it some time in November, which should be sufficient time to > get the remaining issues ironed out for 3.5 under the normal > development process, and then included under the banner of PEP 476 for > backporting to the maintenance branches. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Sun Sep 21 00:19:49 2014 From: robertc at robertcollins.net (Robert Collins) Date: Sun, 21 Sep 2014 10:19:49 +1200 Subject: [Python-Dev] web-sig mailing list moderating every post? Message-ID: I'm not sure of the right place to bring this up - I tried to on the web-sig list itself, but the moderator rejected the post. What I tried to post there was """Looks like *every* post to web-sig gets manually moderated. That seems like it will make discussion rather hard: can we get that changed (or is there some historical need for it - if so, perhaps we should use python-dev or some other list) ?""" -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ncoghlan at gmail.com Sun Sep 21 01:03:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Sep 2014 09:03:39 +1000 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: On 21 September 2014 08:22, Guido van Rossum wrote: > Sounds good. Maybe we should put the specifically targeted releases in PEP > 476? > > Nick, do Christian's issues need to be mentioned in the PEP or should we > just keep those in the corresponding tracker items? They should be mentioned in the PEP, as they will impact the way the proposed change interacts with the platform trust database - I didn't realise the differences on Windows and Mac OS X myself until Christian mentioned them. To be completely independent of the system trust database in a reliable, cross-platform way, folks will need to use a custom SSL context that doesn't enable the system trust store, rather than relying on the OpenSSL config options - the latter will reliably *add* certificates, but they won't reliably ignore the default ones provided by the system. We may also need some clarification from Ned regarding the status of OpenSSL and the potential impact switching from dynamic linking to static linking of OpenSSL may have in terms of the "OPENSSL_X509_TEA_DISABLE" setting. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sun Sep 21 03:53:58 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Sep 2014 18:53:58 -0700 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: OK, I'll hold off a bit on approving the PEP, but my intention is to approve it. Go Alex go! On Sat, Sep 20, 2014 at 4:03 PM, Nick Coghlan wrote: > On 21 September 2014 08:22, Guido van Rossum wrote: > > Sounds good. Maybe we should put the specifically targeted releases in > PEP > > 476? > > > > Nick, do Christian's issues need to be mentioned in the PEP or should we > > just keep those in the corresponding tracker items? > > They should be mentioned in the PEP, as they will impact the way the > proposed change interacts with the platform trust database - I didn't > realise the differences on Windows and Mac OS X myself until Christian > mentioned them. > > To be completely independent of the system trust database in a > reliable, cross-platform way, folks will need to use a custom SSL > context that doesn't enable the system trust store, rather than > relying on the OpenSSL config options - the latter will reliably *add* > certificates, but they won't reliably ignore the default ones provided > by the system. > > We may also need some clarification from Ned regarding the status of > OpenSSL and the potential impact switching from dynamic linking to > static linking of OpenSSL may have in terms of the > "OPENSSL_X509_TEA_DISABLE" setting. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Sun Sep 21 12:57:05 2014 From: christian at python.org (Christian Heimes) Date: Sun, 21 Sep 2014 12:57:05 +0200 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: <541EAF01.6000300@python.org> On 21.09.2014 01:03, Nick Coghlan wrote: > We may also need some clarification from Ned regarding the status of > OpenSSL and the potential impact switching from dynamic linking to > static linking of OpenSSL may have in terms of the > "OPENSSL_X509_TEA_DISABLE" setting. You may want to ask Hynek, too. He initially discovered the issue and made me aware how Apple is tying keychain into OpenSSL. You may find the code in: http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.h http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.c The TEA (Trust Evaluation Agent) switch is global and affects all SSL context instances. There is non canonical way to set the TEA flag for a single context or socket. /* -1: not set * 0: set to false * 1: set to true */ static tea_enabled = -1; void X509_TEA_set_state(int change) { tea_enabled = (change) ? 1 : 0; } int X509_TEA_is_enabled() { if (tea_enabled < 0) tea_enabled = (NULL == getenv(X509_TEA_ENV_DISABLE)); return tea_enabled != 0; } From larry at hastings.org Mon Sep 22 02:26:53 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 22 Sep 2014 01:26:53 +0100 Subject: [Python-Dev] PEP476: Enabling certificate validation by default In-Reply-To: References: Message-ID: <541F6CCD.7070609@hastings.org> On 09/20/2014 11:05 PM, Nick Coghlan wrote: > On 21 September 2014 03:05, Alex Gaynor wrote: >> That sounds reasonable to me -- at this point I don't expect this to make it >> into 3.4.2; Nick has some working code on the ticket: >> http://bugs.python.org/issue22417 it's mostly missing documentation. > I also think it's more sensible to target 2.7.9 & 3.4.3 for this > change, especially given the remaining rough edges in custom trust > database configuration on WIndows and Mac OS X that Christian pointed > out in http://bugs.python.org/issue22449 FWIW, I tagged 3.4.2rc1 about 24 hours ago, so it certainly didn't make it into that. I could live with it going into 3.4.2 final if the community was convinced that was the right thing to do. But I'm more than happy about the conservative turn the plans have taken. If this isn't going in until 3.4.3, does that mean we need 3.4.3 on any sort of particular schedule? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Sep 22 16:15:51 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 22 Sep 2014 15:15:51 +0100 Subject: [Python-Dev] [RELEASE] Python 3.4.2rc1 is now available Message-ID: <54202F17.3010008@hastings.org> On behalf of the Python development community and the Python 3.4 release team, I'm chuffed to announce the availability of Python 3.4.2rc1. Python 3.4.2 has many bugfixes and other small improvements over 3.4.1. One new feature for Mac OS X users: the OS X installers are now distributed as signed installer package files compatible with the OS X Gatekeeper security feature. You can download it here: https://www.python.org/download/releases/3.4.2 Be seeing you, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Sep 22 16:58:50 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 22 Sep 2014 16:58:50 +0200 Subject: [Python-Dev] [python-committers] [RELEASE] Python 3.4.2rc1 is now available In-Reply-To: <54202F17.3010008@hastings.org> References: <54202F17.3010008@hastings.org> Message-ID: Someone broke test_pydoc. Example: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.4/builds/481/steps/test/logs/stdio Victor 2014-09-22 16:15 GMT+02:00 Larry Hastings : > > > On behalf of the Python development community and the Python 3.4 release > team, I'm chuffed to announce the availability of Python 3.4.2rc1. Python > 3.4.2 has many bugfixes and other small improvements over 3.4.1. One new > feature for Mac OS X users: the OS X installers are now distributed as > signed installer package files compatible with the OS X Gatekeeper security > feature. > > You can download it here: > > https://www.python.org/download/releases/3.4.2 > > > Be seeing you, > > > /arry > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > From larry at hastings.org Tue Sep 23 00:12:38 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 22 Sep 2014 23:12:38 +0100 Subject: [Python-Dev] [python-committers] [RELEASE] Python 3.4.2rc1 is now available In-Reply-To: References: <54202F17.3010008@hastings.org> Message-ID: <54209ED6.30608@hastings.org> On 09/22/2014 03:58 PM, Victor Stinner wrote: > Someone broke test_pydoc. Example: > http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.4/builds/481/steps/test/logs/stdio I broke it while making the release. Known bug, happened before, for 3.4.1rc1. http://bugs.python.org/issue21431 We'll get it right for 3.4.2 final. I don't think we need to respin 3.4.2rc1 / add a 3.4.2rc2 for this. On 09/22/2014 06:02 PM, Terry Reedy wrote: > On 9/22/2014 10:15 AM, Larry Hastings wrote: >> You can download it here: >> >> https://www.python.org/download/releases/3.4.2 > Unfortunately, I cannot. This actually links to > https://www.python.org/download/releases/3.4.1 WFM. I'm guessing this was a CDN caching problem--I forgot to manually purge the cache. Does it work for you now? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Sep 23 02:20:04 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 23 Sep 2014 01:20:04 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <20140919103146.0d763a24@limelight.wooz.org> References: <20140919103146.0d763a24@limelight.wooz.org> Message-ID: <5420BCB4.1080007@hastings.org> On 09/19/2014 03:31 PM, Barry Warsaw wrote: > I think we need a Python 3.5 Release Schedule PEP. Just checked it in as PEP 478. It should show up here in a few minutes: http://legacy.python.org/dev/peps/pep-0478/ Key facts: * Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. * Final release is September 13, 2015, just over a year from now. Comments? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Sep 23 02:22:28 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 22 Sep 2014 20:22:28 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5420BCB4.1080007@hastings.org> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> Message-ID: <25668161-BD2D-420D-A1A0-59482B22B1D8@stufft.io> > On Sep 22, 2014, at 8:20 PM, Larry Hastings wrote: > > > On 09/19/2014 03:31 PM, Barry Warsaw wrote: >> I think we need a Python 3.5 Release Schedule PEP. > > Just checked it in as PEP 478. It should show up here in a few minutes: > http://legacy.python.org/dev/peps/pep-0478/ > > Key facts: > Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. > Final release is September 13, 2015, just over a year from now. > > Comments? It says 3.4.0 all through it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 23 11:46:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 23 Sep 2014 19:46:32 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5420BCB4.1080007@hastings.org> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> Message-ID: On 23 September 2014 10:20, Larry Hastings wrote: > > On 09/19/2014 03:31 PM, Barry Warsaw wrote: > > I think we need a Python 3.5 Release Schedule PEP. > > > Just checked it in as PEP 478. It should show up here in a few minutes: > > http://legacy.python.org/dev/peps/pep-0478/ Thanks. Some updates on specific things either already accepted, or under consideration for inclusion: Already implemented: PEP 465 (matrix multiplication operator): http://www.python.org/dev/peps/pep-0465/ Standard streams default to the "surrogateescape" error handler if ASCII is reported as the IO encoding (improves tolerance of the POSIX C locale) Already accepted: PEP 461 (binary printf-style formatting): http://www.python.org/dev/peps/pep-0461/ PEP 471 (os.scandir): http://www.python.org/dev/peps/pep-0471/ Under consideration (in addition to the items already listed in the PEP): PEP 432 (simplifying the startup sequence) PEP 475 (retry system calls failing with EINTR) Improved Windows console Unicode support (see https://pypi.python.org/pypi/win_unicode_console for details) Changing the encoding and error handling of an existing stream (http://bugs.python.org/issue15216) (432 was previously listed as deferred, but it went back on my todo list once 3.4 was out the door) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 23 12:04:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 23 Sep 2014 20:04:56 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> Message-ID: On 23 September 2014 19:46, Nick Coghlan wrote: > Under consideration (in addition to the items already listed in the PEP): > > PEP 432 (simplifying the startup sequence) > PEP 475 (retry system calls failing with EINTR) > Improved Windows console Unicode support (see > https://pypi.python.org/pypi/win_unicode_console for details) > Changing the encoding and error handling of an existing stream > (http://bugs.python.org/issue15216) A few more specific Unicode related proposals that may be worth mentioning in the release PEP (aside from the binary data formatting mini-language one, they're generally not PEP worthy in their own right, but they do relate directly to addressing some of the remaining concerns with binary data handling in Python 3, which is a fairly high profile topic): Allowing "backslashreplace" to be used on input: http://bugs.python.org/issue22286 Adding "codecs.convert_surrogateescape": http://bugs.python.org/issue18814 Adding "wsgiref.util.dump_wsgistr" and "wsgiref.util.load_wsgistr": http://bugs.python.org/issue22264 Adding "bytes.hex", "bytearray.hex" and "memoryview.hex": http://bugs.python.org/issue9951 Adding a binary data formatting mini-language (depends on 9951): http://bugs.python.org/issue22385 There's also your own PEP 457 to formalise the syntax used to communicate signature information from Argument Clinic to the inspect module as a string - I don't know if you were planning to revisit that for 3.5 and potentially make creating inspect.Signature objects from such strings a public API. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From willingc at willingconsulting.com Tue Sep 23 15:56:29 2014 From: willingc at willingconsulting.com (Carol Willing) Date: Tue, 23 Sep 2014 06:56:29 -0700 Subject: [Python-Dev] Python 3.4.2rc1 Mac OS X Message-ID: <54217C0D.90400@willingconsulting.com> To all who contributed to Mac OS X improvements: Thank you! I downloaded and installed last night on Mavericks 10.9. It was quick, straightforward, and completed in seconds. For contrast, I talked several new users at an Intro to Python workshop this past weekend how to install 3.4.1; it took about 5 minutes to guide each user through installing 3.4.1. The new 3.4.2rc1 installer is a *big* improvement. It also seems as if launching the interpreter from the command line is also faster now :) -- *Carol Willing* Developer | Willing Consulting +1 760 456 9366 | https://willingconsulting.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Tue Sep 23 19:05:38 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 23 Sep 2014 17:05:38 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5420BCB4.1080007@hastings.org> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> Message-ID: <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Larry Hastings wrote: > > On 09/19/2014 03:31 PM, Barry Warsaw wrote: > I think we need a Python 3.5 Release Schedule PEP. > > Just checked it in as PEP 478.? It should show up here in a few minutes: > http://legacy.python.org/dev/peps/pep-0478/ > > Key facts: > . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. > . Final release is September 13, 2015, just over a year from now. > > Comments? Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet. I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... Thoughts, anyone? Cheers, Steve > /arry From nad at acm.org Tue Sep 23 19:27:11 2014 From: nad at acm.org (Ned Deily) Date: Tue, 23 Sep 2014 10:27:11 -0700 Subject: [Python-Dev] Python 3.4.2rc1 Mac OS X References: <54217C0D.90400@willingconsulting.com> Message-ID: In article <54217C0D.90400 at willingconsulting.com>, Carol Willing wrote: > [...] The new 3.4.2rc1 installer is a *big* improvement. Thanks for the feedback, Carol. Let us know via bugs.python.org of any issues you see. BTW, the new installer format will be coming to Python 2.7.9 as well. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Wed Sep 24 03:48:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Sep 2014 11:48:22 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On 24 September 2014 03:05, Steve Dower wrote: > Larry Hastings wrote: >> >> On 09/19/2014 03:31 PM, Barry Warsaw wrote: >> I think we need a Python 3.5 Release Schedule PEP. >> >> Just checked it in as PEP 478. It should show up here in a few minutes: >> http://legacy.python.org/dev/peps/pep-0478/ >> >> Key facts: >> . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. >> . Final release is September 13, 2015, just over a year from now. >> >> Comments? > > Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. > > I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet. > > I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). > > I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. > > So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. > > For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... > > Thoughts, anyone? It's ultimately up to Larry as RM, but I'd personally favour targeting the newer compiler and runtime, even with the slight risk of potentially needing to slip our schedule. There's also a fair amount of wiggle room between the first beta and the first release candidate. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Sep 24 03:50:46 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 23 Sep 2014 21:50:46 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <81F8B543-837A-42AD-9A15-D755745E451F@stufft.io> > On Sep 23, 2014, at 9:48 PM, Nick Coghlan wrote: > > On 24 September 2014 03:05, Steve Dower > wrote: >> Larry Hastings wrote: >>> >>> On 09/19/2014 03:31 PM, Barry Warsaw wrote: >>> I think we need a Python 3.5 Release Schedule PEP. >>> >>> Just checked it in as PEP 478. It should show up here in a few minutes: >>> http://legacy.python.org/dev/peps/pep-0478/ >>> >>> Key facts: >>> . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. >>> . Final release is September 13, 2015, just over a year from now. >>> >>> Comments? >> >> Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. >> >> I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet. >> >> I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). >> >> I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. >> >> So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. >> >> For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... >> >> Thoughts, anyone? > > It's ultimately up to Larry as RM, but I'd personally favour targeting > the newer compiler and runtime, even with the slight risk of > potentially needing to slip our schedule. There's also a fair amount > of wiggle room between the first beta and the first release candidate. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io This new compiler has the incredibly awesome feature of being forwards compatible right? Like in 10 years stuff compiled with a newer compiler will still work? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Sep 24 04:18:28 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 23 Sep 2014 22:18:28 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <4b8245e3c2da4d34bba37f6b6d4d9c03@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <, <81F8B543-837A-42AD-9A15-D755745E451F@stufft.io> <>> <4b8245e3c2da4d34bba37f6b6d4d9c03@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <894AE3E7-A2E3-44A8-A8E4-6B759F6D243A@stufft.io> > On Sep 23, 2014, at 10:14 PM, Steve Dower wrote: > > "This new compiler has the incredibly awesome feature of being forwards compatible > right? Like in 10 years stuff compiled with a newer compiler will still work?" > > That's the promise at least :) > > All the macros that leaked implementation details (like file descriptors) are now isolated so if they change it won't break existing applications. It'll still be possible for newer compiler versions to break them, but the design now obviously discourages it. There's also an official guarantee, so if it is broken in future it'll be treated as a bug. As much as I'd love to make solid promises, I can only pass on the promises that were made to me. > > But yes, we should have forward compatibility with later MSVC versions, which will help avoid the situation where it's hard to get hold of the right compiler... > > Top-posted from my Windows Phone Yea I think it makes incredible sense to aim for it then, even if it makes things slip. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Wed Sep 24 04:14:45 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 24 Sep 2014 02:14:45 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <81F8B543-837A-42AD-9A15-D755745E451F@stufft.io> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> , <81F8B543-837A-42AD-9A15-D755745E451F@stufft.io> Message-ID: <4b8245e3c2da4d34bba37f6b6d4d9c03@DM2PR0301MB0734.namprd03.prod.outlook.com> "This new compiler has the incredibly awesome feature of being forwards compatible right? Like in 10 years stuff compiled with a newer compiler will still work?" That's the promise at least :) All the macros that leaked implementation details (like file descriptors) are now isolated so if they change it won't break existing applications. It'll still be possible for newer compiler versions to break them, but the design now obviously discourages it. There's also an official guarantee, so if it is broken in future it'll be treated as a bug. As much as I'd love to make solid promises, I can only pass on the promises that were made to me. But yes, we should have forward compatibility with later MSVC versions, which will help avoid the situation where it's hard to get hold of the right compiler... Top-posted from my Windows Phone ________________________________ From: Donald Stufft Sent: ?9/?23/?2014 18:50 To: Nick Coghlan Cc: Steve Dower; python-dev at python.org Subject: Re: [Python-Dev] 3.5 release schedule PEP On Sep 23, 2014, at 9:48 PM, Nick Coghlan > wrote: On 24 September 2014 03:05, Steve Dower > wrote: Larry Hastings wrote: On 09/19/2014 03:31 PM, Barry Warsaw wrote: I think we need a Python 3.5 Release Schedule PEP. Just checked it in as PEP 478. It should show up here in a few minutes: http://legacy.python.org/dev/peps/pep-0478/ Key facts: . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. . Final release is September 13, 2015, just over a year from now. Comments? Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet. I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... Thoughts, anyone? It's ultimately up to Larry as RM, but I'd personally favour targeting the newer compiler and runtime, even with the slight risk of potentially needing to slip our schedule. There's also a fair amount of wiggle room between the first beta and the first release candidate. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io This new compiler has the incredibly awesome feature of being forwards compatible right? Like in 10 years stuff compiled with a newer compiler will still work? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Wed Sep 24 07:13:46 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 24 Sep 2014 06:13:46 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <5422530A.4070507@timgolden.me.uk> On 23/09/2014 18:05, Steve Dower wrote: > I'm also considering/experimenting with installing into "Program > Files" by default, but I suspect that isn't going to work out yet. I'd like to see that go forward: I think it's increasingly difficult to justify Python's position at c:\pythonxx. But it does run the risk of breaking All The Things. When it was discussed, people spoke about symlinks (or hardlinks / junctions / whatever) in the appropriate places to support hardcoded paths, but I don't know how feasible that will turn out to be. > > I'd like to move the Windows versions onto the next release of VC > (currently "VC 14" until the branding team figures out what to call > it). There isn't a promised RTM date for VC 14 yet, so it looks like > the best available compiler by Beta 1 will be a "Go Live" RC. (The > "Go Live" marking basically means "we think this is ready for use, > but expect a round of minor updates/fixes soon - the compiler is > least likely to be updated, my guess is that it'll be Visual Studio > UI mostly). My only real misgiving here is that, for a few years, we'll need *three* versions installed to build the active branches: 2008 for 2.7; 2010 for 3.4; and 2014 for 3.5. Am I wrong? TJG From ncoghlan at gmail.com Wed Sep 24 09:12:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Sep 2014 17:12:35 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5422530A.4070507@timgolden.me.uk> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422530A.4070507@timgolden.me.uk> Message-ID: On 24 Sep 2014 15:15, "Tim Golden" wrote: > > On 23/09/2014 18:05, Steve Dower wrote: >> >> I'm also considering/experimenting with installing into "Program >> Files" by default, but I suspect that isn't going to work out yet. > > > I'd like to see that go forward: I think it's increasingly difficult to justify Python's position at c:\pythonxx. But it does run the risk of > breaking All The Things. When it was discussed, people spoke about > symlinks (or hardlinks / junctions / whatever) in the appropriate > places to support hardcoded paths, but I don't know how feasible > that will turn out to be. It might be better to offer that as a supported option in 3.5, and then make it the default in 3.6. That will offer a couple of years to work out the issues, rather than a few months. >> I'd like to move the Windows versions onto the next release of VC >> (currently "VC 14" until the branding team figures out what to call >> it). There isn't a promised RTM date for VC 14 yet, so it looks like >> the best available compiler by Beta 1 will be a "Go Live" RC. (The >> "Go Live" marking basically means "we think this is ready for use, >> but expect a round of minor updates/fixes soon - the compiler is >> least likely to be updated, my guess is that it'll be Visual Studio >> UI mostly). > > > My only real misgiving here is that, for a few years, we'll need *three* versions installed to build the active branches: 2008 for 2.7; 2010 for 3.4; and 2014 for 3.5. Am I wrong? 3.4 will go into security fix only mode a few months after 3.5 is released, as happened with 3.3. It does suggest that we shouldn't switch the compiler until: 1. At least a beta of VC 2014 is generally available 2. We're a few weeks out from 3.5 beta 1 (with VC 2014 as an option until then) Cheers, Nick. > > TJG > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Sep 24 11:04:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Sep 2014 10:04:15 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5422530A.4070507@timgolden.me.uk> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422530A.4070507@timgolden.me.uk> Message-ID: On 24 September 2014 06:13, Tim Golden wrote: > My only real misgiving here is that, for a few years, we'll need *three* > versions installed to build the active branches: 2008 for 2.7; 2010 for 3.4; > and 2014 for 3.5. Am I wrong? Also, will 2014 express edition be able to fully build extensions for Python, 32 and 64 bit? Will it be able to build Python itself? I ask because I'm currently discovering how much fun (basically none) it is to set up the SDK compilers for building extensions, and it would be fantastic if that pain point could be removed. (It's also personally relevant to me now, as my python-dev provided MSDN subscription has run out, so I'm back to only having access to free tools :-)) Paul From mal at egenix.com Wed Sep 24 12:56:20 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 24 Sep 2014 12:56:20 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <5422A354.1020801@egenix.com> On 24.09.2014 03:48, Nick Coghlan wrote: > On 24 September 2014 03:05, Steve Dower wrote: >> Larry Hastings wrote: >>> >>> On 09/19/2014 03:31 PM, Barry Warsaw wrote: >>> I think we need a Python 3.5 Release Schedule PEP. >>> >>> Just checked it in as PEP 478. It should show up here in a few minutes: >>> http://legacy.python.org/dev/peps/pep-0478/ >>> >>> Key facts: >>> . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. >>> . Final release is September 13, 2015, just over a year from now. >>> >>> Comments? >> >> Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. >> >> I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet. >> >> I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). >> >> I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. >> >> So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. >> >> For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... >> >> Thoughts, anyone? > > It's ultimately up to Larry as RM, but I'd personally favour targeting > the newer compiler and runtime, even with the slight risk of > potentially needing to slip our schedule. There's also a fair amount > of wiggle room between the first beta and the first release candidate. I'd rather be conservative here and wait for another Python release before switching VC versions. There are a few important questions that need answers before we can consider a new VC version: * Will there be free versions available ? * Will those free editions include the 64-bit compilers ? * Will those free editions include the optimizing compilers ? * Is there a roadmap for how long these free versions will remain officially available ? * Are there issues compiling 3rd party libraries with it ? E.g. the numeric and science stacks, the web stacks, the deployment stacks, etc. * What license terms will the new version have ? E.g. GPL compatibility issues, weird exceptions, * What will the pricing structure look like ? While core devs will get free MSDN licenses, most other 3rd party providers will have to buy licenses for the compiler, unless they can use the free versions. An alternative would be targeting VC13 instead of VC14, in case it has good answers for the above questions. It's been around for a year now, so there should be more experience available with this version. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 24 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-09-27: PyDDF Sprint 2014 ... 3 days to go 2014-09-30: Python Meeting Duesseldorf ... 6 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Wed Sep 24 14:34:25 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Sep 2014 14:34:25 +0200 Subject: [Python-Dev] 3.5 release schedule PEP References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422530A.4070507@timgolden.me.uk> Message-ID: <20140924143425.6c67c557@fsol> On Wed, 24 Sep 2014 17:12:35 +1000 Nick Coghlan wrote: > On 24 Sep 2014 15:15, "Tim Golden" wrote: > > > > On 23/09/2014 18:05, Steve Dower wrote: > >> > >> I'm also considering/experimenting with installing into "Program > >> Files" by default, but I suspect that isn't going to work out yet. > > > > > > I'd like to see that go forward: I think it's increasingly difficult to > justify Python's position at c:\pythonxx. But it does run the risk of > > breaking All The Things. When it was discussed, people spoke about > > symlinks (or hardlinks / junctions / whatever) in the appropriate > > places to support hardcoded paths, but I don't know how feasible > > that will turn out to be. > > It might be better to offer that as a supported option in 3.5, and then > make it the default in 3.6. I would be surprised if this weren't already a supported option. Decent Windows installers generally allow you to override the installation path (it's been a while I haven't used the Windows Python installer, sorry). I think making the move in 3.5 would be a good idea. Experts can override the installation path and choose C:\PythonXY. There's no actual breakage since the path changes for every major release, anyway (i.e. people would have had to change the path from "C:\Python34" to "C:\Python35"; instead, they will have to change it to "C:\Program Files\Python35"). Regards Antoine. From victor.stinner at gmail.com Wed Sep 24 17:04:33 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Sep 2014 17:04:33 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <25668161-BD2D-420D-A1A0-59482B22B1D8@stufft.io> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <25668161-BD2D-420D-A1A0-59482B22B1D8@stufft.io> Message-ID: 2014-09-23 2:22 GMT+02:00 Donald Stufft : <> I think we need a Python 3.5 Release Schedule PEP. >> >> Just checked it in as PEP 478. It should show up here in a few minutes: >> >> http://legacy.python.org/dev/peps/pep-0478/ >> Comments? > > It says 3.4.0 all through it. It was too distrubing to read "3.4" in the "3.5" schedule. I modified the PEP directly, sorry Larry. Victor From python-dev at mgmiller.net Wed Sep 24 15:16:48 2014 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 25 Sep 2014 01:16:48 +1200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: Message-ID: <5422C440.2020001@mgmiller.net> On 09/24/2014 10:00 PM, Nick Coghlan wrote: > Subject: Re: [Python-Dev] 3.5 release schedule PEP > > On 24 Sep 2014 15:15, "Tim Golden" wrote: > > > > On 23/09/2014 18:05, Steve Dower wrote: > >> I'm also considering/experimenting with installing into "Program > >> Files" by default, but I suspect that isn't going to work out yet. > > > > I'd like to see that go forward: I think it's increasingly difficult to > justify Python's position at c:\pythonxx. But it does run the risk of > > breaking All The Things. > > It might be better to offer that as a supported option in 3.5, and then make it > the default in 3.6. > > That will offer a couple of years to work out the issues, rather than a few months. Hi all, ProgramFiles was the default in Python 1.X. It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue. I've never once encountered a bug due to install to ProgramFiles, or heard of anyone who has, and have been using Python for everything since the year 2000. If any rare bugs happen to surface, they can likely be fixed or replaced with a line of code, or worked around by installing elsewhere. The potential existence of such bugs isn't enough reason to stay stuck in 1990 while leaving users vulnerable to tampering attacks for another few years. -Mike From p.f.moore at gmail.com Wed Sep 24 21:01:09 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Sep 2014 20:01:09 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5422C440.2020001@mgmiller.net> References: <5422C440.2020001@mgmiller.net> Message-ID: On 24 September 2014 14:16, Mike Miller wrote: > It has been a supported option for just shy of 15 years on 2.X... most if > not all the bugs (setuptools) were fixed a decade ago, and right now > thousands, if not millions of people are running it under Program Files > right now. I can vouch for several thousand because a company I work for > distributes Python and pip there for its customers all around the world w/o > issue. One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt. Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use). I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch. Paul From victor.stinner at gmail.com Wed Sep 24 21:32:52 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Sep 2014 21:32:52 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> Message-ID: Most Windows setup are desktop configured with a single user. I would not be shocked if pip installs modules only for the current user by default. Maybe it could be an option in Python installer (pip system wide or user). Victor Le mercredi 24 septembre 2014, Paul Moore a ?crit : > On 24 September 2014 14:16, Mike Miller > wrote: > > It has been a supported option for just shy of 15 years on 2.X... most > if > > not all the bugs (setuptools) were fixed a decade ago, and right now > > thousands, if not millions of people are running it under Program Files > > right now. I can vouch for several thousand because a company I work for > > distributes Python and pip there for its customers all around the world > w/o > > issue. > > One thing that I presume would be an issue. Isn't Program Files > protected in newer versions of Windows? I haven't tested this myself, > so I may be wrong about this. So take the following with a pinch of > salt. > > Assuming so, that means that if Python is installed there, the > standard "pip install XXX" would not work unless run in an elevated > shell. We are currently trying to focus on a unified message for how > users should install distributions from PyPI, by using pip install. > I'm not sure it's a good idea to complicate that message yet by adding > provisos about managing the system Python (which is the only one most > Windows users will use). > > I know this is only the same situation as Unix users have, but Windows > users don't have a distro offering packaged versions of PyPI modules. > I also know we should be moving towards promoting --user, but I don't > think we're quite ready for that yet. And my speculation doesn't > compete with your real-life experience, certainly. But I would suggest > carefully checking before making the switch. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Sep 24 22:05:17 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Sep 2014 22:05:17 +0200 Subject: [Python-Dev] 3.5 release schedule PEP References: <5422C440.2020001@mgmiller.net> Message-ID: <20140924220517.7fc23e82@fsol> On Wed, 24 Sep 2014 21:32:52 +0200 Victor Stinner wrote: > Most Windows setup are desktop configured with a single user. I would not > be shocked if pip installs modules only for the current user by default. > Maybe it could be an option in Python installer (pip system wide or user). pip install --user From Steve.Dower at microsoft.com Wed Sep 24 22:24:14 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 24 Sep 2014 20:24:14 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> Message-ID: <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> > Paul Moore wrote: > On 24 September 2014 14:16, Mike Miller wrote: >> It has been a supported option for just shy of 15 years on 2.X... >> most if not all the bugs (setuptools) were fixed a decade ago, and >> right now thousands, if not millions of people are running it under >> Program Files right now. I can vouch for several thousand because a >> company I work for distributes Python and pip there for its customers >> all around the world w/o issue. > > One thing that I presume would be an issue. Isn't Program Files protected in > newer versions of Windows? I haven't tested this myself, so I may be wrong about > this. So take the following with a pinch of salt. It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning. > Assuming so, that means that if Python is installed there, the standard "pip > install XXX" would not work unless run in an elevated shell. We are currently > trying to focus on a unified message for how users should install distributions > from PyPI, by using pip install. > I'm not sure it's a good idea to complicate that message yet by adding > provisos about managing the system Python (which is the only one most Windows > users will use). This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not. There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite. > I know this is only the same situation as Unix users have, but Windows users > don't have a distro offering packaged versions of PyPI modules. > I also know we should be moving towards promoting --user, but I don't think > we're quite ready for that yet. And my speculation doesn't compete with your > real-life experience, certainly. But I would suggest carefully checking before > making the switch. A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6. Cheers, Steve > Paul From donald at stufft.io Wed Sep 24 22:43:42 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 24 Sep 2014 16:43:42 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> > On Sep 24, 2014, at 4:24 PM, Steve Dower wrote: > >> Paul Moore wrote: >> On 24 September 2014 14:16, Mike Miller wrote: >>> It has been a supported option for just shy of 15 years on 2.X... >>> most if not all the bugs (setuptools) were fixed a decade ago, and >>> right now thousands, if not millions of people are running it under >>> Program Files right now. I can vouch for several thousand because a >>> company I work for distributes Python and pip there for its customers >>> all around the world w/o issue. >> >> One thing that I presume would be an issue. Isn't Program Files protected in >> newer versions of Windows? I haven't tested this myself, so I may be wrong about >> this. So take the following with a pinch of salt. > > It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning. > >> Assuming so, that means that if Python is installed there, the standard "pip >> install XXX" would not work unless run in an elevated shell. We are currently >> trying to focus on a unified message for how users should install distributions >> from PyPI, by using pip install. >> I'm not sure it's a good idea to complicate that message yet by adding >> provisos about managing the system Python (which is the only one most Windows >> users will use). > > This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not. > > There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite. > >> I know this is only the same situation as Unix users have, but Windows users >> don't have a distro offering packaged versions of PyPI modules. >> I also know we should be moving towards promoting --user, but I don't think >> we're quite ready for that yet. And my speculation doesn't compete with your >> real-life experience, certainly. But I would suggest carefully checking before >> making the switch. > > A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6. One thing about *nix is even though you can?t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Wed Sep 24 22:50:30 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 24 Sep 2014 20:50:30 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> Message-ID: <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> Donald Stufft wrote: > One thing about *nix is even though you can?t write to your normal Python > install location without root, invoking pip with permissions (assuming you have > them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have > an equivalent or do you need to launch a whole new shell? Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings. I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users. Cheers, Steve From mal at egenix.com Wed Sep 24 23:00:53 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 24 Sep 2014 23:00:53 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422A354.1020801@egenix.com> Message-ID: <54233105.5020106@egenix.com> Thanks for the insights, Steve. More below... On 24.09.2014 18:52, Steve Dower wrote: > M.-A. Lemburg wrote: >> >> I'd rather be conservative here and wait for another Python release before >> switching VC versions. There are a few important questions that need answers >> before we can consider a new VC version: >> >> * Will there be free versions available ? >> >> * Will those free editions include the 64-bit compilers ? >> >> * Will those free editions include the optimizing compilers ? >> >> * Is there a roadmap for how long these free versions will remain >> officially available ? >> >> * Are there issues compiling 3rd party libraries with it ? >> >> E.g. the numeric and science stacks, the web stacks, >> the deployment stacks, etc. >> >> * What license terms will the new version have ? >> >> E.g. GPL compatibility issues, weird exceptions, >> >> * What will the pricing structure look like ? >> >> While core devs will get free MSDN licenses, most other 3rd party >> providers will have to buy licenses for the compiler, unless >> they can use the free versions. >> >> An alternative would be targeting VC13 instead of VC14, in case it has good >> answers for the above questions. It's been around for a year now, so there >> should be more experience available with this version. > > (Nit - it's actually VC12 a.k.a. "Visual Studio 2013" - VC13 was skipped. This is what happens when you have separate engineering and marketing teams :) ) Ah, ok :-) > I don't have good answers to all of these yet, but none of them are going to be any worse than for VC12. I've forwarded these questions to the people on the VC team who do get to choose the answers, and while I'm not expecting to hear specifics back from them, they are at least aware of the concerns and how important their product is to our community. > > There will be free versions available, but I don't know what format they'll be in. Those free editions should include identical compilers to the paid ones - the cases where that hasn't been true have been bugs or due to assumptions that were proven to be incorrect. > > The main improvement in this version is that all versions from VC14 should be binary compatible, and so there will always be a free compiler, but it may be VC15/16/etc. and not VC14. That's good news. > There are certainly issues with 3rd party libraries, largely because all projects have a tendency to take dependencies on compiler/library internals. OpenSSL, for example, redefines the stdout/in/err macros based on the VC version, but the new definitions are no longer valid with VC14, and so they are fixing that. Python itself has a few issues that I have already fixed in my branch. There will certainly be other issues, but an advantage of starting early is that bugs in the compiler itself can be fixed in the compiler. > > The license should not change significantly from previous versions. GPL incompatibilities are because the GPL wants to be incompatible with licenses based on different ideologies - AFAIK there's never been anything in the VC licenses preventing whatever redistribution license you like. As example: there once was a special clause which explicitly disallowed "Excluded License[s]" to be used together with VC redistibutable files. I think this is no longer the case, but there may be new things in the EULAs. > Part of my improvements to /PCBuild will help avoid the need for Visual Studio entirely, but the free versions should always be sufficient for building and debugging. I have no insight or control over the pricing structure. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 24 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-09-27: PyDDF Sprint 2014 ... 3 days to go 2014-09-30: Python Meeting Duesseldorf ... 6 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Steve.Dower at microsoft.com Wed Sep 24 23:29:25 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 24 Sep 2014 21:29:25 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5423354E.10408@mgmiller.net> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> Message-ID: > Mike Miller wrote: > Paul Moore wrote: >>> One thing that I presume would be an issue. Isn't Program Files >>> protected in newer versions of Windows? > > Yes, that's the feature that protects from malicious users/code editing "import > os" to run "format c:\", spam your address book, or look for credit card > numbers, etc. > > It is the same on Mac and other Unix systems, even Windows since Vista came out, > almost 10 years ago. > >> >> This is my main concern. Until pip install --user is the default (or >> the fallback if there are no write permissions on the destination), >>> real-life experience, certainly. But I would suggest carefully >>> checking before making the switch. > > Windows users know how to elevate now, especially developer types. Sure, but there are plenty of not-quite developer types using Python, plenty using locked down machines, and plenty who simply don't trust arbitrary code when it wants to elevate. The ability to become an admin does not preclude the need to support non-admin users. > It should be billed as a "feature for your protection" not something to be > feared. Microsoft decided security was worth the pain of changing over its > billions of users. Why not Python? > > In my experience pip --user works just fine also. We use it on our unmanned > media players successfully. This is good to know. Maybe we aren't as far away as we think. > We also write Windows services, which run under a system account, where it is > imperative everything is running from a secure file system. > >> A good reason to decide early on a change like this, >> or at least to promote it as an option in 3.5 and make it the default in 3.6. > > It's already an option! It always has been --> Custom install. I can't remember > a time when it didn't work perfectly. Agreed. I mean it'll become an obvious option (one of the radio buttons at the start), and eventually the default option. As a slight aside, do you enable the option to compile PYC/PYO files on install? Unless your users are running as admin, you won't get caching on these for the stdlib when it's installed into Program Files. Cheers, Steve > -Mike From python-dev at mgmiller.net Wed Sep 24 23:19:10 2014 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 25 Sep 2014 09:19:10 +1200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <5423354E.10408@mgmiller.net> Paul Moore wrote: >> One thing that I presume would be an issue. Isn't Program Files protected in >> newer versions of Windows? Yes, that's the feature that protects from malicious users/code editing "import os" to run "format c:\", spam your address book, or look for credit card numbers, etc. It is the same on Mac and other Unix systems, even Windows since Vista came out, almost 10 years ago. > > This is my main concern. Until pip install --user is the default > (or the fallback if there are no write permissions on the destination), >> real-life experience, certainly. But I would suggest carefully checking before >> making the switch. Windows users know how to elevate now, especially developer types. It should be billed as a "feature for your protection" not something to be feared. Microsoft decided security was worth the pain of changing over its billions of users. Why not Python? In my experience pip --user works just fine also. We use it on our unmanned media players successfully. We also write Windows services, which run under a system account, where it is imperative everything is running from a secure file system. > A good reason to decide early on a change like this, > or at least to promote it as an option in 3.5 and make it the default in 3.6. It's already an option! It always has been --> Custom install. I can't remember a time when it didn't work perfectly. -Mike From python-dev at mgmiller.net Wed Sep 24 23:23:04 2014 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 25 Sep 2014 09:23:04 +1200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> Message-ID: <54233638.8050104@mgmiller.net> On 09/25/2014 08:43 AM, Donald Stufft wrote: > > One thing about *nix is even though you can?t write to your normal Python > install location without root, invoking pip with permissions (assuming you have > them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have > an equivalent or do you need to launch a whole new shell? There is the "runas" command, though it is a tad harder to use. There is a third-party windows sudo command also that works as you'd expect. -Mike From python-dev at mgmiller.net Wed Sep 24 23:30:27 2014 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 25 Sep 2014 09:30:27 +1200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <542337F3.6000109@mgmiller.net> On 09/25/2014 08:50 AM, Steve Dower wrote: > Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings. There is the runas command, though it could be easier to use. There is also a third-part "sudo" clone. > I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users. Shoul mention this won't take any choices away from users, such as IT managed environs, they could still install wherever they need to. I understand pip is not actually bundled with Python, just an installer script. That means it is decoupled from changes in the Python installer. Pypa can have pip default to, or fall back to --user later if it decides it is a good idea. -Mike From p.f.moore at gmail.com Wed Sep 24 23:56:20 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Sep 2014 22:56:20 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> Message-ID: On 24 September 2014 22:29, Steve Dower wrote: >> In my experience pip --user works just fine also. We use it on our unmanned >> media players successfully. > > This is good to know. Maybe we aren't as far away as we think. Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice. Reports like this are great for confirming that it's a viable approach. Paul From solipsis at pitrou.net Thu Sep 25 03:08:24 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Sep 2014 03:08:24 +0200 Subject: [Python-Dev] 3.5 release schedule PEP References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> Message-ID: <20140925030824.1654310f@fsol> On Wed, 24 Sep 2014 22:56:20 +0100 Paul Moore wrote: > On 24 September 2014 22:29, Steve Dower wrote: > >> In my experience pip --user works just fine also. We use it on our unmanned > >> media players successfully. > > > > This is good to know. Maybe we aren't as far away as we think. > > Indeed. Moving towards having --user as the norm is definitely > something we want to look at for pip. One of the biggest concerns is > how well-exercised the whole user site directory area is in practice. What do you mean by well-exercised? Regards Antoine. From Steve.Dower at microsoft.com Wed Sep 24 18:52:46 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 24 Sep 2014 16:52:46 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5422A354.1020801@egenix.com> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422A354.1020801@egenix.com> Message-ID: M.-A. Lemburg wrote: > > I'd rather be conservative here and wait for another Python release before > switching VC versions. There are a few important questions that need answers > before we can consider a new VC version: > > * Will there be free versions available ? > > * Will those free editions include the 64-bit compilers ? > > * Will those free editions include the optimizing compilers ? > > * Is there a roadmap for how long these free versions will remain > officially available ? > > * Are there issues compiling 3rd party libraries with it ? > > E.g. the numeric and science stacks, the web stacks, > the deployment stacks, etc. > > * What license terms will the new version have ? > > E.g. GPL compatibility issues, weird exceptions, > > * What will the pricing structure look like ? > > While core devs will get free MSDN licenses, most other 3rd party > providers will have to buy licenses for the compiler, unless > they can use the free versions. > > An alternative would be targeting VC13 instead of VC14, in case it has good > answers for the above questions. It's been around for a year now, so there > should be more experience available with this version. (Nit - it's actually VC12 a.k.a. "Visual Studio 2013" - VC13 was skipped. This is what happens when you have separate engineering and marketing teams :) ) I don't have good answers to all of these yet, but none of them are going to be any worse than for VC12. I've forwarded these questions to the people on the VC team who do get to choose the answers, and while I'm not expecting to hear specifics back from them, they are at least aware of the concerns and how important their product is to our community. There will be free versions available, but I don't know what format they'll be in. Those free editions should include identical compilers to the paid ones - the cases where that hasn't been true have been bugs or due to assumptions that were proven to be incorrect. The main improvement in this version is that all versions from VC14 should be binary compatible, and so there will always be a free compiler, but it may be VC15/16/etc. and not VC14. There are certainly issues with 3rd party libraries, largely because all projects have a tendency to take dependencies on compiler/library internals. OpenSSL, for example, redefines the stdout/in/err macros based on the VC version, but the new definitions are no longer valid with VC14, and so they are fixing that. Python itself has a few issues that I have already fixed in my branch. There will certainly be other issues, but an advantage of starting early is that bugs in the compiler itself can be fixed in the compiler. The license should not change significantly from previous versions. GPL incompatibilities are because the GPL wants to be incompatible with licenses based on different ideologies - AFAIK there's never been anything in the VC licenses preventing whatever redistribution license you like. Part of my improvements to /PCBuild will help avoid the need for Visual Studio entirely, but the free versions should always be sufficient for building and debugging. I have no insight or control over the pricing structure. Cheers, Steve > -- > Marc-Andre Lemburg > eGenix.com From chris.barker at noaa.gov Wed Sep 24 18:42:29 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 24 Sep 2014 09:42:29 -0700 Subject: [Python-Dev] Python 3.4.2rc1 Mac OS X In-Reply-To: References: <54217C0D.90400@willingconsulting.com> Message-ID: On Tue, Sep 23, 2014 at 10:27 AM, Ned Deily wrote: > Thanks for the feedback, Carol. Let us know via bugs.python.org of any > issues you see. BTW, the new installer format will be coming to Python > 2.7.9 as well. > Are the supported platforms going to be the same? i.e.: 10.3+ -- Intel32+PPC32 (Or is this deprecated already?) 10.6+ -- Intel32_Intel64 I'm actually wondering if it's time for a pure Intel64, probably 10.7+.... Building extensions for Universal builds and older OS versions really is a pain... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Sep 25 03:59:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Sep 2014 11:59:36 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote: > Donald Stufft wrote: >> One thing about *nix is even though you can?t write to your normal Python >> install location without root, invoking pip with permissions (assuming you have >> them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have >> an equivalent or do you need to launch a whole new shell? > > Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings. When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root. ChrisA From python-dev at mgmiller.net Thu Sep 25 03:56:05 2014 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 25 Sep 2014 13:56:05 +1200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> Message-ID: <54237635.7000400@mgmiller.net> Yes, we enable the compile script. As we require admin rights to install Python and system (not user) modules with pip, the stdlib .pycs do get created under ProgramFiles at install time. There might well be a situation where a system pipped module doesn't get compiled, but to be honest the difference is negligible on modern machines, those needed to run a supported Windows. I suppose this issue could be improved in the future by caching unwritable system .pyc in a user folder, however it feels like a micro-optimization that shouldn't hold up the change. -Mike On 09/25/2014 09:29 AM, Steve Dower wrote: > As a slight aside, do you enable the option to compile PYC/PYO files on install? Unless your users are running as admin, you won't get caching on these for the stdlib when it's installed into Program Files. > > Cheers, > Steve > >> -Mike From v+python at g.nevcal.com Thu Sep 25 05:42:12 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 24 Sep 2014 20:42:12 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <54238F14.1010306@g.nevcal.com> On 9/24/2014 6:59 PM, Chris Angelico wrote: > That is, could pip > defer the declaration until it's parsed its command line args and > decided that it'll be installing into the system directory, but NOT > make that declaration if it's given --user, or if it's running inside > a venv, or anything else that means it doesn't need that power? Other programs do it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Sep 25 06:11:41 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 24 Sep 2014 21:11:41 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <542395FD.1090703@hastings.org> On 09/23/2014 06:48 PM, Nick Coghlan wrote: > On 24 September 2014 03:05, Steve Dower wrote: >> I'd like to move the Windows versions onto the next release of VC (currently "VC 14" until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). > It's ultimately up to Larry as RM, but I'd personally favour targeting > the newer compiler and runtime, even with the slight risk of > potentially needing to slip our schedule. There's also a fair amount > of wiggle room between the first beta and the first release candidate. First, let me say that I'm absolutely willing to listen to the choir of expert modern Windows developers. I haven't done much of anything with Windows since 2007, and I don't claim to be current. So if I'm being wrong-headed on this, you're invited to educate me. Also, having a retooled CRTL with forwards-compatible interfaces indeed sounds awesome, and is a worthy goal. With all that said: I'm comfortable shipping everything up to RCs on a beta compiler. But once we hit RC1, I assert we *must* be using a released product. Release Candidates are supposed to be viable, releasable software, and surely we wouldn't ship 3.5.0 on a beta compiler. Therefore: if VC14 doesn't ship by 3.5 RC1, currently set at August 5, 2015, I decree we have to ship 3.5 with the previous version. Reasonable? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Sep 25 06:12:21 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 24 Sep 2014 21:12:21 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <25668161-BD2D-420D-A1A0-59482B22B1D8@stufft.io> Message-ID: <54239625.5030201@hastings.org> On 09/24/2014 08:04 AM, Victor Stinner wrote: > It was too distrubing to read "3.4" in the "3.5" schedule. I modified > the PEP directly, sorry Larry. No sweat. I would have fixed it myself, but yesterday was a big travel day. Thanks for fixing it! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Sep 25 08:34:31 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 25 Sep 2014 07:34:31 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <20140925030824.1654310f@fsol> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> Message-ID: On 25 September 2014 02:08, Antoine Pitrou wrote: >> Indeed. Moving towards having --user as the norm is definitely >> something we want to look at for pip. One of the biggest concerns is >> how well-exercised the whole user site directory area is in practice. > > What do you mean by well-exercised? Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. There's probably no way to improve that much other than by making --user the default and waiting for reports of any issues, but feedback like Mike's adds a certain level of confidence that there are no significant problems. Paul From songofacandy at gmail.com Thu Sep 25 09:22:23 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 25 Sep 2014 16:22:23 +0900 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> Message-ID: FYI, homebrew's Python uses prefix option, so I can't use `--user`. Is it a bug? $ /usr/local/bin/pip -V pip 1.5.6 from /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg (python 2.7) $ /usr/local/bin/pip install --user tornado ... error: can't combine user with prefix, exec_prefix/home, or install_(plat)base $ cat /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/distutils.cfg [global] verbose=1 [install] force=1 prefix=/usr/local On Thu, Sep 25, 2014 at 3:34 PM, Paul Moore wrote: > On 25 September 2014 02:08, Antoine Pitrou wrote: >>> Indeed. Moving towards having --user as the norm is definitely >>> something we want to look at for pip. One of the biggest concerns is >>> how well-exercised the whole user site directory area is in practice. >> >> What do you mean by well-exercised? > > Basically, although --user is available in pip (and the underlying > facilities in Python have been around for some time), it's difficult > to gauge how many people are using them, and as a result what level of > testing has happened in real-life situations. There's probably no way > to improve that much other than by making --user the default and > waiting for reports of any issues, but feedback like Mike's adds a > certain level of confidence that there are no significant problems. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki From solipsis at pitrou.net Thu Sep 25 10:54:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Sep 2014 10:54:35 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> Message-ID: <20140925105435.66bc53dc@fsol> On Thu, 25 Sep 2014 07:34:31 +0100 Paul Moore wrote: > On 25 September 2014 02:08, Antoine Pitrou wrote: > >> Indeed. Moving towards having --user as the norm is definitely > >> something we want to look at for pip. One of the biggest concerns is > >> how well-exercised the whole user site directory area is in practice. > > > > What do you mean by well-exercised? > > Basically, although --user is available in pip (and the underlying > facilities in Python have been around for some time), it's difficult > to gauge how many people are using them, and as a result what level of > testing has happened in real-life situations. I'm using it often. I'm also unsure how broken it could be. The user site-packages is just another site-packages directory. Regards Antoine. From antoine at python.org Thu Sep 25 11:44:32 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 25 Sep 2014 11:44:32 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> Message-ID: <5423E400.40008@python.org> Le 25/09/2014 09:22, INADA Naoki a ?crit : > FYI, homebrew's Python uses prefix option, so I can't use `--user`. > Is it a bug? > > $ /usr/local/bin/pip -V > pip 1.5.6 from /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg > (python 2.7) > > $ /usr/local/bin/pip install --user tornado > ... > error: can't combine user with prefix, exec_prefix/home, or install_(plat)base Yes, it is. http://bugs.python.org/issue22269 Regards Antoine. From ncoghlan at gmail.com Thu Sep 25 12:17:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Sep 2014 20:17:12 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <5422C440.2020001@mgmiller.net> References: <5422C440.2020001@mgmiller.net> Message-ID: On 24 September 2014 23:16, Mike Miller wrote: > Hi all, > > ProgramFiles was the default in Python 1.X. > > It has been a supported option for just shy of 15 years on 2.X... most if > not all the bugs (setuptools) were fixed a decade ago, and right now > thousands, if not millions of people are running it under Program Files > right now. I can vouch for several thousand because a company I work for > distributes Python and pip there for its customers all around the world w/o > issue. > > I've never once encountered a bug due to install to ProgramFiles, or heard > of anyone who has, and have been using Python for everything since the year > 2000. If any rare bugs happen to surface, they can likely be fixed or > replaced with a line of code, or worked around by installing elsewhere. Right, the problem I was thinking of was actually a transient one during the 3.4 pre-release cycle (the ensurepip integration initially didn't work properly when installing into Program Files). So consider that a switch to +1 from me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu Sep 25 17:43:59 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 25 Sep 2014 11:43:59 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <20140925105435.66bc53dc@fsol> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> Message-ID: <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> > On Sep 25, 2014, at 4:54 AM, Antoine Pitrou wrote: > > On Thu, 25 Sep 2014 07:34:31 +0100 > Paul Moore wrote: >> On 25 September 2014 02:08, Antoine Pitrou wrote: >>>> Indeed. Moving towards having --user as the norm is definitely >>>> something we want to look at for pip. One of the biggest concerns is >>>> how well-exercised the whole user site directory area is in practice. >>> >>> What do you mean by well-exercised? >> >> Basically, although --user is available in pip (and the underlying >> facilities in Python have been around for some time), it's difficult >> to gauge how many people are using them, and as a result what level of >> testing has happened in real-life situations. > > I'm using it often. I'm also unsure how broken it could be. The user > site-packages is just another site-packages directory. > Broken like the prefix problem :) Basically people have Python in a ton of different configurations and it?s hard to figure out if ?user will work out of the box in all of them or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Sep 25 17:54:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 25 Sep 2014 16:54:35 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> Message-ID: On 25 September 2014 16:43, Donald Stufft wrote: > Basically people have Python in a ton of different configurations and it?s > hard to figure out if ?user will work out of the box in all of them or not. I guess that "Using the python.org Python installer on Windows" is a limited enough subset that we probably could check that --user worked in that situation. The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in "Program Files" should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it). Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable. Paul From donald at stufft.io Thu Sep 25 18:00:33 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 25 Sep 2014 12:00:33 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> Message-ID: <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> > On Sep 25, 2014, at 11:54 AM, Paul Moore wrote: > > On 25 September 2014 16:43, Donald Stufft wrote: >> Basically people have Python in a ton of different configurations and it?s >> hard to figure out if ?user will work out of the box in all of them or not. > > I guess that "Using the python.org Python installer on Windows" is a > limited enough subset that we probably could check that --user worked > in that situation. > > The problem is, how do we implement it? A special case so that pip > defaults to --user sometimes, but not others? (I'm strongly against > that) Leave the default as not --user and document that Windows users > with Python in "Program Files" should always specify --user? (I'm > against that because it makes the documentation highly confusing, and > we've just done a lot of work to simplify it). > > Basically, I'd like to hold off moving to "Program Files" as a default > until *after* we have enough confidence in user installs that we are > willing to switch pip to --user as the default behaviour everywhere. > And yes, I'm aware that the first "we" in that was "python-dev" and > the second was "PyPA". And that expecting python-dev to wait for PyPA > to approve the change may well be unacceptable. > > Paul My thoughts on the pip side has basically always been that pip should either: 1) Just always default to ?user and add a ?system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window. 2) Switch to ?user based on if the user has permission to write to the site-packages or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Thu Sep 25 18:05:36 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 25 Sep 2014 16:05:36 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <0c358dfe0fe4465b9ce9671d412931c9@DM2PR0301MB0734.namprd03.prod.outlook.com> Chris Angelico wrote: > On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote: >> Donald Stufft wrote: >>> One thing about *nix is even though you can?t write to your normal >>> Python install location without root, invoking pip with permissions >>> (assuming you have >>> them) is as easy as prefacing it with ``sudo`` in most cases. Does >>> Windows have an equivalent or do you need to launch a whole new shell? >> >> Unfortunately not. The "easy way" is for the executable to declare that it > needs administrative privileges, and then the OS will take over and let you > approve/reject/sign-in/etc. according to your settings. > > When you say the executable declares this, is that a static (compile/link time) > declaration, or a run-time one? That is, could pip defer the declaration until > it's parsed its command line args and decided that it'll be installing into the > system directory, but NOT make that declaration if it's given --user, or if it's > running inside a venv, or anything else that means it doesn't need that power? > If so, that could actually be a very powerful feature: a user without admin > privs (or choosing not to exercise them) is still able to install into a venv, > but if s/he forgets to activate the venv, the "hey, I want to modify system > files" prompt will be the alarm that says the situation is not what was > expected. That's what happens on my Linux system - I get errors back if I don't > use sudo, but in a venv, everything works without being root. It's a compile time option (it's specified in the manifest you can embed into an executable) but it is possible to explicitly launch a separate process with the flag. Most programs that look like they can elevate on-demand are really launching a background process to do the work - you can't change the user for a process once it's started in Windows. This process can obviously be the same one you started with (maybe with different command line args), and the main observable difference is that you get fewer access denied errors. So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :) Cheers, Steve > ChrisA > From p.f.moore at gmail.com Thu Sep 25 19:01:44 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 25 Sep 2014 18:01:44 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <0c358dfe0fe4465b9ce9671d412931c9@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> <0c358dfe0fe4465b9ce9671d412931c9@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On 25 September 2014 17:05, Steve Dower wrote: > So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :) I thought one issue with running an elevated command line subprocess from a non-elevated one, was that the elevated one didn't have access to the non-elevated console, so it popped up its own independent console window, which disappeared immediately the process completed (hence losing any error messages). I definitely recall easy_install did that at one stage, and it was a real pain. Or is that something the parent process can affect, and the cmd/easy_install pair just didn't do so? Paul From Steve.Dower at microsoft.com Thu Sep 25 19:13:25 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 25 Sep 2014 17:13:25 +0000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> <0c358dfe0fe4465b9ce9671d412931c9@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: > Paul Moore wrote: > On 25 September 2014 17:05, Steve Dower wrote: >> So yes, pip can certainly do this, and if it's already running >> elevated then it shouldn't reprompt, but it's not entirely trivial to >> get this right ("are you denied write access to that directory because >> you're not admin or because it's on read-only media?") and it's >> considerably easier to try it, fail on access issues, but provide a >> flag for the user to force elevation. "pip --sudo install ..." would >> be fine by me :) > > I thought one issue with running an elevated command line subprocess from a > non-elevated one, was that the elevated one didn't have access to the > non-elevated console, so it popped up its own independent console window, which > disappeared immediately the process completed (hence losing any error messages). > I definitely recall easy_install did that at one stage, and it was a real pain. > Or is that something the parent process can affect, and the cmd/easy_install > pair just didn't do so? I'm not sure you can do it automatically, but if you own both sides of the application you can set up a pipe between the two processes and let the unelevated side forward stdio. Again, this isn't trivial to get right. The design for the elevation model seems to have focused mainly on GUI rather than console, probably assuming that people who need to elevate from the console will elevate the shell itself (this is a guess - I have no insight into how the Windows team designed this), so there are limitations we have to work within. > Paul From p.f.moore at gmail.com Thu Sep 25 19:44:16 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 25 Sep 2014 18:44:16 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <60D853CA-3601-4C47-B94C-BA884EDD90F1@stufft.io> <81258fc1fe88460799e1b7d2460c58c5@DM2PR0301MB0734.namprd03.prod.outlook.com> <0c358dfe0fe4465b9ce9671d412931c9@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On 25 September 2014 18:13, Steve Dower wrote: > Again, this isn't trivial to get right. The design for the elevation model seems to have focused mainly on GUI rather than console, probably assuming that people who need to elevate from the console will elevate the shell itself (this is a guess - I have no insight into how the Windows team designed this), so there are limitations we have to work within. Yeah, I think my view is that we don't really want to have to support code that complex, so let's keep it simple. To use pip to install into a protected location, pip must be run from an elevated shell. Given that doing so is inconvenient (specifically, even more inconvenient than sudo on Unix) let's ensure that for the normal case the user doesn't need to do that. Paul From martin at v.loewis.de Thu Sep 25 19:45:28 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 25 Sep 2014 19:45:28 +0200 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <20140924143425.6c67c557@fsol> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <5422530A.4070507@timgolden.me.uk> <20140924143425.6c67c557@fsol> Message-ID: <542454B8.7000308@v.loewis.de> Am 24.09.14 14:34, schrieb Antoine Pitrou: > On Wed, 24 Sep 2014 17:12:35 +1000 > Nick Coghlan wrote: >> On 24 Sep 2014 15:15, "Tim Golden" wrote: >>> >>> On 23/09/2014 18:05, Steve Dower wrote: >>>> >>>> I'm also considering/experimenting with installing into "Program >>>> Files" by default, but I suspect that isn't going to work out yet. >>> >> It might be better to offer that as a supported option in 3.5, and then >> make it the default in 3.6. > > I would be surprised if this weren't already a supported option. Decent > Windows installers generally allow you to override the installation > path (it's been a while I haven't used the Windows Python installer, > sorry). I regularly test whether it installs properly into Program Files (although regularly actually means "for every major release"). Also, users regularly report if it doesn't work. As it turns out, 3.4 indeed got a new problem when installing into Program Files, namely that pip would not install. As discussed elsewhere in this thread, installation into Program Files requires elevated privileges, which meant that I had to reschedule the pip installation procedure within the MSI to run in the system context. > > I think making the move in 3.5 would be a good idea. Experts can > override the installation path and choose C:\PythonXY. There's no > actual breakage since the path changes for every major release, anyway > (i.e. people would have had to change the path from "C:\Python34" to > "C:\Python35"; instead, they will have to change it to "C:\Program > Files\Python35"). I expect breakage elsewhere (as also discussed): - some code will fail because of the space in the path - some installations will fail because of the restricted access to Program Files (which, of course, is also the main reason why people want that install location) It's of course now up to Steve to rule on the default install location; if he changes it, he will get lots of both praise and blame, if he doesn't change it, he will get a little blame for continuing the tradition. Regards, Martin From ethan at stoneleaf.us Thu Sep 25 20:14:30 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 25 Sep 2014 11:14:30 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <542395FD.1090703@hastings.org> References: <20140919103146.0d763a24@limelight.wooz.org> <5420BCB4.1080007@hastings.org> <5b5e50a7d12f45549956bcdbfeb1ba89@DM2PR0301MB0734.namprd03.prod.outlook.com> <542395FD.1090703@hastings.org> Message-ID: <54245B86.1030305@stoneleaf.us> On 09/24/2014 09:11 PM, Larry Hastings wrote: > > Therefore: if VC14 doesn't ship by 3.5 RC1, currently set at August 5, 2015, I decree we have to ship 3.5 with the > previous version. > > Reasonable? Seems reasonable to me. -- ~Ethan~ From d2mp1a9 at newsguy.com Thu Sep 25 22:00:16 2014 From: d2mp1a9 at newsguy.com (Bob Hanson) Date: Thu, 25 Sep 2014 13:00:16 -0700 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX Message-ID: Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX: Also see for thread on same being started today. --Bob Hanson From solipsis at pitrou.net Fri Sep 26 00:17:46 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Sep 2014 00:17:46 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: Message-ID: <20140926001746.10743a46@fsol> On Thu, 25 Sep 2014 13:00:16 -0700 Bob Hanson wrote: > Critical bash vulnerability CVE-2014-6271 may affect Python on > *n*x and OSX: > > > > > > > > > > Also see for thread on > same being started today. Fortunately, Python's subprocess has its `shell` argument default to False. However, `os.system` invokes the shell implicitly and is therefore a possible attack vector. Regards Antoine. From ncoghlan at gmail.com Fri Sep 26 00:44:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Sep 2014 08:44:46 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> Message-ID: On 26 Sep 2014 01:56, "Paul Moore" wrote: > > Basically, I'd like to hold off moving to "Program Files" as a default > until *after* we have enough confidence in user installs that we are > willing to switch pip to --user as the default behaviour everywhere. > And yes, I'm aware that the first "we" in that was "python-dev" and > the second was "PyPA". And that expecting python-dev to wait for PyPA > to approve the change may well be unacceptable. This sounds like a reasonable plan to me (I've been keen to have pip do this on the Linux side of things for a while). If the pip team were willing to start down this path, changing the default in 3.5 next year might still be cutting things a little fine. While I think it's worth aiming for that, I also wouldn't see it as a big deal if the change slipped to 3.6 in 2017. Regards, Nick. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Sep 26 00:59:17 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 25 Sep 2014 17:59:17 -0500 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926001746.10743a46@fsol> References: <20140926001746.10743a46@fsol> Message-ID: This was helpful: http://sarge.readthedocs.org/en/latest/internals.html#how-shell-quoting-works -- Wes Turner On Thu, Sep 25, 2014 at 5:17 PM, Antoine Pitrou wrote: > On Thu, 25 Sep 2014 13:00:16 -0700 > Bob Hanson wrote: >> Critical bash vulnerability CVE-2014-6271 may affect Python on >> *n*x and OSX: >> >> >> >> >> >> >> >> >> >> Also see for thread on >> same being started today. > > Fortunately, Python's subprocess has its `shell` argument default to > False. However, `os.system` invokes the shell implicitly and is > therefore a possible attack vector. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com From cs at zip.com.au Fri Sep 26 01:14:37 2014 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 26 Sep 2014 09:14:37 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926001746.10743a46@fsol> References: <20140926001746.10743a46@fsol> Message-ID: <20140925231437.GA84686@cskk.homeip.net> On 26Sep2014 00:17, Antoine Pitrou wrote: >On Thu, 25 Sep 2014 13:00:16 -0700 >Bob Hanson wrote: >> Critical bash vulnerability CVE-2014-6271 may affect Python on >> *n*x and OSX: >> [...] >Fortunately, Python's subprocess has its `shell` argument default to >False. However, `os.system` invokes the shell implicitly and is >therefore a possible attack vector. Only if /bin/sh is bash :-) Not always the case, fortunately. Cheers, Cameron Simpson Death is life's way of telling you you've been fired. - R. Geis From steve at pearwood.info Fri Sep 26 01:40:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Sep 2014 09:40:17 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926001746.10743a46@fsol> References: <20140926001746.10743a46@fsol> Message-ID: <20140925234016.GA19757@ando.pearwood.info> On Fri, Sep 26, 2014 at 12:17:46AM +0200, Antoine Pitrou wrote: > On Thu, 25 Sep 2014 13:00:16 -0700 > Bob Hanson wrote: > > Critical bash vulnerability CVE-2014-6271 may affect Python on > > *n*x and OSX: [...] See also: http://adminlogs.info/2014/09/25/again-bash-cve-2014-7169/ > Fortunately, Python's subprocess has its `shell` argument default to > False. However, `os.system` invokes the shell implicitly and is > therefore a possible attack vector. Perhaps I'm missing something, but aren't there easier ways to attack os.system than the bash env vulnerability? If I'm accepting and running arbitrary strings from an untrusted user, there's no need for them to go to the trouble of feeding me: "env x='() { :;}; echo gotcha' bash -c 'echo do something useful'" when they can just feed me: "echo gotcha" In other words, os.system is *already* an attack vector, unless you only use it with trusted strings. I don't think the bash env vulnerability adds to the attack surface. Have I missed something? -- Steven From chris.barker at noaa.gov Fri Sep 26 00:44:57 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 25 Sep 2014 15:44:57 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft wrote: > 1) Just always default to ?user and add a ?system or similar flag, this > is super easy to change but is a backwards incompatible change and > would need to go through a deprecation window. > Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window. > 2) Switch to ?user based on if the user has permission to write to the > site-packages or not. > ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like: """ pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install """ "In the face of ambiguity, refuse the temptation to guess." -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Sep 26 01:53:22 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Sep 2014 01:53:22 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> Message-ID: <20140926015322.4399c47e@fsol> On Fri, 26 Sep 2014 09:40:17 +1000 Steven D'Aprano wrote: > Perhaps I'm missing something, but aren't there easier ways to attack > os.system than the bash env vulnerability? If I'm accepting and running > arbitrary strings from an untrusted user, there's no need for them to go > to the trouble of feeding me: > > "env x='() { :;}; echo gotcha' bash -c 'echo do something useful'" > > when they can just feed me: > > "echo gotcha" > > In other words, os.system is *already* an attack vector, unless you only > use it with trusted strings. I don't think the bash env vulnerability > adds to the attack surface. > > Have I missed something? The part where the attack payload is passed through the environment, not through hypothetical user-injected command-line arguments. Regards Antoine. From wes.turner at gmail.com Fri Sep 26 02:10:22 2014 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 25 Sep 2014 19:10:22 -0500 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926015322.4399c47e@fsol> References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> <20140926015322.4399c47e@fsol> Message-ID: > The part where the attack payload is passed through the environment, not through hypothetical user-injected command-line arguments. So, best advice would be to: 1. Upgrade bash (and standby for an additional patch, according to reddit) 2. Upgrade to WSGI, if possible 3. Be careful about including things from os.environ? 3. Always avoid allowing user input with os.system, os.popen, and subprocess.xyz(cmd, shell=True) [which doesn't need "#aftershock"] Am I missing something? seeAlso: * http://cwe.mitre.org/top25/#CWE-78 "Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')" * http://cwe.mitre.org/data/definitions/78.html * https://www.owasp.org/index.php/Top_10_2013-A1-Injection * https://en.wikipedia.org/wiki/Code_injection#Shell_injection On Thu, Sep 25, 2014 at 6:53 PM, Antoine Pitrou wrote: > On Fri, 26 Sep 2014 09:40:17 +1000 > Steven D'Aprano wrote: >> Perhaps I'm missing something, but aren't there easier ways to attack >> os.system than the bash env vulnerability? If I'm accepting and running >> arbitrary strings from an untrusted user, there's no need for them to go >> to the trouble of feeding me: >> >> "env x='() { :;}; echo gotcha' bash -c 'echo do something useful'" >> >> when they can just feed me: >> >> "echo gotcha" >> >> In other words, os.system is *already* an attack vector, unless you only >> use it with trusted strings. I don't think the bash env vulnerability >> adds to the attack surface. >> >> Have I missed something? > > The part where the attack payload is passed through the environment, not > through hypothetical user-injected command-line arguments. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com -- Wes Turner https://westurner.github.io/ From rosuav at gmail.com Fri Sep 26 02:23:26 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Sep 2014 10:23:26 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926015322.4399c47e@fsol> References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> <20140926015322.4399c47e@fsol> Message-ID: On Fri, Sep 26, 2014 at 9:53 AM, Antoine Pitrou wrote: >> In other words, os.system is *already* an attack vector, unless you only >> use it with trusted strings. I don't think the bash env vulnerability >> adds to the attack surface. >> >> Have I missed something? > > The part where the attack payload is passed through the environment, not > through hypothetical user-injected command-line arguments. Which means this also affects anything that invokes shell scripts (if they use bash, rather than sh), even if it doesn't use os.system(). I'm just in process of checking and patching my systems (most of them are just 'apt-get update; apt-get upgrade' followed by a quick check), and can confirm that it does happen in Python. All you have to do is invoke bash, either explicitly or through your target's shebang. >>> import os, subprocess >>> os.environ["HAHA"]="() { :;}; echo This is crafted from the environment." >>> subprocess.call(["./test.sh"]) This is crafted from the environment. This is from my test script. 0 >>> open("./test.sh").read() '#!/bin/bash\necho This is from my test script.\n\n' >>> subprocess.call(["bash","-c","echo This is from the command line."]) This is crafted from the environment. This is from the command line. 0 But this is a bash issue, not a Python one. ChrisA From jeanpierreda at gmail.com Fri Sep 26 02:29:23 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 25 Sep 2014 17:29:23 -0700 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926015322.4399c47e@fsol> References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> <20140926015322.4399c47e@fsol> Message-ID: On Thu, Sep 25, 2014 at 4:53 PM, Antoine Pitrou wrote: >> In other words, os.system is *already* an attack vector, unless you only >> use it with trusted strings. I don't think the bash env vulnerability >> adds to the attack surface. >> >> Have I missed something? > > The part where the attack payload is passed through the environment, not > through hypothetical user-injected command-line arguments. As I understand it, if the attacker can help specify the environment (e.g. this is a CGI script), and you run os.system('echo hi'), you can get pwned. Even safe uses of os.system are vulnerable unless you point /bin/sh at a secure shell (e.g. patched bash). -- Devin From rosuav at gmail.com Fri Sep 26 02:33:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Sep 2014 10:33:07 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> <20140926015322.4399c47e@fsol> Message-ID: On Fri, Sep 26, 2014 at 10:29 AM, Devin Jeanpierre wrote: > As I understand it, if the attacker can help specify the environment > (e.g. this is a CGI script), and you run os.system('echo hi'), you can > get pwned. Even safe uses of os.system are vulnerable unless you point > /bin/sh at a secure shell (e.g. patched bash). /bin/sh may well not point to bash anyway - it doesn't on any of my systems. Debian provides dash instead, much faster than bash. But if you're invoking a script that calls for bash, then it's vulnerable. ChrisA From donald at stufft.io Fri Sep 26 02:38:05 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 25 Sep 2014 20:38:05 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: > On Sep 25, 2014, at 6:44 PM, Chris Barker wrote: > > On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft > wrote: > 1) Just always default to ?user and add a ?system or similar flag, this > is super easy to change but is a backwards incompatible change and > would need to go through a deprecation window. > > Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window. > > 2) Switch to ?user based on if the user has permission to write to the > site-packages or not. > > ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like: > > """ > pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install > """ > > "In the face of ambiguity, refuse the temptation to guess.? I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else. Immediately switching to --user as a default (outside of a virtual environment) would break two important use cases: - A root/admin user attempting to install into a global site-packages - A user who uses complete Python installations to isolate themselves such as those created by pyenv. It would techincally break anyone relying on the fact that pip will currently throw an error if you attempt to ``pip install`` something without sudo, however I do not consider that to be an actual use case that is going to have wide signifcance. So we get down to how do we not break the two important use cases above. At first I thought about detecting uid 0 (and something similar on Windows?) and turning off the --user default in those cases. However that still left us dealing with something for the second use case. After thinking about it I realized there isn't a good way to determine if something is a user controlled Python installation instead of a system controlled one. This lead me to the realization that something that could be used to "detect" this is whether or not the current user has the ability to write to the site-packages as a means of determining "is my current user allowed to manage that particular Python". Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too). However this particular thing is somewhat offtopic for this list and the particulars of pip moving to --user by default is best discussed on our issue tracker here: https://github.com/pypa/pip/issues/1668. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Fri Sep 26 02:59:54 2014 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 26 Sep 2014 10:59:54 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140925234016.GA19757@ando.pearwood.info> References: <20140925234016.GA19757@ando.pearwood.info> Message-ID: <20140926005954.GA63336@cskk.homeip.net> On 26Sep2014 09:40, Steven D'Aprano wrote: >On Fri, Sep 26, 2014 at 12:17:46AM +0200, Antoine Pitrou wrote: >> On Thu, 25 Sep 2014 13:00:16 -0700 >> Bob Hanson wrote: >> > Critical bash vulnerability CVE-2014-6271 may affect Python on >> > *n*x and OSX: >[...] > >See also: >http://adminlogs.info/2014/09/25/again-bash-cve-2014-7169/ > >> Fortunately, Python's subprocess has its `shell` argument default to >> False. However, `os.system` invokes the shell implicitly and is >> therefore a possible attack vector. > >Perhaps I'm missing something, but aren't there easier ways to attack >os.system than the bash env vulnerability? If I'm accepting and running >arbitrary strings from an untrusted user, there's no need for them to go >to the trouble of feeding me: > >"env x='() { :;}; echo gotcha' bash -c 'echo do something useful'" > >when they can just feed me: > >"echo gotcha" > >In other words, os.system is *already* an attack vector, unless you only >use it with trusted strings. I don't think the bash env vulnerability >adds to the attack surface. > >Have I missed something? Yes. Although it is possible to craft safe things for os.system, the issue is delivery of the attacks: plenty of totally standard things expose the shell to outside-supplied strings. And on most of those things, "the shell" is bash. The issue with the bash-imports-functions-badly issue is that shell functions are exported through the environment. Other things use the environment to pass info. The loud examples on the net are CGI scripts (query parameters passed through the environment, along with other things) and hostile DHCP servers (DHCP settings passed by the _client_ dhcpd to action scripts, as root). Both of these let someone outside your system deliver bash-exploit strings to bash scripts. Your cable/adsl modem? Probably an embedded Linux box, possibly using bash, and certainly a dhcp client of the ISP. Better still, for many people that same comprimisable modem is the DHCP _server_ for their home LAN... Cheers, Cameron Simpson Rebel without a clue, Born to be mild. - Kevin.Lowey at Usask.CA, DoD #0975 From tseaver at palladion.com Fri Sep 26 03:30:08 2014 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 25 Sep 2014 21:30:08 -0400 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926005954.GA63336@cskk.homeip.net> References: <20140925234016.GA19757@ando.pearwood.info> <20140926005954.GA63336@cskk.homeip.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/25/2014 08:59 PM, Cameron Simpson wrote: > Your cable/adsl modem? Probably an embedded Linux box, possibly using > bash, and certainly a dhcp client of the ISP. Better still, for many > people that same comprimisable modem is the DHCP _server_ for their > home LAN... Generally those devices are *not* using bash as '/bin/sh': it is too heavyweigh. Most will use 'busybox' or some other Swiss-army command for stuff which is separate commands on a "normal" linux system. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlQkwaAACgkQ+gerLs4ltQ4mYwCguMEUfwXZTM4FRS80HPCZx8DY hogAoNVIjWfn1R2obPH9LhRGFkzBR4Pw =oXxP -----END PGP SIGNATURE----- From cs at zip.com.au Fri Sep 26 04:55:47 2014 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 26 Sep 2014 12:55:47 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: References: Message-ID: <20140926025547.GA80827@cskk.homeip.net> On 25Sep2014 21:30, Tres Seaver wrote: >On 09/25/2014 08:59 PM, Cameron Simpson wrote: >> Your cable/adsl modem? Probably an embedded Linux box, possibly using >> bash, and certainly a dhcp client of the ISP. Better still, for many >> people that same comprimisable modem is the DHCP _server_ for their >> home LAN... > >Generally those devices are *not* using bash as '/bin/sh': it is too >heavyweigh. Most will use 'busybox' or some other Swiss-army command for >stuff which is separate commands on a "normal" linux system. Fair point. Cheers, Cameron Simpson There is no reason anyone would want a computer in their home. --Ken Olson, president, chairman and founder of Digital Equipment Corp., 1977 From marko at pacujo.net Fri Sep 26 06:31:15 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 26 Sep 2014 07:31:15 +0300 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140925234016.GA19757@ando.pearwood.info> (Steven D'Aprano's message of "Fri, 26 Sep 2014 09:40:17 +1000") References: <20140926001746.10743a46@fsol> <20140925234016.GA19757@ando.pearwood.info> Message-ID: <87d2ajypd8.fsf@elektro.pacujo.net> Steven D'Aprano : > Perhaps I'm missing something, but aren't there easier ways to attack > os.system than the bash env vulnerability? The main concern is the cases where you provide a service accessible through an SSH login and try to sandbox the client with limited functionality. SSH passes some environment variables on to the service which can then be used as an XSS vector. For example, if you wrote an SVN server's SSH front end with Python and used subprocess.Popen(shell=True) to execute the SVN operations, you could become a victim. Marko From p.f.moore at gmail.com Fri Sep 26 09:09:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 26 Sep 2014 08:09:45 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On 26 September 2014 01:38, Donald Stufft wrote: > Either way I'm fairly commited to making --user the default, the only > question > on my mind is what exactly does that look like (e.g. does root get --user by > default?) and how we get from where we are now to that point. I think that > raising an error here is fairly unfriendly when we can know with pretty good > certainity what the user wanted (and they can explicitly declare that if > they want too). A couple of points come to mind (although they may be more suitable for distutils-sig). 1. Do user installs "leak" into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate). Note that both of these points apply both to venv and virtualenvs. Paul From jeremy at jeremysanders.net Fri Sep 26 09:28:15 2014 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Fri, 26 Sep 2014 09:28:15 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> Message-ID: Antoine Pitrou wrote: > Fortunately, Python's subprocess has its `shell` argument default to > False. However, `os.system` invokes the shell implicitly and is > therefore a possible attack vector. Of course anything called by subprocess with shell=False may invoke the shell itself if it runs other processes. Jeremy From mcepl at cepl.eu Fri Sep 26 09:28:39 2014 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Fri, 26 Sep 2014 09:28:39 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> <20140925231437.GA84686@cskk.homeip.net> Message-ID: On 2014-09-25, 23:14 GMT, Cameron Simpson wrote: >>Fortunately, Python's subprocess has its `shell` argument default to >>False. However, `os.system` invokes the shell implicitly and is >>therefore a possible attack vector. > > Only if /bin/sh is bash :-) Not always the case, fortunately. Where does your faith that other /bin/sh implementations (dash, busybox, etc.) are less buggy comes from? On the contrary, bash being the most used, beaten, patched, and studied of them all has plenty of arguments to claim to be the most secure /bin/sh implementation around. You just don't know about those other guys bugs. No reason to believe hackers don't know about them either. Mat?j From hasan.diwan at gmail.com Fri Sep 26 10:10:53 2014 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Fri, 26 Sep 2014 01:10:53 -0700 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: References: <20140926001746.10743a46@fsol> <20140925231437.GA84686@cskk.homeip.net> Message-ID: Mat?j, On 26 September 2014 00:28, Mat?j Cepl wrote: > Where does your faith that other /bin/sh implementations (dash, > busybox, etc.) are less buggy comes from? The fact that they are simpler, in terms of lines of code. It's no guarantee, but the less a given piece of code does, the less bugs it will have. -- H -- OpenPGP: https://hasan.d8u.us/gpg.key Sent from my mobile device Envoy? de mon portable -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri Sep 26 12:35:42 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 26 Sep 2014 13:35:42 +0300 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926001746.10743a46@fsol> References: <20140926001746.10743a46@fsol> Message-ID: On 26.09.14 01:17, Antoine Pitrou wrote: > Fortunately, Python's subprocess has its `shell` argument default to > False. However, `os.system` invokes the shell implicitly and is > therefore a possible attack vector. Fortunately dash (which is used as /bin/sh in Debian and Ubuntu) is not vulnerable. $ x='() { :;}; echo gotcha' ./python -c 'import os; os.system("echo do something useful")' do something useful From solipsis at pitrou.net Fri Sep 26 13:16:05 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Sep 2014 13:16:05 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> <20140925231437.GA84686@cskk.homeip.net> Message-ID: <20140926131605.65b9c34d@fsol> On Fri, 26 Sep 2014 01:10:53 -0700 Hasan Diwan wrote: > Mat?j, > > On 26 September 2014 00:28, Mat?j Cepl wrote: > > > Where does your faith that other /bin/sh implementations (dash, > > busybox, etc.) are less buggy comes from? > > > The fact that they are simpler, in terms of lines of code. It's no > guarantee, but the less a given piece of code does, the less bugs it will > have. -- H And that they have less "features" (which is certainly correlated to their simplicity). IIUC, the misimplemented feature leading to this vulnerability is a bash-ism. Regards Antoine. From stefan_ml at behnel.de Fri Sep 26 14:56:05 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Sep 2014 14:56:05 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: References: <20140926001746.10743a46@fsol> Message-ID: Jeremy Sanders schrieb am 26.09.2014 um 09:28: > Antoine Pitrou wrote: > >> Fortunately, Python's subprocess has its `shell` argument default to >> False. However, `os.system` invokes the shell implicitly and is >> therefore a possible attack vector. > > Of course anything called by subprocess with shell=False may invoke the > shell itself if it runs other processes. Ok, but does that really make it a relevant topic for python-dev? Stefan From solipsis at pitrou.net Fri Sep 26 15:03:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Sep 2014 15:03:51 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> Message-ID: <20140926150351.441aec31@fsol> On Fri, 26 Sep 2014 14:56:05 +0200 Stefan Behnel wrote: > Jeremy Sanders schrieb am 26.09.2014 um 09:28: > > Antoine Pitrou wrote: > > > >> Fortunately, Python's subprocess has its `shell` argument default to > >> False. However, `os.system` invokes the shell implicitly and is > >> therefore a possible attack vector. > > > > Of course anything called by subprocess with shell=False may invoke the > > shell itself if it runs other processes. > > Ok, but does that really make it a relevant topic for python-dev? No. I don't know why the OP posted here. (but we have all kinds of borderline discussion threads these days, and people don't seem to care when they are asked to move the discussion elsewhere, so...) Regards Antoine. From donald at stufft.io Fri Sep 26 15:31:20 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 26 Sep 2014 09:31:20 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: <0933E401-854D-4CA6-ADF4-C5723FBDD03D@stufft.io> > On Sep 26, 2014, at 3:09 AM, Paul Moore wrote: > > On 26 September 2014 01:38, Donald Stufft wrote: >> Either way I'm fairly commited to making --user the default, the only >> question >> on my mind is what exactly does that look like (e.g. does root get --user by >> default?) and how we get from where we are now to that point. I think that >> raising an error here is fairly unfriendly when we can know with pretty good >> certainity what the user wanted (and they can explicitly declare that if >> they want too). > > A couple of points come to mind (although they may be more suitable > for distutils-sig). > > 1. Do user installs "leak" into virtualenvs? If so, then in effect > --use-system-packages is switched back on again if --user installs > become the norm. Which is almost certainly not what is wanted. > 2. pip install should default to not being --user when run from within > a virtualenv (same logic as the isolated Python case, but much more > important that behaviour remains as now, because the whole *point* of > virtualenvs is to isolate). > > Note that both of these points apply both to venv and virtualenvs. 1. No they don?t leak as far as I?m aware. 2. Yea, I think we throw an error when you use ?user inside a virtual environment. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at jeremysanders.net Fri Sep 26 15:43:08 2014 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Fri, 26 Sep 2014 15:43:08 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX References: <20140926001746.10743a46@fsol> Message-ID: Stefan Behnel wrote: > Ok, but does that really make it a relevant topic for python-dev? Sorry - I thought I was reading python-general. gmane makes it too easy to post :-). However, I think it's worth pointing that out, in case people think that Popen is a security panacea. J From p.f.moore at gmail.com Fri Sep 26 15:53:46 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 26 Sep 2014 14:53:46 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: <0933E401-854D-4CA6-ADF4-C5723FBDD03D@stufft.io> References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> <0933E401-854D-4CA6-ADF4-C5723FBDD03D@stufft.io> Message-ID: On 26 September 2014 14:31, Donald Stufft wrote: > Yea, I think we throw an error when you use ?user inside a virtual > environment. So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention? Paul From donald at stufft.io Fri Sep 26 15:57:15 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 26 Sep 2014 09:57:15 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> <0933E401-854D-4CA6-ADF4-C5723FBDD03D@stufft.io> Message-ID: <4A0C6785-275D-44D1-A154-5164D670B350@stufft.io> > On Sep 26, 2014, at 9:53 AM, Paul Moore wrote: > > On 26 September 2014 14:31, Donald Stufft wrote: >> Yea, I think we throw an error when you use ?user inside a virtual >> environment. > > So if --user became the default, what would happen? I'd like pip > inside a virtualenv to install into the environment without needing a > --system flag. Is that the intention? > > Paul default = ??user? if not running_inside_virtualenv() else ??system" --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Sep 26 18:07:31 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 26 Sep 2014 18:07:31 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140926160731.8854556247@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-09-19 - 2014-09-26) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4677 (+15) closed 29587 (+43) total 34264 (+58) Open issues with patches: 2197 Issues opened (39) ================== #22444: Floor divide should return int http://bugs.python.org/issue22444 opened by belopolsky #22445: Memoryviews require more strict contiguous checks then necessa http://bugs.python.org/issue22445 opened by seberg #22449: SSLContext.load_verify_locations behavior on Windows and OSX http://bugs.python.org/issue22449 opened by christian.heimes #22450: urllib doesn't put Accept: */* in the headers http://bugs.python.org/issue22450 opened by rhettinger #22452: addTypeEqualityFunc is not used in assertListEqual http://bugs.python.org/issue22452 opened by simonzack #22453: PyObject_REPR macro causes refcount leak http://bugs.python.org/issue22453 opened by Chris.Colbert #22454: Adding the opposite function of shlex.split() http://bugs.python.org/issue22454 opened by Sworddragon #22455: idna/punycode give wrong results on narrow builds http://bugs.python.org/issue22455 opened by bukzor #22456: __base__ undocumented http://bugs.python.org/issue22456 opened by Arfrever #22457: load_tests not invoked in root __init__.py when start=package http://bugs.python.org/issue22457 opened by rbcollins #22458: Add fractions benchmark http://bugs.python.org/issue22458 opened by scoder #22460: idle editor: replace all in selection http://bugs.python.org/issue22460 opened by bagratte #22462: Modules/pyexpat.c violates PEP 384 http://bugs.python.org/issue22462 opened by Mark.Shannon #22463: Warnings when building on AIX http://bugs.python.org/issue22463 opened by jelie #22465: Number agreement error in section 3.2 of web docs http://bugs.python.org/issue22465 opened by pauamma #22466: problem with installing python 2.7.8 http://bugs.python.org/issue22466 opened by elctr0 #22468: Tarfile using fstat on GZip file object http://bugs.python.org/issue22468 opened by bartolsthoorn #22470: Possible integer overflow in error handlers http://bugs.python.org/issue22470 opened by serhiy.storchaka #22472: OSErrors should use str and not repr on paths http://bugs.python.org/issue22472 opened by r.david.murray #22473: The gloss on asyncio "future with run_forever" example is conf http://bugs.python.org/issue22473 opened by r.david.murray #22474: No explanation of how a task gets destroyed in asyncio 'task' http://bugs.python.org/issue22474 opened by r.david.murray #22475: asyncio task get_stack documentation seems to contradict itsel http://bugs.python.org/issue22475 opened by r.david.murray #22476: asyncio task chapter confusion about 'task', 'future', and 'sc http://bugs.python.org/issue22476 opened by r.david.murray #22477: GCD in Fractions http://bugs.python.org/issue22477 opened by brg at gladman.plus.com #22480: SystemExit out of run_until_complete causes AttributeError whe http://bugs.python.org/issue22480 opened by chrysn #22482: logging: fileConfig doesn't support formatter styles http://bugs.python.org/issue22482 opened by domzippilli #22486: Add math.gcd() http://bugs.python.org/issue22486 opened by scoder #22489: .gitignore file http://bugs.python.org/issue22489 opened by rbcollins #22490: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs http://bugs.python.org/issue22490 opened by tdsmith #22491: Support Unicode line boundaries in regular expression http://bugs.python.org/issue22491 opened by serhiy.storchaka #22492: small addition to print() docs: no binary streams. http://bugs.python.org/issue22492 opened by georg.brandl #22493: Deprecate the use of flags not at the start of regular express http://bugs.python.org/issue22493 opened by serhiy.storchaka #22494: default logging time string is not localized http://bugs.python.org/issue22494 opened by sdague #22495: merge large parts of test_binop.py and test_fractions.py http://bugs.python.org/issue22495 opened by wolma #22496: urllib2 fails against IIS (urllib2 can't parse 401 reply www-a http://bugs.python.org/issue22496 opened by deronnax #22497: msiexec not creating msvcr90.dll with python -2.7.6.msi http://bugs.python.org/issue22497 opened by dykesk #22499: [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636 http://bugs.python.org/issue22499 opened by nikratio #22500: Argparse always stores True for positional arguments http://bugs.python.org/issue22500 opened by Tristan.Fisher #22501: Optimise PyLong division by 1 or -1 http://bugs.python.org/issue22501 opened by scoder Most recent 15 issues with no replies (15) ========================================== #22495: merge large parts of test_binop.py and test_fractions.py http://bugs.python.org/issue22495 #22494: default logging time string is not localized http://bugs.python.org/issue22494 #22493: Deprecate the use of flags not at the start of regular express http://bugs.python.org/issue22493 #22492: small addition to print() docs: no binary streams. http://bugs.python.org/issue22492 #22489: .gitignore file http://bugs.python.org/issue22489 #22475: asyncio task get_stack documentation seems to contradict itsel http://bugs.python.org/issue22475 #22470: Possible integer overflow in error handlers http://bugs.python.org/issue22470 #22468: Tarfile using fstat on GZip file object http://bugs.python.org/issue22468 #22465: Number agreement error in section 3.2 of web docs http://bugs.python.org/issue22465 #22463: Warnings when building on AIX http://bugs.python.org/issue22463 #22462: Modules/pyexpat.c violates PEP 384 http://bugs.python.org/issue22462 #22460: idle editor: replace all in selection http://bugs.python.org/issue22460 #22455: idna/punycode give wrong results on narrow builds http://bugs.python.org/issue22455 #22453: PyObject_REPR macro causes refcount leak http://bugs.python.org/issue22453 #22429: asyncio: pending call to loop.stop() if a coroutine raises a B http://bugs.python.org/issue22429 Most recent 15 issues waiting for review (15) ============================================= #22501: Optimise PyLong division by 1 or -1 http://bugs.python.org/issue22501 #22493: Deprecate the use of flags not at the start of regular express http://bugs.python.org/issue22493 #22492: small addition to print() docs: no binary streams. http://bugs.python.org/issue22492 #22490: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs http://bugs.python.org/issue22490 #22489: .gitignore file http://bugs.python.org/issue22489 #22486: Add math.gcd() http://bugs.python.org/issue22486 #22470: Possible integer overflow in error handlers http://bugs.python.org/issue22470 #22462: Modules/pyexpat.c violates PEP 384 http://bugs.python.org/issue22462 #22457: load_tests not invoked in root __init__.py when start=package http://bugs.python.org/issue22457 #22450: urllib doesn't put Accept: */* in the headers http://bugs.python.org/issue22450 #22449: SSLContext.load_verify_locations behavior on Windows and OSX http://bugs.python.org/issue22449 #22445: Memoryviews require more strict contiguous checks then necessa http://bugs.python.org/issue22445 #22442: subprocess.check_call hangs on large PIPEd data. http://bugs.python.org/issue22442 #22437: re module: number of named groups is limited to 100 max http://bugs.python.org/issue22437 #22435: socketserver.TCPSocket leaks socket to garbage collector if se http://bugs.python.org/issue22435 Top 10 most discussed issues (10) ================================= #22477: GCD in Fractions http://bugs.python.org/issue22477 31 msgs #22444: Floor divide should return int http://bugs.python.org/issue22444 23 msgs #22486: Add math.gcd() http://bugs.python.org/issue22486 23 msgs #22501: Optimise PyLong division by 1 or -1 http://bugs.python.org/issue22501 16 msgs #18814: Add codecs.convert_surrogateescape to "clean" surrogate escape http://bugs.python.org/issue18814 15 msgs #22445: Memoryviews require more strict contiguous checks then necessa http://bugs.python.org/issue22445 14 msgs #1602: windows console doesn't print or input Unicode http://bugs.python.org/issue1602 12 msgs #22458: Add fractions benchmark http://bugs.python.org/issue22458 9 msgs #22396: AIX posix_fadvise and posix_fallocate http://bugs.python.org/issue22396 8 msgs #22466: problem with installing python 2.7.8 http://bugs.python.org/issue22466 8 msgs Issues closed (41) ================== #4888: misplaced (or misleading) assert in ceval.c http://bugs.python.org/issue4888 closed by berker.peksag #12845: PEP-3118: C-contiguity with zero strides http://bugs.python.org/issue12845 closed by skrah #14293: Message methods delegated via __getattr__ inaccessible using s http://bugs.python.org/issue14293 closed by r.david.murray #15799: httplib client and statusline http://bugs.python.org/issue15799 closed by karlcow #17462: argparse FAQ: how it is different from optparse http://bugs.python.org/issue17462 closed by berker.peksag #18993: There is an overshadowed and invalid test in testmock.py http://bugs.python.org/issue18993 closed by berker.peksag #20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 closed by serhiy.storchaka #21079: EmailMessage.is_attachment == False if filename is present http://bugs.python.org/issue21079 closed by r.david.murray #21091: EmailMessage.is_attachment should be a method http://bugs.python.org/issue21091 closed by r.david.murray #21332: subprocess bufsize=1 docs are misleading http://bugs.python.org/issue21332 closed by pitrou #21431: 3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are http://bugs.python.org/issue21431 closed by python-dev #21860: Correct FileIO docstrings http://bugs.python.org/issue21860 closed by berker.peksag #22278: urljoin duplicate slashes http://bugs.python.org/issue22278 closed by orsenthil #22359: Remove incorrect uses of recursive make http://bugs.python.org/issue22359 closed by pitrou #22362: Warn about octal escapes > 0o377 in re http://bugs.python.org/issue22362 closed by serhiy.storchaka #22408: Tkinter doesn't handle Unicode dead key combinations on Window http://bugs.python.org/issue22408 closed by ned.deily #22415: Fix re debugging output http://bugs.python.org/issue22415 closed by serhiy.storchaka #22420: Use print(file=sys.stderr) instead of sys.stderr.write() in ID http://bugs.python.org/issue22420 closed by terry.reedy #22427: TemporaryDirectory attempts to clean up twice http://bugs.python.org/issue22427 closed by serhiy.storchaka #22438: eventlet broke by python 2.7.x http://bugs.python.org/issue22438 closed by benjamin.peterson #22440: Setting SSLContext object's check_hostname manually might acci http://bugs.python.org/issue22440 closed by christian.heimes #22443: read(1) blocks on unflushed output http://bugs.python.org/issue22443 closed by Sworddragon #22446: Shortening code in abc.py http://bugs.python.org/issue22446 closed by benjamin.peterson #22447: logging.config.fileConfig attempts to write to file even when http://bugs.python.org/issue22447 closed by vinay.sajip #22448: call_at/call_later with Timer cancellation can result in (prac http://bugs.python.org/issue22448 closed by yselivanov #22451: filtertuple, filterstring and filterunicode don't have optimiz http://bugs.python.org/issue22451 closed by pitrou #22459: str.strip() documentation: wrong example http://bugs.python.org/issue22459 closed by python-dev #22461: Test failure: Lib/test/test_pydoc.py line 851, "topic?key=def" http://bugs.python.org/issue22461 closed by georg.brandl #22464: Speed up fractions implementation http://bugs.python.org/issue22464 closed by scoder #22467: Lib/http/server.py, inconsistent header casing http://bugs.python.org/issue22467 closed by r.david.murray #22469: Allow the "backslashreplace" error handler support decoding http://bugs.python.org/issue22469 closed by serhiy.storchaka #22471: Python build problems via Homebrew on Mac OS X when GNU core/f http://bugs.python.org/issue22471 closed by ned.deily #22478: tests for urllib2net are in bad shapes http://bugs.python.org/issue22478 closed by orsenthil #22479: strange behavior of importing random module http://bugs.python.org/issue22479 closed by haypo #22481: Lists within tuples mutability issue http://bugs.python.org/issue22481 closed by ezio.melotti #22483: Copyright infringement on PyPI http://bugs.python.org/issue22483 closed by dstufft #22484: Build doc archives for RC versions, docs download broken for 3 http://bugs.python.org/issue22484 closed by python-dev #22485: Documentation download links (3.4.2 http://bugs.python.org/issue22485 closed by berker.peksag #22487: ABC register doesn't check abstract methods http://bugs.python.org/issue22487 closed by r.david.murray #22488: 3.4 rc2 docs download link broken http://bugs.python.org/issue22488 closed by ned.deily #22498: frozenset allows modification via -= operator http://bugs.python.org/issue22498 closed by ezio.melotti From Steve.Dower at microsoft.com Fri Sep 26 20:01:31 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 26 Sep 2014 18:01:31 +0000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 Message-ID: Hi all, (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 This package contains all the tools and headers required to build C extension modules for Python 2.7 32-bit and 64-bit (note that some extension modules require 3rd party dependencies such as OpenSSL or libxml2 that are not included). Other versions of Python built with Visual C++ 2008 are also supported. You can install the package without requiring administrative privileges and, with the latest version of setuptools (when it releases), use tools such as pip, wheel, or a setup.py file to produce binaries on Windows. Unfortunately, this package isn't necessarily going to help with building CPython 2.7 itself, as the build process is complicated and Visual Studio 2008 is practically required. However, as most people aren't building CPython on Windows, this isn't a huge issue. This compiler package should be sufficient for most extension modules. I should also point out that VC9 is no longer supported by Microsoft. This means there won't be any improvements or bug fixes coming, and there's no official support offered. Feel free to contact me directly if there are issues with the package. Cheers, Steve From donald at stufft.io Fri Sep 26 20:07:39 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 26 Sep 2014 14:07:39 -0400 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: Awesome! > On Sep 26, 2014, at 2:01 PM, Steve Dower wrote: > > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 > > This package contains all the tools and headers required to build C extension modules for Python 2.7 32-bit and 64-bit (note that some extension modules require 3rd party dependencies such as OpenSSL or libxml2 that are not included). Other versions of Python built with Visual C++ 2008 are also supported. > > You can install the package without requiring administrative privileges and, with the latest version of setuptools (when it releases), use tools such as pip, wheel, or a setup.py file to produce binaries on Windows. > > Unfortunately, this package isn't necessarily going to help with building CPython 2.7 itself, as the build process is complicated and Visual Studio 2008 is practically required. However, as most people aren't building CPython on Windows, this isn't a huge issue. This compiler package should be sufficient for most extension modules. > > I should also point out that VC9 is no longer supported by Microsoft. This means there won't be any improvements or bug fixes coming, and there's no official support offered. Feel free to contact me directly if there are issues with the package. > > Cheers, > Steve > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io From mail at timgolden.me.uk Fri Sep 26 20:29:44 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 26 Sep 2014 19:29:44 +0100 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: <5425B098.4020306@timgolden.me.uk> On 26/09/2014 19:01, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. > Official announcements are coming when setuptools makes their next > release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. > VC9). We've produced this package to help library developers build > wheels for Windows, but also to help users unblock themselves when > they need to build C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: > http://aka.ms/vcpython27 Wow. I am lost for words. TJG From dario670 at gmail.com Fri Sep 26 19:50:01 2014 From: dario670 at gmail.com (=?utf-8?B?ZGFyaW82NzBAZ21haWwuY29t?=) Date: Fri, 26 Sep 2014 10:50:01 -0700 (PDT) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <000f4242.0350d4132fe376a0@gmail.com> T-Mobile. America's First Nationwide 4G Network. ------ Original message------From: Python trackerDate: Fri, Sep 26, 2014 12:07 PMTo: python-dev at python.org;Subject:[Python-Dev] Summary of Python tracker IssuesACTIVITY SUMMARY (2014-09-19 - 2014-09-26)Python tracker at http://bugs.python.org/To view or respond to any of the issues listed below, click on the issue.Do NOT respond to this message.Issues counts and deltas: open 4677 (+15) closed 29587 (+43) total 34264 (+58)Open issues with patches: 2197 Issues opened (39)==================#22444: Floor divide should return inthttp://bugs.python.org/issue22444 opened by belopolsky#22445: Memoryviews require more strict contiguous checks then necessahttp://bugs.python.org/issue22445 opened by seberg#22449: SSLContext.load_verify_locations behavior on Windows and OSXhttp://bugs.python.org/issue22449 opened by christian.heimes#22450: urllib doesn't put Accept: */* in the headershttp://bugs.python.org/issue22450 opened by rhettinger#22452: addTypeEqualityFunc is not used in assertListEqualhttp://bugs.python.org/issue22452 opened by simonzack#22453: PyObject_REPR macro causes refcount leakhttp://bugs.python.org/issue22453 opened by Chris.Colbert#22454: Adding the opposite function of shlex.split()http://bugs.python.org/issue22454 opened by Sworddragon#22455: idna/punycode give wrong results on narrow buildshttp://bugs.python.org/issue22455 opened by bukzor#22456: __base__ undocumentedhttp://bugs.python.org/issue22456 opened by Arfrever#22457: load_tests not invoked in root __init__.py when start=package http://bugs.python.org/issue22457 opened by rbcollins#22458: Add fractions benchmarkhttp://bugs.python.org/issue22458 opened by scoder#22460: idle editor: replace all in selectionhttp://bugs.python.org/issue22460 opened by bagratte#22462: Modules/pyexpat.c violates PEP 384http://bugs.python.org/issue22462 opened by Mark.Shannon#22463: Warnings when building on AIXhttp://bugs.python.org/issue22463 opened by jelie#22465: Number agreement error in section 3.2 of web docshttp://bugs.python.org/issue22465 opened by pauamma#22466: problem with installing python 2.7.8http://bugs.python.org/issue22466 opened by elctr0#22468: Tarfile using fstat on GZip file objecthttp://bugs.python.org/issue22468 opened by bartolsthoorn#22470: Possible integer overflow in error handlershttp://bugs.python.org/issue22470 opened by serhiy.storchaka#22472: OSErrors should use str and not repr on pathshttp://bugs.python.org/issue22472 opened by r.david.murray#22473: The gloss on asyncio "future with run_forever" example is confhttp://bugs.python.org/issue22473 opened by r.david.murray#22474: No explanation of how a task gets destroyed in asyncio 'task' http://bugs.python.org/issue22474 opened by r.david.murray#22475: asyncio task get_stack documentation seems to contradict itselhttp://bugs.python.org/issue22475 opened by r.david.murray#22476: asyncio task chapter confusion about 'task', 'future', and 'schttp://bugs.python.org/issue22476 opened by r.david.murray#22477: GCD in Fractionshttp://bugs.python.org/issue22477 opened by brg at gladman.plus.com#22480: SystemExit out of run_until_complete causes AttributeError whehttp://bugs.python.org/issue22480 opened by chrysn#22482: logging: fileConfig doesn't support formatter styleshttp://bugs.python.org/issue22482 opened by domzippilli#22486: Add math.gcd()http://bugs.python.org/issue22486 opened by scoder#22489: .gitignore filehttp://bugs.python.org/issue22489 opened by rbcollins#22490: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installshttp://bugs.python.org/issue22490 opened by tdsmith#22491: Support Unicode line boundaries in regular expressionhttp://bugs.python.org/issue22491 opened by serhiy.storchaka#22492: small addition to print() docs: no binary streams.http://bugs.python.org/issue22492 opened by georg.brandl#22493: Deprecate the use of flags not at the start of regular expresshttp://bugs.python.org/issue22493 opened by serhiy.storchaka#22494: default logging time string is not localizedhttp://bugs.python.org/issue22494 opened by sdague#22495: merge large parts of test_binop.py and test_fractions.pyhttp://bugs.python.org/issue22495 opened by wolma#22496: urllib2 fails against IIS (urllib2 can't parse 401 reply www-ahttp://bugs.python.org/issue22496 opened by deronnax#22497: msiexec not creating msvcr90.dll with python -2.7.6.msihttp://bugs.python.org/issue22497 opened by dykesk#22499: [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636http://bugs.python.org/issue22499 opened by nikratio#22500: Argparse always stores True for positional argumentshttp://bugs.python.org/issue22500 opened by Tristan.Fisher#22501: Optimise PyLong division by 1 or -1http://bugs.python.org/issue22501 opened by scoderMost recent 15 issues with no replies (15)==========================================#22495: merge large parts of test_binop.py and test_fractions.pyhttp://bugs.python.org/issue22495#22494: default logging time string is not localizedhttp://bugs.python.org/issue22494#22493: Deprecate the use of flags not at the start of regular expresshttp://bugs.python.org/issue22493#22492: small addition to print() docs: no binary streams.http://bugs.python.org/issue22492#22489: .gitignore filehttp://bugs.python.org/issue22489#22475: asyncio task get_stack documentation seems to contradict itselhttp://bugs.python.org/issue22475#22470: Possible integer overflow in error handlershttp://bugs.python.org/issue22470#22468: Tarfile using fstat on GZip file objecthttp://bugs.python.org/issue22468#22465: Number agreement error in section 3.2 of web docshttp://bugs.python.org/issue22465#22463: Warnings when building on AIXhttp://bugs.python.org/issue22463#22462: Modules/pyexpat.c violates PEP 384http://bugs.python.org/issue22462#22460: idle editor: replace all in selectionhttp://bugs.python.org/issue22460#22455: idna/punycode give wrong results on narrow buildshttp://bugs.python.org/issue22455#22453: PyObject_REPR macro causes refcount leakhttp://bugs.python.org/issue22453#22429: asyncio: pending call to loop.stop() if a coroutine raises a Bhttp://bugs.python.org/issue22429Most recent 15 issues waiting for review (15)=============================================#22501: Optimise PyLong division by 1 or -1http://bugs.python.org/issue22501#22493: Deprecate the use of flags not at the start of regular expresshttp://bugs.python.org/issue22493#22492: small addition to print() docs: no binary streams.http://bugs.python.org/issue22492#22490: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installshttp://bugs.python.org/issue22490#22489: .gitignore filehttp://bugs.python.org/issue22489#22486: Add math.gcd()http://bugs.python.org/issue22486#22470: Possible integer overflow in error handlershttp://bugs.python.org/issue22470#22462: Modules/pyexpat.c violates PEP 384http://bugs.python.org/issue22462#22457: load_tests not invoked in root __init__.py when start=package http://bugs.python.org/issue22457#22450: urllib doesn't put Accept: */* in the headershttp://bugs.python.org/issue22450#22449: SSLContext.load_verify_locations behavior on Windows and OSXhttp://bugs.python.org/issue22449#22445: Memoryviews require more strict contiguous checks then necessahttp://bugs.python.org/issue22445#22442: subprocess.check_call hangs on large PIPEd data.http://bugs.python.org/issue22442#22437: re module: number of named groups is limited to 100 maxhttp://bugs.python.org/issue22437#22435: socketserver.TCPSocket leaks socket to garbage collector if sehttp://bugs.python.org/issue22435Top 10 most discussed issues (10)=================================#22477: GCD in Fractionshttp://bugs.python.org/issue22477 31 msgs#22444: Floor divide should return inthttp://bugs.python.org/issue22444 23 msgs#22486: Add math.gcd()http://bugs.python.org/issue22486 23 msgs#22501: Optimise PyLong division by 1 or -1http://bugs.python.org/issue22501 16 msgs#18814: Add codecs.convert_surrogateescape to "clean" surrogate escapehttp://bugs.python.org/issue18814 15 msgs#22445: Memoryviews require more strict contiguous checks then necessahttp://bugs.python.org/issue22445 14 msgs#1602: windows console doesn't print or input Unicodehttp://bugs.python.org/issue1602 12 msgs#22458: Add fractions benchmarkhttp://bugs.python.org/issue22458 9 msgs#22396: AIX posix_fadvise and posix_fallocatehttp://bugs.python.org/issue22396 8 msgs#22466: problem with installing python 2.7.8http://bugs.python.org/issue22466 8 msgsIssues closed (41)==================#4888: misplaced (or misleading) assert in ceval.chttp://bugs.python.org/issue4888 closed by berker.peksag#12845: PEP-3118: C-contiguity with zero strideshttp://bugs.python.org/issue12845 closed by skrah#14293: Message methods delegated via __getattr__ inaccessible using shttp://bugs.python.org/issue14293 closed by r.david.murray#15799: httplib client and statuslinehttp://bugs.python.org/issue15799 closed by karlcow#17462: argparse FAQ: how it is different from optparsehttp://bugs.python.org/issue17462 closed by berker.peksag#18993: There is an overshadowed and invalid test in testmock.pyhttp://bugs.python.org/issue18993 closed by berker.peksag#20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 closed by serhiy.storchaka#21079: EmailMessage.is_attachment == False if filename is presenthttp://bugs.python.org/issue21079 closed by r.david.murray#21091: EmailMessage.is_attachment should be a methodhttp://bugs.python.org/issue21091 closed by r.david.murray#21332: subprocess bufsize=1 docs are misleadinghttp://bugs.python.org/issue21332 closed by pitrou#21431: 3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values arehttp://bugs.python.org/issue21431 closed by python-dev#21860: Correct FileIO docstringshttp://bugs.python.org/issue21860 closed by berker.peksag#22278: urljoin duplicate slasheshttp://bugs.python.org/issue22278 closed by orsenthil#22359: Remove incorrect uses of recursive makehttp://bugs.python.org/issue22359 closed by pitrou#22362: Warn about octal escapes > 0o377 in rehttp://bugs.python.org/issue22362 closed by serhiy.storchaka#22408: Tkinter doesn't handle Unicode dead key combinations on Windowhttp://bugs.python.org/issue22408 closed by ned.deily#22415: Fix re debugging outputhttp://bugs.python.org/issue22415 closed by serhiy.storchaka#22420: Use print(file=sys.stderr) instead of sys.stderr.write() in IDhttp://bugs.python.org/issue22420 closed by terry.reedy#22427: TemporaryDirectory attempts to clean up twicehttp://bugs.python.org/issue22427 closed by serhiy.storchaka#22438: eventlet broke by python 2.7.xhttp://bugs.python.org/issue22438 closed by benjamin.peterson#22440: Setting SSLContext object's check_hostname manually might accihttp://bugs.python.org/issue22440 closed by christian.heimes#22443: read(1) blocks on unflushed outputhttp://bugs.python.org/issue22443 closed by Sworddragon#22446: Shortening code in abc.pyhttp://bugs.python.org/issue22446 closed by benjamin.peterson#22447: logging.config.fileConfig attempts to write to file even when http://bugs.python.org/issue22447 closed by vinay.sajip#22448: call_at/call_later with Timer cancellation can result in (prachttp://bugs.python.org/issue22448 closed by yselivanov#22451: filtertuple, filterstring and filterunicode don't have optimizhttp://bugs.python.org/issue22451 closed by pitrou#22459: str.strip() documentation: wrong examplehttp://bugs.python.org/issue22459 closed by python-dev#22461: Test failure: Lib/test/test_pydoc.py line 851, "topic?key=def"http://bugs.python.org/issue22461 closed by georg.brandl#22464: Speed up fractions implementationhttp://bugs.python.org/issue22464 closed by scoder#22467: Lib/http/server.py, inconsistent header casinghttp://bugs.python.org/issue22467 closed by r.david.murray#22469: Allow the "backslashreplace" error handler support decodinghttp://bugs.python.org/issue22469 closed by serhiy.storchaka#22471: Python build problems via Homebrew on Mac OS X when GNU core/fhttp://bugs.python.org/issue22471 closed by ned.deily#22478: tests for urllib2net are in bad shapeshttp://bugs.python.org/issue22478 closed by orsenthil#22479: strange behavior of importing random modulehttp://bugs.python.org/issue22479 closed by haypo#22481: Lists within tuples mutability issuehttp://bugs.python.org/issue22481 closed by ezio.melotti#22483: Copyright infringement on PyPIhttp://bugs.python.org/issue22483 closed by dstufft#22484: Build doc archives for RC versions, docs download broken for 3http://bugs.python.org/issue22484 closed by python-dev#22485: Documentation download links (3.4.2http://bugs.python.org/issue22485 closed by berker.peksag#22487: ABC register doesn't check abstract methodshttp://bugs.python.org/issue22487 closed by r.david.murray#22488: 3.4 rc2 docs download link brokenhttp://bugs.python.org/issue22488 closed by ned.deily#22498: frozenset allows modification via -= operatorhttp://bugs.python.org/issue22498 closed by ezio.melotti_______________________________________________Python-Dev mailing listPython-Dev at python.orghttps://mail.python.org/mailman/listinfo/python-devUnsubscribe: https://mail.python.org/mailman/options/python-dev/dario670%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Sep 26 21:01:40 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 26 Sep 2014 14:01:40 -0500 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: At long last! Building C extensions on Windows will no longer be a pain in the rear! On Fri, Sep 26, 2014 at 1:01 PM, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. > Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). > We've produced this package to help library developers build wheels for > Windows, but also to help users unblock themselves when they need to build > C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: > http://aka.ms/vcpython27 > > This package contains all the tools and headers required to build C > extension modules for Python 2.7 32-bit and 64-bit (note that some > extension modules require 3rd party dependencies such as OpenSSL or libxml2 > that are not included). Other versions of Python built with Visual C++ 2008 > are also supported. > > You can install the package without requiring administrative privileges > and, with the latest version of setuptools (when it releases), use tools > such as pip, wheel, or a setup.py file to produce binaries on Windows. > > Unfortunately, this package isn't necessarily going to help with building > CPython 2.7 itself, as the build process is complicated and Visual Studio > 2008 is practically required. However, as most people aren't building > CPython on Windows, this isn't a huge issue. This compiler package should > be sufficient for most extension modules. > > I should also point out that VC9 is no longer supported by Microsoft. This > means there won't be any improvements or bug fixes coming, and there's no > official support offered. Feel free to contact me directly < > steve.dower at microsoft.com> if there are issues with the package. > > Cheers, > Steve > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Sep 26 21:18:06 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 26 Sep 2014 12:18:06 -0700 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: <5425BBEE.8050504@hastings.org> On 09/26/2014 11:01 AM, Steve Dower wrote: > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. That's gorgeous! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Sep 26 21:29:59 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 26 Sep 2014 20:29:59 +0100 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: On 26 September 2014 19:01, Steve Dower wrote: > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 That is fantastic news! Paul From cs at zip.com.au Sat Sep 27 00:11:57 2014 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 27 Sep 2014 08:11:57 +1000 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926131605.65b9c34d@fsol> References: <20140926131605.65b9c34d@fsol> Message-ID: <20140926221157.GA22121@cskk.homeip.net> On 26Sep2014 13:16, Antoine Pitrou wrote: >On Fri, 26 Sep 2014 01:10:53 -0700 >Hasan Diwan wrote: >> On 26 September 2014 00:28, Mat?j Cepl wrote: >> > Where does your faith that other /bin/sh implementations (dash, >> > busybox, etc.) are less buggy comes from? >> >> The fact that they are simpler, in terms of lines of code. It's no >> guarantee, but the less a given piece of code does, the less bugs it will >> have. -- H > >And that they have less "features" (which is certainly correlated to >their simplicity). IIUC, the misimplemented feature leading to this >vulnerability is a bash-ism. IIRC you could export functions in ksh. Or maybe only aliases. But that implies most POSIX shells may support it. I've never seen the point myself; it is not a feature I've ever needed. Cheers, Cameron Simpson Follow! But! Follow only if ye be men of valor, for the entrance to this cave is guarded by a creature so foul, so cruel that no man yet has fought with it and lived! Bones of four fifty men lie strewn about its lair. So, brave knights, if you do doubt your courage or your strength, come no further, for death awaits you all with nasty big pointy teeth. - Tim The Enchanter From chris.barker at noaa.gov Fri Sep 26 19:03:31 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 26 Sep 2014 10:03:31 -0700 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft wrote: > 2) Switch to ?user based on if the user has permission to write to the >> site-packages or not. >> > > ouch -- no. Why not a clear error message if pip can't write to > site-packages -- something like: > > I fairly strongly believe that the current default is doing a great > disservice > to users. I believe that for *most* people --user is the correct option for > them to be using and the fact that it's not the default and requires opt in > is a historical artifact more than anything else. > OK -- fine -- I think that history may be important -- at least for Py2, but I agree that --user is a better default. But what I'm objecting to is is switching the install mode based on the permissions that the user happens to be running with at the time. None of us should be routinely running as admin. So what I do (and I image a LOT of people do) is try to do whatever I need to do, and only if I get an error to I switch to admin mode (or add sudo, or ???). What I'm suggesting is that folks that simply expect stuff to get installed into the usual system python is not thinking ahead of time "oh, I need be admin for this", but rather, simply do it, and then, if you get a permission error, make the switch to either admin in mode, or add --user. If we do switch the default to --user, then this issue goes away. In short -- too much magic is bad. All this is also making me think that virtualenv and friends is the real solution to user installed packages anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Sep 27 07:08:11 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 27 Sep 2014 01:08:11 -0400 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On 9/26/2014 1:03 PM, Chris Barker wrote: > On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft > wrote: > >> 2) Switch to ?user based on if the user has permission to >> write to the >> site-packages or not. >> >> >> ouch -- no. Why not a clear error message if pip can't write to >> site-packages -- something like: > I fairly strongly believe that the current default is doing a great > disservice > to users. I believe that for *most* people --user is the correct > option for > them to be using and the fact that it's not the default and requires > opt in > is a historical artifact more than anything else. > > > OK -- fine -- I think that history may be important -- at least for Py2, > but I agree that --user is a better default. > > But what I'm objecting to is is switching the install mode based on the > permissions that the user happens to be running with at the time. > > None of us should be routinely running as admin. So what I do (and I > image a LOT of people do) is try to do whatever I need to do, and only > if I get an error to I switch to admin mode (or add sudo, or ???). What > I'm suggesting is that folks that simply expect stuff to get installed > into the usual system python is not thinking ahead of time "oh, I need > be admin for this", but rather, simply do it, and then, if you get a > permission error, make the switch to either admin in mode, or add --user. > > If we do switch the default to --user, then this issue goes away. > > In short -- too much magic is bad. Pip on Windows should act like a normal Windows program. If I install Python for all users, I expect pipped packages to be installed for all users too, unless I specify otherwise. If installation (for all users) requires admin privileges, I expect a UAC box to pop up and ask for the admin password. This is pretty routine, at least with Win7. Most every program I install does this either on installation or on first running. Some Windows operations also pop up a box. There are only a few things that require that I actually login as an admin user. -- Terry Jan Reedy From ncoghlan at gmail.com Sat Sep 27 09:10:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 27 Sep 2014 17:10:40 +1000 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On 27 Sep 2014 14:19, "Chris Barker" wrote: > > All this is also making me think that virtualenv and friends is the real solution to user installed packages anyway. The main use case that doesn't cover is system scripting on Linux, where you may want access to all the platform specific libraries. We're well and truly off topic for python-dev at this point, though... Probably best to reboot the thread on distutils-sig, rather than continuing this one. Cheers, Nick. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From catch-all at masklinn.net Sat Sep 27 09:23:55 2014 From: catch-all at masklinn.net (Xavier Morel) Date: Sat, 27 Sep 2014 09:23:55 +0200 Subject: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX In-Reply-To: <20140926221157.GA22121@cskk.homeip.net> References: <20140926131605.65b9c34d@fsol> <20140926221157.GA22121@cskk.homeip.net> Message-ID: On 2014-09-27, at 00:11 , Cameron Simpson wrote: > On 26Sep2014 13:16, Antoine Pitrou wrote: >> On Fri, 26 Sep 2014 01:10:53 -0700 >> Hasan Diwan wrote: >>> On 26 September 2014 00:28, Mat?j Cepl wrote: >>> > Where does your faith that other /bin/sh implementations (dash, >>> > busybox, etc.) are less buggy comes from? >>> >>> The fact that they are simpler, in terms of lines of code. It's no >>> guarantee, but the less a given piece of code does, the less bugs it will >>> have. -- H >> >> And that they have less "features" (which is certainly correlated to >> their simplicity). IIUC, the misimplemented feature leading to this >> vulnerability is a bash-ism. > > IIRC you could export functions in ksh. Or maybe only aliases. But that implies most POSIX shells may support it. From my understanding KSH's function export is so a function becomes available in the caller of a script e.g. if you define a function in your .kshrc it's internal to the file (and won't be available in the interactive shell) unless you export it: http://users.speakeasy.net/~arkay/216-7.4KshFunctions.html KSH (and ZSH) will also load functions from files on $FPATH, but AFAIK that's it. From p.f.moore at gmail.com Sat Sep 27 11:55:44 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 27 Sep 2014 10:55:44 +0100 Subject: [Python-Dev] 3.5 release schedule PEP In-Reply-To: References: <5422C440.2020001@mgmiller.net> <0844ea1b995d42ae8b4111203bbe125e@DM2PR0301MB0734.namprd03.prod.outlook.com> <5423354E.10408@mgmiller.net> <20140925030824.1654310f@fsol> <20140925105435.66bc53dc@fsol> <9936BEE3-35FD-412F-82F7-88C08F9DD2B5@stufft.io> <7B370651-74C4-481D-8B72-87DC8E747061@stufft.io> Message-ID: On 27 September 2014 06:08, Terry Reedy wrote: > Pip on Windows should act like a normal Windows program. If I install > Python for all users, I expect pipped packages to be installed for all users > too, unless I specify otherwise. If installation (for all users) requires > admin privileges, I expect a UAC box to pop up and ask for the admin > password. This is pretty routine, at least with Win7. Most every program I > install does this either on installation or on first running. Some Windows > operations also pop up a box. There are only a few things that require that > I actually login as an admin user. The main problem is that there is little or no prior art on Windows for *console* programs that require elevation. Those few that do need it require you to start the program from an elevated prompt, but that's not a good user experience. But having said that, I agree with your point, the UAC changes are designed specifically to get people used to the "think about what you're doing" approach, and we should be doing the same. Paul From solipsis at pitrou.net Sat Sep 27 14:12:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 27 Sep 2014 14:12:26 +0200 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 References: Message-ID: <20140927141226.5ad2979d@fsol> On Fri, 26 Sep 2014 18:01:31 +0000 Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) When you mention "setuptools", do you imply it doesn't work with plain distutils? Regards Antoine. From ncoghlan at gmail.com Sat Sep 27 15:01:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 27 Sep 2014 23:01:43 +1000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: <20140927141226.5ad2979d@fsol> References: <20140927141226.5ad2979d@fsol> Message-ID: On 27 September 2014 22:12, Antoine Pitrou wrote: > On Fri, 26 Sep 2014 18:01:31 +0000 > Steve Dower wrote: >> Hi all, >> >> (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) > > When you mention "setuptools", do you imply it doesn't work with plain > distutils? I'd expect it to have the same problem the SDK compilers currently do: as far as I am aware, distutils simply doesn't recognise them as available compilers. I personally believe we should treat handling both this and the SDK compilers properly as a platform-enablement bug for distutils and ensure they work properly with the currently maintained branches (including 2.7). David Cournapeau offered (on distutils-sig) to tackle that for at least the SDK compilers (the thread was before Steve's announcement of the more readily available VC9 compiler download). Cheers, Nick. P.S. Note that pip always runs setup.py via setuptools, so updating setuptools also deals with the general "pip install" case of building from source. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Sep 27 15:10:48 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 27 Sep 2014 14:10:48 +0100 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: <20140927141226.5ad2979d@fsol> Message-ID: On 27 September 2014 14:01, Nick Coghlan wrote: > I personally believe we should treat handling both this and the SDK > compilers properly as a platform-enablement bug for distutils and > ensure they work properly with the currently maintained branches > (including 2.7). +1 From christian at python.org Sat Sep 27 16:18:59 2014 From: christian at python.org (Christian Heimes) Date: Sat, 27 Sep 2014 16:18:59 +0200 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: On 26.09.2014 20:01, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 Awesome! :) Thanks a lot, Steve! Is it possible to compile extensions from Python's numerical stack such as NumPy, SciPy and SciKit, too? From Steve.Dower at microsoft.com Sat Sep 27 17:22:53 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 27 Sep 2014 15:22:53 +0000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: , Message-ID: <7130e9ed93e640688b1022d6bddde07f@DM2PR0301MB0734.namprd03.prod.outlook.com> It'll help with the numerical stack, but only a little. The devs involved have largely figured it out already and I can't provide a good Fortran compiler or BLAS library, which is what they need. It'll be much more valuable for the small packages that have one vital C extension - currently those are basically unusable without a wheel or a compiler. Many DB and XML packages seem to fall into this category. It also works for Cython, so anything that uses Cython should work with just these compilers. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Christian Heimes Sent: ?9/?27/?2014 7:19 To: python-dev at python.org Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 On 26.09.2014 20:01, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 Awesome! :) Thanks a lot, Steve! Is it possible to compile extensions from Python's numerical stack such as NumPy, SciPy and SciKit, too? _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Sep 27 17:31:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 27 Sep 2014 17:31:55 +0200 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: <20140927141226.5ad2979d@fsol> Message-ID: <20140927173155.49b2878e@fsol> On Sat, 27 Sep 2014 14:10:48 +0100 Paul Moore wrote: > On 27 September 2014 14:01, Nick Coghlan wrote: > > I personally believe we should treat handling both this and the SDK > > compilers properly as a platform-enablement bug for distutils and > > ensure they work properly with the currently maintained branches > > (including 2.7). > > +1 +1 as well. Regards Antoine. From Steve.Dower at microsoft.com Sat Sep 27 17:16:41 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 27 Sep 2014 15:16:41 +0000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: <20140927141226.5ad2979d@fsol> References: , <20140927141226.5ad2979d@fsol> Message-ID: Plain distutils won't detect it. It would be easy enough to fix 2.7.9, but "update Python" is a big/impossible ask for a lot of people, whereas "update setuptools" is easy and also covers Python 2.6 and <3.3. The compiler installer can't set the keys that distutils looks for without losing the per-user installation, and it may also corrupt actual installs of Visual Studio. A monkey patch via setuptools was the best way to handle this - covers pip and Cython and can be ported to other libraries that care but avoid setuptools. Now that we have a patch, there's very limited value in fixing 2.7.9, IMO. But I'm willing to be convinced - we can always add a version check to the setuptools patch. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Antoine Pitrou Sent: ?9/?27/?2014 5:13 To: python-dev at python.org Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 On Fri, 26 Sep 2014 18:01:31 +0000 Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) When you mention "setuptools", do you imply it doesn't work with plain distutils? Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sat Sep 27 17:59:26 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 27 Sep 2014 15:59:26 +0000 (UTC) Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 References: Message-ID: <1635163120433524790.548636sturla.molden-gmail.com@news.gmane.org> Christian Heimes wrote: > Is it possible to compile extensions from Python's numerical stack such > as NumPy, SciPy and SciKit, too? The official NumPy installer is currently built with VC9, so probably yes. Other parts of the SciPy stack needs a Fortran compiler as well, so those might be more tricky. Currently the limitation to Fortran 77 is considered lifted, Fortran 90 and later will be allowed, so g77 is no longer an option. In practice you will need Intel ifort or a patched MinGW gfortran. Because of this the SciPy community has been creating a customized MinGW toolchain (including gfortran) for building binary wheels on Windows. It is patched to make sure that e.g. the MinGW runtime does not conflict with the VC9 code in the official Python 2.7 installer and that libgfortran uses the correct C runtime. The stack alignment is also changed to make it VC9 compatible. There was also a customization of the C++ exception handling. In addition to this the MinGW runtime and libgfortran are statically linked, so there are no extra runtime DLLs to install. https://github.com/numpy/numpy/wiki/Mingw-static-toolchain The toolchain also contains a build of OpenBLAS to use as BLAS and LAPACK when building NumPy and the SciPy stack. Intel MKL or ATLAS might be preferred though, due to concerns about the maturity of OpenBLAS. Sturla Molden From sturla.molden at gmail.com Sat Sep 27 18:28:00 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 27 Sep 2014 16:28:00 +0000 (UTC) Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 References: <7130e9ed93e640688b1022d6bddde07f@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <924848234433526499.479484sturla.molden-gmail.com@news.gmane.org> Steve Dower wrote: > It'll help with the numerical stack, but only a little. The devs involved > have largely figured it out already and I can't provide a good Fortran > compiler or BLAS library, which is what they need. We finally have a MinGW based toolchain that can be used. Making sure it was compatible with VC9 is actually worse than most would expect, as there are subtile incompatibilities between vanilla MinGW and VC9, beyond just linking the same C runtime DLL. But it was worked out: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain As for BLAS, the NumPy/SciPy devs have procured a permission from Intel to use MKL in binary wheels. But still there will be official binaries linked with a free BLAS library available. Currently we use ATLAS, but the plan is to use OpenBLAS (successor to GotoBLAS2) when it matures. OpenBLAS is currently the fastest abd most scalable BLAS library available, actually better than MKL, but it is severely underfunded. It is not a good situation for the industry that the only open BLAS library with the performance of MKL is a Chinese student project in HPC. ATLAS is unfortunately far less performant and scalable. Apple and Cray solved the problem on their platforms by building high-performance BLAS and LAPACK libraries into their operating systems (Apple Accelerate Framework and Cray libsci). But AFAIK, Windows does not have a BLAS library from Microsoft. Sturla Molden From Steve.Dower at microsoft.com Sat Sep 27 17:38:36 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 27 Sep 2014 15:38:36 +0000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: <20140927173155.49b2878e@fsol> References: <20140927141226.5ad2979d@fsol> , <20140927173155.49b2878e@fsol> Message-ID: The SDK compilers are not easily fixable (I've tried). The installation is basically corrupt as far as the non-x86 compilers are concerned. The package we just put out is exactly the same files as the SDK with those issues fixed. There's no reason to encourage the SDK at all now (which was the point of the whole exercise from my point of view). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Antoine Pitrou Sent: ?9/?27/?2014 8:32 To: python-dev at python.org Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 On Sat, 27 Sep 2014 14:10:48 +0100 Paul Moore wrote: > On 27 September 2014 14:01, Nick Coghlan wrote: > > I personally believe we should treat handling both this and the SDK > > compilers properly as a platform-enablement bug for distutils and > > ensure they work properly with the currently maintained branches > > (including 2.7). > > +1 +1 as well. Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Sep 28 16:45:01 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 28 Sep 2014 15:45:01 +0100 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: On 26 September 2014 19:01, Steve Dower wrote: > You can install the package without requiring administrative privileges and, with the latest version of setuptools (when it releases), use tools such as pip, wheel, or a setup.py file to produce binaries on Windows. >From the setuptools changelog, it seems that setuptools 6.0+ should cover this. However, I just got the following error: >.\ve-2.7\Scripts\pip.exe wheel blist Downloading/unpacking blist Running setup.py (path:C:\Work\Projects\buildwheel\ve-2.7\build\blist\setup.py) egg_info for package blist warning: no files found matching 'blist.rst' Building wheels for collected packages: blist Running setup.py bdist_wheel for blist Destination directory: c:\work\projects\buildwheel\wheelhouse Complete output from command C:\Work\Projects\buildwheel\ve-2.7\Scripts\python.exe -c "import setuptools;__file__='C:\\Work\\Projects\\buildwheel\\ve-2.7\\build\\blist\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d c:\work\projects\buildwheel\wheelhouse: running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-2.7 creating build\lib.win-amd64-2.7\blist copying blist\_btuple.py -> build\lib.win-amd64-2.7\blist copying blist\_sorteddict.py -> build\lib.win-amd64-2.7\blist copying blist\_sortedlist.py -> build\lib.win-amd64-2.7\blist copying blist\__init__.py -> build\lib.win-amd64-2.7\blist running build_ext building 'blist._blist' extension Traceback (most recent call last): File "", line 1, in File "C:\Work\Projects\buildwheel\ve-2.7\build\blist\setup.py", line 52, in long_description=open('README.rst').read() File "C:\Apps\Python27\Lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Apps\Python27\Lib\distutils\dist.py", line 953, in run_commands self.run_command(cmd) File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\wheel\bdist_wheel.py", line 175, in run self.run_command('build') File "C:\Apps\Python27\Lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Apps\Python27\Lib\distutils\command\build.py", line 127, in run self.run_command(cmd_name) File "C:\Apps\Python27\Lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\Apps\Python27\Lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\command\build_ext.py", line 54, in run _build_ext.run(self) File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line 337, in run self.build_extensions() File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line 446, in build_extensions self.build_extension(ext) File "C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\command\build_ext.py", line 187, in build_extension _build_ext.build_extension(self, ext) File "C:\Apps\Python27\Lib\distutils\command\build_ext.py", line 496, in build_extension depends=ext.depends) File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 473, in compile self.initialize() File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 383, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Work\Projects\buildwheel\ve-2.7\lib\site-packages\setuptools\msvc9_support.py", line 52, in query_vcvarsall return unpatched['query_vcvarsall'](version, *args, **kwargs) File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 299, in query_vcvarsall raise ValueError(str(list(result.keys()))) ValueError: [u'path', u'include', u'lib'] ---------------------------------------- Failed building wheel for blist Failed to build blist Cleaning up... PS 15:36 [00:02] C:\Work\Projects\buildwheel >.\ve-2.7\Scripts\pip.exe list pip (1.5.6) setuptools (6.0.1) wheel (0.24.0) This looks suspiciously like http://bugs.python.org/issue7511, which is worrying. Is this not something I should have expected setuptools to have patched? Paul From swdev.bali at gmail.com Mon Sep 29 02:22:43 2014 From: swdev.bali at gmail.com (Eko Wibowo) Date: Mon, 29 Sep 2014 07:22:43 +0700 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: <5428A653.7080708@gmail.com> +1 Wow. After PTVS, this is the next big thing for Python developer on Windows. Hoping that someday Python will be treated by Microsoft just like .NET Framework: so we can easily shipped our own modules, not with a complete Python interpreter and packages Thanks Steve! On 9/27/2014 1:01 AM, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. > > From g.rodola at gmail.com Mon Sep 29 21:12:51 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 29 Sep 2014 21:12:51 +0200 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: On Fri, Sep 26, 2014 at 8:01 PM, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. > Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). > We've produced this package to help library developers build wheels for > Windows, but also to help users unblock themselves when they need to build > C extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: > http://aka.ms/vcpython27 > > This package contains all the tools and headers required to build C > extension modules for Python 2.7 32-bit and 64-bit (note that some > extension modules require 3rd party dependencies such as OpenSSL or libxml2 > that are not included). Other versions of Python built with Visual C++ 2008 > are also supported. > > You can install the package without requiring administrative privileges > and, with the latest version of setuptools (when it releases), use tools > such as pip, wheel, or a setup.py file to produce binaries on Windows. > > Unfortunately, this package isn't necessarily going to help with building > CPython 2.7 itself, as the build process is complicated and Visual Studio > 2008 is practically required. However, as most people aren't building > CPython on Windows, this isn't a huge issue. This compiler package should > be sufficient for most extension modules. > > I should also point out that VC9 is no longer supported by Microsoft. This > means there won't be any improvements or bug fixes coming, and there's no > official support offered. Feel free to contact me directly < > steve.dower at microsoft.com> if there are issues with the package. > > Cheers, > Steve > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com > This is awesome, thanks. I guess Python 2.6 is not supported, isn't it? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Mon Sep 29 18:05:18 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 29 Sep 2014 16:05:18 +0000 Subject: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: References: Message-ID: <98b2cfee4fd94696b2b7228a69fbc6dc@DM2PR0301MB0734.namprd03.prod.outlook.com> Paul Moore wrote: > File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 299, in > query_vcvarsall > > raise ValueError(str(list(result.keys()))) > > ValueError: [u'path', u'include', u'lib'] > > ---------------------------------------- > Failed building wheel for blist > Failed to build blist > Cleaning up... > > > PS 15:36 [00:02] C:\Work\Projects\buildwheel > >.\ve-2.7\Scripts\pip.exe list > pip (1.5.6) > setuptools (6.0.1) > wheel (0.24.0) > > This looks suspiciously like http://bugs.python.org/issue7511, which is > worrying. Is this not something I should have expected setuptools to have > patched? Always embarrassing to have to admit this, but there is a bug in my code... :( distutils validates that four environment variables are set after the vcvarsall.bat script - PATH, INCLUDE, LIB and LIBPATH. I neglected to set LIBPATH for 64-bit targets (LIBPATH is used by the compiler when compiling with /clr, which probably no Python extension has done ever). If you have a dirty machine like mine, then it's probably already set globally, which can really mess up your testing. I'm putting up an updated installer for the compilers that will set this variable, so if you haven't grabbed a copy yet hold off for the next 12 hours or so. Otherwise, you can set your LIBPATH variable to any valid path (or maybe just any value - the compiler may not even look if /clr is not passed). Sorry, Steve > Paul From bkabrda at redhat.com Tue Sep 30 12:13:18 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Tue, 30 Sep 2014 06:13:18 -0400 (EDT) Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> Message-ID: <231225407.29537466.1412071998878.JavaMail.zimbra@redhat.com> ----- Original Message ----- > On 20 September 2014 00:23, Donald Stufft wrote: > > > > On Sep 19, 2014, at 10:16 AM, Barry Warsaw wrote: > > > > If the user wants to invoke Python 3, it's not hard to type 'python3' and I > > think that's the message we should be spreading. That already seems pretty > > ingrained in user habits afaict. > > > > > > My biggest problem with ``python3``, is what happens after 3.9. I know > > Guido > > doesn?t particularly like two digit version numbers and it?s been suggested > > on > > this list that instead of 3.10 we?re likely to move directly into 4.0 > > regardless of > > if it?s a ?big? change or not. > > FWIW, I think we actually do this better on Windows these days, where > PEP 397 made "py" a switchable entry point. I'd like to bring the same > scheme to POSIX systems at some point, but that's a *long* way down > the todo list (like, so far down I can't even see it any more). I'd be > willing to review proposals, though :) We did a similar thing with Ruby in Fedora, it's called Rubypick [1]. Is that a direction worth pursuing? Also, what is actually result of this thread? AFAICS the PEP still hasn't changed. IMO when there's only python3 installed, there should be no /usr/bin/python, which also seems to be aligned with what Guido says. Would it be possible to update the PEP accordingly? Thanks! > Cheers, > Nick. -- Regards, Slavek Kabrda [1] https://github.com/fedora-ruby/rubypick From ncoghlan at gmail.com Tue Sep 30 13:30:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 30 Sep 2014 21:30:56 +1000 Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: <231225407.29537466.1412071998878.JavaMail.zimbra@redhat.com> References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <231225407.29537466.1412071998878.JavaMail.zimbra@redhat.com> Message-ID: On 30 September 2014 20:13, Bohuslav Kabrda wrote: > ----- Original Message ----- >> On 20 September 2014 00:23, Donald Stufft wrote: >> > >> > On Sep 19, 2014, at 10:16 AM, Barry Warsaw wrote: >> > >> > If the user wants to invoke Python 3, it's not hard to type 'python3' and I >> > think that's the message we should be spreading. That already seems pretty >> > ingrained in user habits afaict. >> > >> > >> > My biggest problem with ``python3``, is what happens after 3.9. I know >> > Guido >> > doesn?t particularly like two digit version numbers and it?s been suggested >> > on >> > this list that instead of 3.10 we?re likely to move directly into 4.0 >> > regardless of >> > if it?s a ?big? change or not. >> >> FWIW, I think we actually do this better on Windows these days, where >> PEP 397 made "py" a switchable entry point. I'd like to bring the same >> scheme to POSIX systems at some point, but that's a *long* way down >> the todo list (like, so far down I can't even see it any more). I'd be >> willing to review proposals, though :) > > We did a similar thing with Ruby in Fedora, it's called Rubypick [1]. Is that a direction worth pursuing? Yes, I think so. The general idea would be to take the PEP 397 launcher CLI and propose a similar API for *nix systems: https://docs.python.org/3/using/windows.html#python-launcher-for-windows > Also, what is actually result of this thread? AFAICS the PEP still hasn't changed. IMO when there's only python3 installed, there should be no /usr/bin/python, which also seems to be aligned with what Guido says. Would it be possible to update the PEP accordingly? Thanks for the prompt. I just pushed an update [1,2] to merge the previously incorrect bullet point with the one following it, so it now reads: * The more general ``python`` command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the ``python2`` command (however, note that some distributions have already chosen to have ``python`` implement the ``python3`` command; see the `Rationale`_ and `Migration Notes`_ below). I also pushed a few tweaks to account for the extension of Python 2.7 maintenance, and to change the verb tense to reflect the fact this was implemented ages ago [3]. Cheers, Nick. [1] https://hg.python.org/peps/rev/3d16b0cd10bc [2] https://hg.python.org/peps/rev/32b6619e9259 [3] https://hg.python.org/peps/rev/0418f146b50f -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bkabrda at redhat.com Tue Sep 30 14:23:21 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Tue, 30 Sep 2014 08:23:21 -0400 (EDT) Subject: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke In-Reply-To: References: <890941063.19033444.1411110661474.JavaMail.zimbra@redhat.com> <1775140119.19048378.1411111899608.JavaMail.zimbra@redhat.com> <20140919101620.01189b45@limelight.wooz.org> <180A8F4E-BB85-45EB-8E07-96689C3B0254@stufft.io> <231225407.29537466.1412071998878.JavaMail.zimbra@redhat.com> Message-ID: <214442788.29702317.1412079801630.JavaMail.zimbra@redhat.com> ----- Original Message ----- > On 30 September 2014 20:13, Bohuslav Kabrda wrote: > > ----- Original Message ----- > >> On 20 September 2014 00:23, Donald Stufft wrote: > >> > > >> > On Sep 19, 2014, at 10:16 AM, Barry Warsaw wrote: > >> > > >> > If the user wants to invoke Python 3, it's not hard to type 'python3' > >> > and I > >> > think that's the message we should be spreading. That already seems > >> > pretty > >> > ingrained in user habits afaict. > >> > > >> > > >> > My biggest problem with ``python3``, is what happens after 3.9. I know > >> > Guido > >> > doesn?t particularly like two digit version numbers and it?s been > >> > suggested > >> > on > >> > this list that instead of 3.10 we?re likely to move directly into 4.0 > >> > regardless of > >> > if it?s a ?big? change or not. > >> > >> FWIW, I think we actually do this better on Windows these days, where > >> PEP 397 made "py" a switchable entry point. I'd like to bring the same > >> scheme to POSIX systems at some point, but that's a *long* way down > >> the todo list (like, so far down I can't even see it any more). I'd be > >> willing to review proposals, though :) > > > > We did a similar thing with Ruby in Fedora, it's called Rubypick [1]. Is > > that a direction worth pursuing? > > Yes, I think so. The general idea would be to take the PEP 397 > launcher CLI and propose a similar API for *nix systems: > https://docs.python.org/3/using/windows.html#python-launcher-for-windows Thanks, I'll try looking into this if I get some free time... > > Also, what is actually result of this thread? AFAICS the PEP still hasn't > > changed. IMO when there's only python3 installed, there should be no > > /usr/bin/python, which also seems to be aligned with what Guido says. > > Would it be possible to update the PEP accordingly? > > Thanks for the prompt. I just pushed an update [1,2] to merge the > previously incorrect bullet point with the one following it, so it now > reads: > > * The more general ``python`` command should be installed whenever > any version of Python 2 is installed and should invoke the same version of > Python as the ``python2`` command (however, note that some distributions > have already chosen to have ``python`` implement the ``python3`` > command; see the `Rationale`_ and `Migration Notes`_ below). Exactly what I was looking for, thanks a lot! > I also pushed a few tweaks to account for the extension of Python 2.7 > maintenance, and to change the verb tense to reflect the fact this was > implemented ages ago [3]. > > Cheers, > Nick. > > [1] https://hg.python.org/peps/rev/3d16b0cd10bc > [2] https://hg.python.org/peps/rev/32b6619e9259 > [3] https://hg.python.org/peps/rev/0418f146b50f > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Regards, Slavek Kabrda