From python-checkins at python.org Sat Mar 1 08:04:20 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 08:04:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4xKTogSXNzdWUgIzIwMjQ2?= =?utf-8?q?=3A_Fix_test_failures_on_FreeBSD=2E_Patch_by_Ryan_Smith-Roberts?= =?utf-8?q?=2E?= Message-ID: <3fbds03nWMz7LlJ@mail.python.org> http://hg.python.org/cpython/rev/c25e1442529f changeset: 89430:c25e1442529f branch: 3.1 parent: 89294:fbb4d48046e5 user: Stefan Krah date: Tue Jan 21 22:58:40 2014 +0100 summary: Issue #20246: Fix test failures on FreeBSD. Patch by Ryan Smith-Roberts. files: Lib/test/test_socket.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1430,7 +1430,7 @@ self.assertRaises(ValueError, self.cli_conn.recvfrom_into, buf, 1024) def _testRecvFromIntoSmallBuffer(self): - self.serv_conn.send(MSG*2048) + self.serv_conn.send(MSG) TIPC_STYPE = 2000 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 08:04:21 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 08:04:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzIwMjQ2?= =?utf-8?q?=3A_Fix_test_failures_on_FreeBSD=2E_Patch_by_Ryan_Smith-Roberts?= =?utf-8?q?=2E?= Message-ID: <3fbds15GN6z7LlJ@mail.python.org> http://hg.python.org/cpython/rev/e82dcd700e8c changeset: 89431:e82dcd700e8c branch: 3.2 parent: 89296:29b1eebecb8e user: Stefan Krah date: Tue Jan 21 22:58:40 2014 +0100 summary: Issue #20246: Fix test failures on FreeBSD. Patch by Ryan Smith-Roberts. files: Lib/test/test_socket.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1974,7 +1974,7 @@ self.assertRaises(ValueError, self.cli_conn.recvfrom_into, buf, 1024) def _testRecvFromIntoSmallBuffer(self): - self.serv_conn.send(MSG*2048) + self.serv_conn.send(MSG) TIPC_STYPE = 2000 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 08:04:22 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 08:04:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2?= Message-ID: <3fbds26mwjz7LtG@mail.python.org> http://hg.python.org/cpython/rev/afb76ca87022 changeset: 89432:afb76ca87022 branch: 3.3 parent: 89428:432cb56db05d parent: 89431:e82dcd700e8c user: Benjamin Peterson date: Sat Mar 01 02:03:57 2014 -0500 summary: merge 3.2 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 08:04:24 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 08:04:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fbds41lc1z7Ltb@mail.python.org> http://hg.python.org/cpython/rev/345391bb8ee1 changeset: 89433:345391bb8ee1 parent: 89429:b6e999c8907c parent: 89432:afb76ca87022 user: Benjamin Peterson date: Sat Mar 01 02:04:10 2014 -0500 summary: merge 3.3 files: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Mar 1 09:11:35 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 01 Mar 2014 09:11:35 +0100 Subject: [Python-checkins] Daily reference leaks (b6e999c8907c): sum=-4 Message-ID: results for b6e999c8907c on branch "default" -------------------------------------------- test_site leaked [-2, 2, -2] references, sum=-2 test_site leaked [-2, 2, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogN3_Je4', '-x'] From python-checkins at python.org Sat Mar 1 13:53:52 2014 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 1 Mar 2014 13:53:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2316135=3A_remove_mention?= =?utf-8?q?s_of_OS/2_from_the_documentation?= Message-ID: <3fbncJ53Rkz7LmX@mail.python.org> http://hg.python.org/cpython/rev/5e05d7d3db9c changeset: 89434:5e05d7d3db9c user: Andrew Kuchling date: Sat Mar 01 07:53:28 2014 -0500 summary: #16135: remove mentions of OS/2 from the documentation files: Doc/distutils/apiref.rst | 11 ----------- Doc/library/curses.rst | 2 +- Doc/library/os.rst | 3 +-- Doc/library/socket.rst | 3 +-- Doc/library/undoc.rst | 2 +- 5 files changed, 4 insertions(+), 17 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -853,17 +853,6 @@ port of GCC (same as cygwin in no-cygwin mode). -:mod:`distutils.emxccompiler` --- OS/2 EMX Compiler -=================================================== - -.. module:: distutils.emxccompiler - :synopsis: OS/2 EMX Compiler support - - -This module provides the EMXCCompiler class, a subclass of -:class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2. - - :mod:`distutils.archive_util` --- Archiving utilities ====================================================== diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -12,7 +12,7 @@ de-facto standard for portable advanced terminal handling. While curses is most widely used in the Unix environment, versions are available -for DOS, OS/2, and possibly other systems as well. This extension module is +for Windows, DOS, and possibly other systems as well. This extension module is designed to match the API of ncurses, an open-source curses library hosted on Linux and the BSD variants of Unix. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -2653,7 +2653,7 @@ Fork a child process. Return ``0`` in the child and the child's process id in the parent. If an error occurs :exc:`OSError` is raised. - Note that some platforms including FreeBSD <= 6.3, Cygwin and OS/2 EMX have + Note that some platforms including FreeBSD <= 6.3 and Cygwin have known issues when using fork() from a thread. .. warning:: @@ -2899,7 +2899,6 @@ :manpage:`times(2)` or the corresponding Windows Platform API documentation. On Windows, only :attr:`user` and :attr:`system` are known; the other attributes are zero. - On OS/2, only :attr:`elapsed` is known; the other attributes are zero. Availability: Unix, Windows. diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -6,8 +6,7 @@ This module provides access to the BSD *socket* interface. It is available on -all modern Unix systems, Windows, MacOS, OS/2, and probably additional -platforms. +all modern Unix systems, Windows, MacOS, and probably additional platforms. .. note:: diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst --- a/Doc/library/undoc.rst +++ b/Doc/library/undoc.rst @@ -20,7 +20,7 @@ documented beyond this mention. There's little need to document these. :mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32, Win64, WinCE, and OS/2 platforms. + --- Implementation of :mod:`os.path` on Win32, Win64, and WinCE platforms. :mod:`posixpath` --- Implementation of :mod:`os.path` on POSIX. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 16:34:34 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 16:34:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_fix_test_on_de?= =?utf-8?q?bug_builds_=28closes_=2320731=29?= Message-ID: <3fbs9k6ZRgz7Lp8@mail.python.org> http://hg.python.org/cpython/rev/dcf4fbf446ca changeset: 89435:dcf4fbf446ca branch: 3.3 parent: 89432:afb76ca87022 user: Benjamin Peterson date: Sat Mar 01 10:31:36 2014 -0500 summary: fix test on debug builds (closes #20731) files: Lib/test/test_coding.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_coding.py b/Lib/test/test_coding.py --- a/Lib/test/test_coding.py +++ b/Lib/test/test_coding.py @@ -59,12 +59,13 @@ msg=c.exception.args[0]) def test_20731(self): - sub = subprocess.Popen([sys.executable, + sub = subprocess.Popen([sys.executable, os.path.join(os.path.dirname(__file__), 'coding20731.py')], stderr=subprocess.PIPE) err = sub.communicate()[1] - self.assertEquals(err, b'') + self.assertEqual(sub.returncode, 0) + self.assertNotIn(b'SyntaxError', err) if __name__ == "__main__": unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 16:34:36 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 1 Mar 2014 16:34:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fbs9m14ywz7LmX@mail.python.org> http://hg.python.org/cpython/rev/a4726b5d0365 changeset: 89436:a4726b5d0365 parent: 89434:5e05d7d3db9c parent: 89435:dcf4fbf446ca user: Benjamin Peterson date: Sat Mar 01 10:34:22 2014 -0500 summary: merge 3.3 files: Lib/test/test_source_encoding.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_source_encoding.py b/Lib/test/test_source_encoding.py --- a/Lib/test/test_source_encoding.py +++ b/Lib/test/test_source_encoding.py @@ -65,7 +65,8 @@ 'coding20731.py')], stderr=subprocess.PIPE) err = sub.communicate()[1] - self.assertEquals(err, b'') + self.assertEqual(sub.returncode, 0) + self.assertNotIn(b'SyntaxError', err) def test_error_message(self): compile(b'# -*- coding: iso-8859-15 -*-\n', 'dummy', 'exec') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 16:57:56 2014 From: python-checkins at python.org (brett.cannon) Date: Sat, 1 Mar 2014 16:57:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Bring_PEP_8_in_line_with_PEP_?= =?utf-8?q?257?= Message-ID: <3fbshh5YZTz7Lt3@mail.python.org> http://hg.python.org/peps/rev/5efe00002b3e changeset: 5390:5efe00002b3e user: Brett Cannon date: Sat Mar 01 10:57:51 2014 -0500 summary: Bring PEP 8 in line with PEP 257 files: pep-0008.txt | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -550,7 +550,6 @@ """Return a foobang Optional plotz says to frobnicate the bizbaz first. - """ - For one liner docstrings, it's okay to keep the closing ``"""`` on -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Mar 1 18:11:54 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:11:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_return_types_o?= =?utf-8?q?f_re_functions=2E?= Message-ID: <3fbvL21Sxqz7Lt3@mail.python.org> http://hg.python.org/cpython/rev/962f340f2487 changeset: 89437:962f340f2487 user: R David Murray date: Thu Feb 27 18:32:32 2014 -0500 summary: whatsnew: return types of re functions. files: Doc/whatsnew/3.4.rst | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1987,6 +1987,13 @@ in theory this should not cause backward compatibility issues since the disallowed command forms didn't make any sense and are unlikely to be in use. +* The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and + the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of + :class:``match`` objects now always return a *bytes* object when the string + to be matched is a :term:`bytes-like object`. Previously the return type + matched the input type, so if your code was depending on the return value + being, say, a ``bytearray``, you will need to change your code. + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 18:11:55 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:11:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_tarfile_stat_c?= =?utf-8?q?onstants_removal=2E?= Message-ID: <3fbvL344H3z7LtV@mail.python.org> http://hg.python.org/cpython/rev/8b50f1ba2a2c changeset: 89438:8b50f1ba2a2c user: R David Murray date: Thu Feb 27 18:55:34 2014 -0500 summary: whatsnew: tarfile stat constants removal. files: Doc/whatsnew/3.4.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1811,6 +1811,9 @@ ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called on badly broken OSX systems (see :issue:`18393`). +* The hardcoded copies of certain :mod:`stat` constants that were included in + the :mod:`tarfile` module namespace have been removed. + Porting to Python 3.4 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 18:11:56 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:11:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_compression_li?= =?utf-8?q?braries_support_=27x=27_mode=2E?= Message-ID: <3fbvL45RbBz7Lp8@mail.python.org> http://hg.python.org/cpython/rev/6bf20f05a7e2 changeset: 89439:6bf20f05a7e2 user: R David Murray date: Thu Feb 27 20:05:42 2014 -0500 summary: whatsnew: compression libraries support 'x' mode. files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1609,6 +1609,10 @@ set just the error handler, without changing the default encoding. (Contributed by Serhiy Storchaka in :issue:`18818`.) +* The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now + support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and + Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) + Significant Optimizations ------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 18:11:57 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:11:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_give_Serhiy_cr?= =?utf-8?q?edit_for_audioop_24-bit_sample_support=2E?= Message-ID: <3fbvL56zMtz7LmX@mail.python.org> http://hg.python.org/cpython/rev/ebd3037cde40 changeset: 89440:ebd3037cde40 user: R David Murray date: Fri Feb 28 13:30:27 2014 -0500 summary: whatsnew: give Serhiy credit for audioop 24-bit sample support. files: Doc/whatsnew/3.4.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -547,7 +547,8 @@ audioop ------- -Added support for 24-bit samples (:issue:`12866`). +The module now supports 24-bit samples. (Contributed by Serhiy Storchaka +in :issue:`12866`.) Added the :func:`~audioop.byteswap` function to convert big-endian samples to little-endian and vice versa (:issue:`19641`). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 18:11:59 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:11:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_unittest_disco?= =?utf-8?q?ver_works_on_namespace_packages_=28=2317457=29=2E?= Message-ID: <3fbvL71J3Nz7Lt3@mail.python.org> http://hg.python.org/cpython/rev/57cb8a6e8f10 changeset: 89441:57cb8a6e8f10 user: R David Murray date: Sat Mar 01 11:57:58 2014 -0500 summary: whatsnew: unittest discover works on namespace packages (#17457). I also added the docs that never got committed, editing the patch provided by Claudiu. I collapsed the now three versionchanged directives for discovery into one. I tried several different styles for this. The most obvious is a bulleted list, but currently the only way I could find to produce that is to have *something* appear on the versionchanged line after the colon, which combined with the blank space before the bulleted list just looks wrong. I also tried the current single-line-three-sentences version with bullet characters before each sentence, and I almost went with that as it made it clear that the three sentences are independent. But I decided to just go with the sentences for simplicity reasons. files: Doc/library/unittest.rst | 21 ++++++++++++--------- Doc/whatsnew/3.4.rst | 3 +++ 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -239,9 +239,10 @@ Unittest supports simple test discovery. In order to be compatible with test discovery, all of the test files must be :ref:`modules ` or -:ref:`packages ` importable from the top-level directory of -the project (this means that their filenames must be valid -:ref:`identifiers `). +:ref:`packages ` (including :term:`namespace packages +`) importable from the top-level directory of +the project (this means that their filenames must be valid :ref:`identifiers +`). Test discovery is implemented in :meth:`TestLoader.discover`, but can also be used from the command line. The basic command-line usage is:: @@ -306,6 +307,9 @@ Test modules and packages can customize test loading and discovery by through the `load_tests protocol`_. +.. versionchanged:: 3.4 + Test discovery supports :term:`namespace packages `. + .. _organizing-tests: @@ -1654,12 +1658,11 @@ .. versionchanged:: 3.4 Modules that raise :exc:`SkipTest` on import are recorded as skips, - not errors. - - .. versionchanged:: 3.4 - Paths are sorted before being imported to ensure execution order for a - given test suite is the same even if the underlying file system's ordering - is not dependent on file name like in ext3/4. + not errors. + Discovery works for :term:`namespace packages `. + Paths are sorted before being imported so that execution order is + the same even if the underlying file system's ordering is not + dependent on file name. The following attributes of a :class:`TestLoader` can be configured either by diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1355,6 +1355,9 @@ formatted messages that were logged. (Contributed by Antoine Pitrou in :issue:`18937`.) +Test discovery now works with namespace packages (Contributed by Claudiu Popa +in :issue:`17457`.) + venv ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 18:12:00 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 1 Mar 2014 18:12:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_memoryview_is_?= =?utf-8?q?Sequence_and_supports_Reversed=2E?= Message-ID: <3fbvL82mF3z7LtS@mail.python.org> http://hg.python.org/cpython/rev/c440c5893d09 changeset: 89442:c440c5893d09 user: R David Murray date: Sat Mar 01 12:09:55 2014 -0500 summary: whatsnew: memoryview is Sequence and supports Reversed. files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -377,6 +377,10 @@ references to local variables from the frame. (Contributed by Antoine Pitrou in :issue:`17934`.) +* :class:`memoryview` is now registered as a :class:`Sequence `, + and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan + and Claudiu Popa in :issue:`18690` and :issue:`19078`.) + New Modules =========== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 23:01:25 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 1 Mar 2014 23:01:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320465=3A_Update_O?= =?utf-8?q?S_X_installer_build_to_use_SQLite_3=2E8=2E0=2E1=2E?= Message-ID: <3fc1m50CGqz7LqS@mail.python.org> http://hg.python.org/cpython/rev/5fa3f6d82d61 changeset: 89443:5fa3f6d82d61 user: Ned Deily date: Sat Mar 01 14:00:46 2014 -0800 summary: Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. files: Mac/BuildScript/build-installer.py | 6 +++--- Misc/NEWS | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -283,9 +283,9 @@ ), ), dict( - name="SQLite 3.8.3", - url="http://www.sqlite.org/2014/sqlite-autoconf-3080300.tar.gz", - checksum='11572878dc0ac74ae370367a464ab5cf', + name="SQLite 3.8.3.1", + url="http://www.sqlite.org/2014/sqlite-autoconf-3080301.tar.gz", + checksum='509ff98d8dc9729b618b7e96612079c6', extra_cflags=('-Os ' '-DSQLITE_ENABLE_FTS4 ' '-DSQLITE_ENABLE_FTS3_PARENTHESIS ' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,6 +46,8 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. +- Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 1 23:05:21 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 1 Mar 2014 23:05:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320465=3A_fix_NEWS?= =?utf-8?b?IHR5cG8gKGl0J3MgMy44LjMuMSk=?= Message-ID: <3fc1rd6cCDz7LvR@mail.python.org> http://hg.python.org/cpython/rev/a8470f88e7b4 changeset: 89444:a8470f88e7b4 user: Ned Deily date: Sat Mar 01 14:04:48 2014 -0800 summary: Issue #20465: fix NEWS typo (it's 3.8.3.1) files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,7 +46,7 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. +- Issue #20465: Update OS X installer build to use SQLite 3.8.3.1. Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:14:56 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:14:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogZml4IHRlc3RfcG9z?= =?utf-8?q?ix=2Etest=5Finitgroups_to_work_without_supplemental_groups_=28c?= =?utf-8?q?loses?= Message-ID: <3fc4k862JMz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/82bcc626ffd4 changeset: 89445:82bcc626ffd4 branch: 3.3 parent: 89435:dcf4fbf446ca user: Benjamin Peterson date: Sat Mar 01 19:14:12 2014 -0500 summary: fix test_posix.test_initgroups to work without supplemental groups (closes #20249) files: Lib/test/test_posix.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1144,7 +1144,7 @@ def test_initgroups(self): # find missing group - g = max(self.saved_groups) + 1 + g = max(self.saved_groups or [0]) + 1 name = pwd.getpwuid(posix.getuid()).pw_name posix.initgroups(name, g) self.assertIn(g, posix.getgroups()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:14:58 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:14:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogZml4IHRlc3RfcG9z?= =?utf-8?q?ix=2Etest=5Finitgroups_to_work_without_supplemental_groups_=28c?= =?utf-8?q?loses?= Message-ID: <3fc4kB0bhlz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/1fbec1c44911 changeset: 89446:1fbec1c44911 branch: 2.7 parent: 89418:43c4073cb2e2 user: Benjamin Peterson date: Sat Mar 01 19:14:12 2014 -0500 summary: fix test_posix.test_initgroups to work without supplemental groups (closes #20249) files: Lib/test/test_posix.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -546,7 +546,7 @@ def test_initgroups(self): # find missing group - g = max(self.saved_groups) + 1 + g = max(self.saved_groups or [0]) + 1 name = pwd.getpwuid(posix.getuid()).pw_name posix.initgroups(name, g) self.assertIn(g, posix.getgroups()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:14:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:14:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMjAyNDkp?= Message-ID: <3fc4kC2Nn1z7LlK@mail.python.org> http://hg.python.org/cpython/rev/a0abc7555d0e changeset: 89447:a0abc7555d0e parent: 89444:a8470f88e7b4 parent: 89445:82bcc626ffd4 user: Benjamin Peterson date: Sat Mar 01 19:14:48 2014 -0500 summary: merge 3.3 (#20249) files: Lib/test/test_posix.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1161,7 +1161,7 @@ def test_initgroups(self): # find missing group - g = max(self.saved_groups) + 1 + g = max(self.saved_groups or [0]) + 1 name = pwd.getpwuid(posix.getuid()).pw_name posix.initgroups(name, g) self.assertIn(g, posix.getgroups()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:16:58 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:16:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_add_Chris_Ange?= =?utf-8?q?lico?= Message-ID: <3fc4mV56Dsz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/406851b1814c changeset: 89448:406851b1814c branch: 3.3 parent: 89445:82bcc626ffd4 user: Benjamin Peterson date: Sat Mar 01 19:16:12 2014 -0500 summary: add Chris Angelico files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -40,6 +40,7 @@ Oliver Andrich Ross Andrus Juancarlo A?ez +Chris Angelico J?r?my Anger Ankur Ankan Jon Anglin -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:16:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:16:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_Chris_Ange?= =?utf-8?q?lico?= Message-ID: <3fc4mW6xPrz7LkL@mail.python.org> http://hg.python.org/cpython/rev/893c9e441cd7 changeset: 89449:893c9e441cd7 branch: 2.7 parent: 89446:1fbec1c44911 user: Benjamin Peterson date: Sat Mar 01 19:16:12 2014 -0500 summary: add Chris Angelico files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -33,6 +33,7 @@ Erik Anders?n Oliver Andrich Ross Andrus +Chris Angelico Ankur Ankan Heidi Annexstad ?ric Araujo -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 01:17:01 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 2 Mar 2014 01:17:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fc4mY1fMCz7Lks@mail.python.org> http://hg.python.org/cpython/rev/c07354bbb3c8 changeset: 89450:c07354bbb3c8 parent: 89447:a0abc7555d0e parent: 89448:406851b1814c user: Benjamin Peterson date: Sat Mar 01 19:16:51 2014 -0500 summary: merge 3.3 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -40,6 +40,7 @@ Oliver Andrich Ross Andrus Juancarlo A?ez +Chris Angelico J?r?my Anger Ankur Ankan Jon Anglin -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 07:08:54 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 2 Mar 2014 07:08:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_pydist=2Ejson_should_be_read_?= =?utf-8?q?from_a_valid_container?= Message-ID: <3fcDZZ1SGgz7LjP@mail.python.org> http://hg.python.org/peps/rev/2d41b932cccc changeset: 5391:2d41b932cccc user: Nick Coghlan date: Sun Mar 02 16:08:11 2014 +1000 summary: pydist.json should be read from a valid container files: pep-0426.txt | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -459,7 +459,13 @@ also be a wheel 1.1 format update after this PEP is approved that mandates provision of 2.0+ metadata. -Other tools involved in Python distribution may also use this format. +Note that these metadata files SHOULD NOT be processed if the version of the +containing location is too low to indicate that they are valid. Specifically, +unversioned ``sdist`` archives, unversioned installation database directories +and version 1.0 of the ``wheel`` specification do not cover ``pydist.json`` +files. + +Other tools involved in Python distribution MAY also use this format. As JSON files are generally awkward to edit by hand, it is RECOMMENDED that these metadata files be generated by build tools based on other -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 2 07:08:55 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 2 Mar 2014 07:08:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Postpone_python=2Emetadata=5F?= =?utf-8?q?hooks_for_the_time_being?= Message-ID: <3fcDZb3n1Lz7LjP@mail.python.org> http://hg.python.org/peps/rev/4a34df0faaec changeset: 5392:4a34df0faaec user: Nick Coghlan date: Sun Mar 02 16:08:45 2014 +1000 summary: Postpone python.metadata_hooks for the time being files: pep-0459.txt | 176 --------------------------------------- 1 files changed, 0 insertions(+), 176 deletions(-) diff --git a/pep-0459.txt b/pep-0459.txt --- a/pep-0459.txt +++ b/pep-0459.txt @@ -47,7 +47,6 @@ * ``python.integrator`` * ``python.commands`` * ``python.exports`` -* ``python.metadata_hooks`` All standard extensions are currently at version ``1.0``, and thus the ``extension_metadata`` field may be omitted without losing access to any @@ -512,181 +511,6 @@ } - -The ``metadata_hooks`` extension -================================ - -The ``metadata_hooks`` extension is used to define operations to be -invoked on the distribution in the following situations: - -* a relevant distribution has been installed or changed on the system -* a relevant distribution has been completely uninstalled from the system - -The following subfields determine when hooks are triggered: - -* ``export_groups``: trigger based on named export groups -* ``extensions``: trigger based on named extensions - -Note that distributions *do* trigger their own install hooks, but do -*not* trigger their own uninstall hooks. - - -Hook signatures ---------------- - -The currently defined metadata hooks are: - -* ``postinstall``: run after a relevant distribution has been installed, - upgraded or downgraded on the current system. May also be run as part - of a system state resync operation. If the hook is not defined, it - indicates no distribution specific actions are needed following - installation. -* ``postuninstall``: run after a relevant distribution has been completely - removed from the current system. If the hook is not defined, it indicates - no distribution specific actions are needed following uninstallation. - -The required signatures of these hooks are as follows:: - - def postinstall(current_meta, previous_meta=None): - """Run following installation or upgrade of a relevant distribution - - *current_meta* is the distribution metadata for the version now - installed on the current system - *previous_meta* is either omitted or ``None`` (indicating a fresh - install) or else the distribution metadata for the version that - was previously installed (indicating an upgrade, downgrade or - resynchronisation of the system state). - """ - - def postuninstall(previous_meta): - """Run after complete uninstallation of a relevant distribution - - *previous_meta* is the distribution metadata for the version that - was previously installed on the current system - """ - -Export group hooks ------------------- - -Export group hooks are named after the export group of interest:: - - "export_groups": { - "ComfyChair.plugins": { - "postinstall": "ComfyChair.plugins:install_hook", - "postuininstall": "ComfyChair.plugins:uninstall_hook" - } - } - -The nominated hooks will then be invoked appropriately for any distribution -that publishes that export group as part of their ``python.exports`` -extension metadata. - -A trailing ".*" may be used to request prefix matching rather than -requiring an exact match on the export group name. - - -Extension hooks ---------------- - -Extension hooks are named after the metadata extension of interest:: - - "extensions": { - "python.exports": { - "postinstall": "pip.export_group_hooks:run_install_hooks", - "postuininstall": "pip.export_group_hooks:run_uninstall_hooks" - } - "python.commands": { - "postinstall": "pip.command_hook:install_wrapper_scripts", - } - } - -(Note: this is just an example, but the intent is that pip *could* implement -that functionality that way if it wanted to). - -A trailing ".*" may be used to request prefix matching rather than -requiring an exact match on the extension name. - - -Guidelines for metadata hook invocation ---------------------------------------- - -.. note:: - - Metadata hooks are likely to run with elevated privileges, this needs - to be considered carefully (e.g. by *requiring* that metadata hook - installation be opt in when using the standard tools and running with - elevated privileges). - -The given parameter names are considered part of the hook signature. -Installation tools MUST call metadata hooks solely with keyword arguments. - -When metadata hooks are defined, it is assumed that they MUST be executed -to obtain a properly working installation of the distribution, and to -properly remove the distribution from a system. - -Installation tools MUST ensure the new or updated distribution is fully -installed, and available through the import system and installation database -when invoking install hooks. - -Installation tools MUST ensure the removed distribution is fully uninstalled, -and no longer available through the import system and installation database -when invoking uninstall hooks. - -Installation tools MUST call metadata hooks with full metadata (including -all extensions), rather than only the core metadata. - -Installation tools SHOULD invoke metadata hooks automatically after -installing a distribution from a binary archive. - -When installing from an sdist, source archive or VCS checkout, installation -tools SHOULD create a binary archive using ``setup.py bdist_wheel`` and -then install the binary archive normally (including invocation of any -metadata hooks). Installation tools SHOULD NOT invoke ``setup.py install`` -directly. - -Installation tools SHOULD treat an exception thrown by a metadata install -hook as a failure of the installation and revert any other changes made -to the system. The installed distribution responsible for the hook that -failed should be clearly indicated to the user. - -Installation tools SHOULD provide a warning to the user for any exception -thrown by a metadata uninstall hook, again clearly indicating to the user -the installed distribution that triggered the warning. - -Installation tools MUST NOT silently ignore metadata hooks, as failing -to call these hooks may result in a misconfigured installation that fails -unexpectedly at runtime. Installation tools MAY refuse to install -distributions that define metadata hooks, or require that users -explicitly opt in to permitting the installation of packages that -define such hooks. - - -Guidelines for metadata hook implementations --------------------------------------------- - -The given parameter names are considered part of the hook signature. -Metadata hook implementations MUST use the given parameter names. - -Metadata hooks SHOULD NOT be used to provide functionality that is -expected to be provided by installation tools (such as rewriting of -shebang lines and generation of executable wrappers for Windows). - -Metadata hook implementations MUST NOT make any assumptions regarding the -current working directory when they are invoked, and MUST NOT make -persistent alterations to the working directory or any other process global -state (other than potentially importing additional modules, or other -expected side effects of running the distribution). - -Metadata install hooks have access to the full metadata for the release being -installed, that of the previous/next release (as appropriate), as well as -to all the normal runtime information (such as available imports). Hook -implementations can use this information to perform additional platform -specific installation steps. To check for the presence or absence of -"extras", hook implementations should use the same runtime checks that -would be used during normal operation (such as checking for the availability -of the relevant dependencies). - - Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 2 07:46:39 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 2 Mar 2014 07:46:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_formatting?= Message-ID: <3fcFQ70LPkz7Ljd@mail.python.org> http://hg.python.org/peps/rev/c728cf303050 changeset: 5393:c728cf303050 user: Nick Coghlan date: Sun Mar 02 16:41:09 2014 +1000 summary: Fix formatting files: pep-0461.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -50,7 +50,7 @@ Proposed semantics for ``bytes`` and ``bytearray`` formatting -======================================= +============================================================= %-interpolation --------------- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 2 07:46:40 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 2 Mar 2014 07:46:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Account_for_metadata_hook_def?= =?utf-8?q?erral_in_PEP_426?= Message-ID: <3fcFQ83rRvz7Ljr@mail.python.org> http://hg.python.org/peps/rev/272debef6b77 changeset: 5394:272debef6b77 user: Nick Coghlan date: Sun Mar 02 16:46:28 2014 +1000 summary: Account for metadata hook deferral in PEP 426 files: pep-0426.txt | 428 ++++++++++++++++++++------------------ 1 files changed, 225 insertions(+), 203 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -1200,6 +1200,12 @@ ``extensions`` key. The keys MUST be valid prefixed names, while the values MUST themselves be nested mappings. +Two key names are reserved and MUST NOT be used by extensions, except as +described below: + +* ``extension_version`` +* ``required_extension`` + The following example shows the ``python.details`` and ``python.commands`` standard extensions from :pep:`459`:: @@ -1235,6 +1241,14 @@ the meaning of the extension. This practice will also make it easier to find authoritative documentation for metadata extensions. +Metadata extensions allow development tools to record information in the +metadata that may be useful during later phases of distribution, but is +not essential for dependency resolution or building the software. + + +Extension versioning +-------------------- + Extensions MUST be versioned, using the ``extension_version`` key. However, if this key is omitted, then the implied version is ``1.0``. @@ -1248,9 +1262,24 @@ extension metadata using the lowest metadata version that includes all of the needed fields. -Metadata extensions allow development tools to record information in the -metadata that may be useful during later phases of distribution, but is -not essential for dependency resolution or building the software. + +Required extensions +------------------- + +A project may consider correct handling of some extensions to be essential +to correct installation of the software. This is indicated by setting the +``required_extension`` field to ``true``. Setting it to ``false`` or +omitting it altogether indicates that processing the extension when +installing the distribution is not considered mandatory by the developers. + +Installation tools MUST fail if ``required_extension`` is set to ``true`` +for an extension and the tool does not have any ability to process that +particular extension (whether directly or through a tool-specific plugin +system). + +If an installation tool encounters a required extension it doesn't +understand when attempting to install from a wheel archive, it MAY fall +back on attempting to install from source rather than failing entirely. Extras (optional dependencies) @@ -1569,8 +1598,6 @@ * Updated obsolescence mechanism -* Metadata hook system - * More flexible system for defining contact points and contributors * Defined a recommended set of project URLs @@ -1709,20 +1736,199 @@ particular extensions to be provided as optional features. Possible future uses for extensions include declaration of plugins for -other distributions, hints for automatic conversion to Linux system -packages, and inclusion of CVE references to mark security releases. +other distributions and hints for automatic conversion to Linux system +packages. + +The ability to declare an extension as required is included primarily to +allow the definition of the metadata hooks extension to be deferred until +some time after the initial adoption of the metadata 2.0 specification. If +a distribution needs a ``postinstall`` hook to run in order to complete +the installation successfully, then earlier versions of tools should fall +back to installing from source rather than installing from a wheel file and +then failing to run the expected postinstall hook. + + +Changes to environment markers +------------------------------ + +There are three substantive changes to environment markers in this version: + +* ``platform_release`` was added, as it provides more useful information + than ``platform_version`` on at least Linux and Mac OS X (specifically, + it provides details of the running kernel version) +* ordered comparison of strings is allowed, as this is more useful for + setting minimum and maximum versions where conditional dependencies + are needed or where a platform is supported +* comparison chaining is explicitly allowed, as this becomes useful in the + presence of ordered comparisons + +The other changes to environment markers are just clarifications and +simplifications to make them easier to use. + +The arbitrariness of the choice of ``.`` and ``_`` in the different +variables was addressed by standardising on ``_`` (as these are all +predefined variables rather than live references into the Python module +namespace) + +The use of parentheses for grouping was explicitly noted to address some +underspecified behaviour in the previous version of the specification. + + +Updated contact information +--------------------------- + +This feature is provided by the ``python.project`` and +``python.integrator`` extensions in :pep:`459`. + +The switch to JSON made it possible to provide a more flexible +system for defining multiple contact points for a project, as well as +listing other contributors. + +The ``type`` concept allows for preservation of the distinction between +the original author of a project, and a lead maintainer that takes over +at a later date. + + +Changes to project URLs +----------------------- + +This feature is provided by the ``python.project`` and +``python.integrator`` extensions in :pep:`459`. + +In addition to allow arbitrary strings as project URL labels, the new +metadata standard also defines a recommend set of four URL labels for +a distribution's home page, documentation, source control and issue tracker. + + +Changes to platform support +--------------------------- + +The new environment marker system makes it possible to define supported +platforms in a way that is actually amenable to automated processing. This +has been used to replace several older fields with poorly defined semantics. + +For the moment, the old ``Requires-External`` field has been removed +entirely. The metadata extension mechanism will hopefully prove to be a more +useful replacement. + + +Updated obsolescence mechanism +------------------------------ + +The marker to indicate when a project is obsolete and should be replaced +has been moved to the obsolete project (the new ``obsoleted_by`` field), +replacing the previous marker on the replacement project (the removed +``Obsoletes-Dist`` field). + +This should allow distribution tools to more easily warn users of +obsolete projects and their suggested replacements. + +The ``Obsoletes-Dist`` header is removed rather than deprecated as it +is not widely supported, and so removing it does not present any significant +barrier to tools and projects adopting the new metadata format. + + +Included text documents +----------------------- + +This feature is provided by the ``python.details`` extension in :pep:`459`. + +Currently, PyPI attempts to determine the description's markup format by +rendering it as reStructuredText, and if that fails, treating it as plain +text. + +Furthermore, many projects simply read their long description in from an +existing README file in ``setup.py``. The popularity of this practice is +only expected to increase, as many online version control systems +(including both GitHub and BitBucket) automatically display such files +on the landing page for the project. + +Standardising on the inclusion of the long description as a separate +file in the ``dist-info`` directory allows this to be simplified: + +* An existing file can just be copied into the ``dist-info`` directory as + part of creating the sdist +* The expected markup format can be determined by inspecting the file + extension of the specified path + +Allowing the intended format to be stated explicitly in the path allows +the format guessing to be removed and more informative error reports to be +provided to users when a rendering error occurs. + +This is especially helpful since PyPI applies additional restrictions to +the rendering process for security reasons, thus a description that renders +correctly on a developer's system may still fail to render on the server. + +The document naming system used to achieve this then makes it relatively +straightforward to allow declaration of alternative markup formats like +HTML, Markdown and AsciiDoc through the use of appropriate file +extensions, as well as to define similar included documents for the +project's license and changelog. + +Grouping the included document names into a single top level field gives +automated tools the option of treating them as arbitrary documents without +worrying about their contents. + +Requiring that the included documents be added to the ``dist-info`` metadata +directory means that the complete metadata for the distribution can be +extracted from an sdist or binary archive simply by extracting that +directory, without needing to check for references to other files in the +sdist. + + +Appendix D: Deferred features +============================= + +Several potentially useful features have been deliberately deferred in +order to better prioritise our efforts in migrating to the new metadata +standard. These all reflect information that may be nice to have in the +new metadata, but which can be readily added in metadata 2.1 without +breaking any use cases already supported by metadata 2.0. + +Once the ``pypi``, ``setuptools``, ``pip``, ``wheel`` and ``distlib`` +projects support creation and consumption of metadata 2.0, then we may +revisit the creation of metadata 2.1 with some or all of these additional +features. + + +MIME type registration +---------------------- + +At some point after acceptance of the PEP, I will likely submit the +following MIME type registration requests to IANA: + +* Full metadata: ``application/vnd.python.pydist+json`` +* Essential dependency resolution metadata: + ``application/vnd.python.pydist-dependencies+json`` + +It's even possible we may be able to just register the ``vnd.python`` +namespace under the banner of the PSF rather than having to register +the individual subformats. + + +String methods in environment markers +------------------------------------- + +Supporting at least ".startswith" and ".endswith" string methods in +environment markers would allow some conditions to be written more +naturally. For example, ``"sys.platform.startswith('win')"`` is a +somewhat more intuitive way to mark Windows specific dependencies, +since ``"'win' in sys.platform"`` is incorrect thanks to ``cygwin`` +and the fact that 64-bit Windows still shows up as ``win32`` is more +than a little strange. Support for metadata hooks --------------------------- -This feature is provided by the ``python.metadata_hooks`` extension in -:pep:`459`. - -The new metadata hook system is designed to allow the wheel format to fully -replace direct installation on deployment targets, by allowing projects to -explicitly define code that should be executed following installation from -a wheel file. +While a draft proposal for a `metadata hook system +`__ +has been created, that proposal is not part of the initial set of standard +metadata extensions in PEP 459. + +A metadata hook system would allow the wheel format to fully replace direct +installation on deployment targets, by allowing projects to explicitly +define code that should be executed following installation from a wheel file. This may be something relatively simple, like the `two line refresh `__ @@ -1741,195 +1947,11 @@ meaning they cannot be deferred to implicit execution on first use of the distribution. -The metadata hook and metadata extension systems allow support for such -activities to be pursued independently by the individual platform -communities, while still interoperating with the cross-platform Python -tools. - -Legacy packages that expect to able to run code on target systems using -``setup.py install`` will no longer work correctly. Such packages will -already break when pip 1.4+ is configured to use a wheel cache directory. - - -Changes to environment markers ------------------------------- - -There are three substantive changes to environment markers in this version: - -* ``platform_release`` was added, as it provides more useful information - than ``platform_version`` on at least Linux and Mac OS X (specifically, - it provides details of the running kernel version) -* ordered comparison of strings is allowed, as this is more useful for - setting minimum and maximum versions where conditional dependencies - are needed or where a platform is supported -* comparison chaining is explicitly allowed, as this becomes useful in the - presence of ordered comparisons - -The other changes to environment markers are just clarifications and -simplifications to make them easier to use. - -The arbitrariness of the choice of ``.`` and ``_`` in the different -variables was addressed by standardising on ``_`` (as these are all -predefined variables rather than live references into the Python module -namespace) - -The use of parentheses for grouping was explicitly noted to address some -underspecified behaviour in the previous version of the specification. - - -Updated contact information ---------------------------- - -This feature is provided by the ``python.project`` and -``python.integrator`` extensions in :pep:`459`. - -The switch to JSON made it possible to provide a more flexible -system for defining multiple contact points for a project, as well as -listing other contributors. - -The ``type`` concept allows for preservation of the distinction between -the original author of a project, and a lead maintainer that takes over -at a later date. - - -Changes to project URLs ------------------------ - -This feature is provided by the ``python.project`` and -``python.integrator`` extensions in :pep:`459`. - -In addition to allow arbitrary strings as project URL labels, the new -metadata standard also defines a recommend set of four URL labels for -a distribution's home page, documentation, source control and issue tracker. - - -Changes to platform support ---------------------------- - -The new environment marker system makes it possible to define supported -platforms in a way that is actually amenable to automated processing. This -has been used to replace several older fields with poorly defined semantics. - -For the moment, the old ``Requires-External`` field has been removed -entirely. The combination of explicit support for post install hooks and the -metadata extension mechanism will hopefully prove to be a more useful -replacement. - - -Updated obsolescence mechanism ------------------------------- - -The marker to indicate when a project is obsolete and should be replaced -has been moved to the obsolete project (the new ``obsoleted_by`` field), -replacing the previous marker on the replacement project (the removed -``Obsoletes-Dist`` field). - -This should allow distribution tools to more easily warn users of -obsolete projects and their suggested replacements. - -The ``Obsoletes-Dist`` header is removed rather than deprecated as it -is not widely supported, and so removing it does not present any significant -barrier to tools and projects adopting the new metadata format. - - -Included text documents ------------------------ - -This feature is provided by the ``python.details`` extension in :pep:`459`. - -Currently, PyPI attempts to determine the description's markup format by -rendering it as reStructuredText, and if that fails, treating it as plain -text. - -Furthermore, many projects simply read their long description in from an -existing README file in ``setup.py``. The popularity of this practice is -only expected to increase, as many online version control systems -(including both GitHub and BitBucket) automatically display such files -on the landing page for the project. - -Standardising on the inclusion of the long description as a separate -file in the ``dist-info`` directory allows this to be simplified: - -* An existing file can just be copied into the ``dist-info`` directory as - part of creating the sdist -* The expected markup format can be determined by inspecting the file - extension of the specified path - -Allowing the intended format to be stated explicitly in the path allows -the format guessing to be removed and more informative error reports to be -provided to users when a rendering error occurs. - -This is especially helpful since PyPI applies additional restrictions to -the rendering process for security reasons, thus a description that renders -correctly on a developer's system may still fail to render on the server. - -The document naming system used to achieve this then makes it relatively -straightforward to allow declaration of alternative markup formats like -HTML, Markdown and AsciiDoc through the use of appropriate file -extensions, as well as to define similar included documents for the -project's license and changelog. - -Grouping the included document names into a single top level field gives -automated tools the option of treating them as arbitrary documents without -worrying about their contents. - -Requiring that the included documents be added to the ``dist-info`` metadata -directory means that the complete metadata for the distribution can be -extracted from an sdist or binary archive simply by extracting that -directory, without needing to check for references to other files in the -sdist. - - -Appendix D: Deferred features -============================= - -Several potentially useful features have been deliberately deferred in -order to better prioritise our efforts in migrating to the new metadata -standard. These all reflect information that may be nice to have in the -new metadata, but which can be readily added in metadata 2.1 without -breaking any use cases already supported by metadata 2.0. - -Once the ``pypi``, ``setuptools``, ``pip``, ``wheel`` and ``distlib`` -projects support creation and consumption of metadata 2.0, then we may -revisit the creation of metadata 2.1 with some or all of these additional -features. - - -MIME type registration ----------------------- - -At some point after acceptance of the PEP, I will likely submit the -following MIME type registration requests to IANA: - -* Full metadata: ``application/vnd.python.pydist+json`` -* Essential dependency resolution metadata: - ``application/vnd.python.pydist-dependencies+json`` - -It's even possible we may be able to just register the ``vnd.python`` -namespace under the banner of the PSF rather than having to register -the individual subformats. - - -String methods in environment markers -------------------------------------- - -Supporting at least ".startswith" and ".endswith" string methods in -environment markers would allow some conditions to be written more -naturally. For example, ``"sys.platform.startswith('win')"`` is a -somewhat more intuitive way to mark Windows specific dependencies, -since ``"'win' in sys.platform"`` is incorrect thanks to ``cygwin`` -and the fact that 64-bit Windows still shows up as ``win32`` is more -than a little strange. - - -Additional install hooks ------------------------- - -In addition to the postinstall and postuninstall hooks currently described -in :pep:`459`, other distribution systems (like RPM) include the notion of -preinstall and postuninstall hooks. These hooks would run with the runtime -dependencies installed, but without the distribution itself. These have -been deliberately omitted for the time being. +For the time being, any such system is being left to the realm of tool +specific metadata extensions. This does mean that affected projects may +choose not to publish wheel files, instead continuing to rely on source +distributions until the relevant extension is well defined and widely +supported. Metabuild system -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 2 09:22:52 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQnVtcCB0byAzLjMu?= =?utf-8?q?5rc1=2E?= Message-ID: <3fcHY82Mfnz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/64e58df40786 changeset: 89451:64e58df40786 branch: 3.3 parent: 89332:a69945e43ff3 user: Georg Brandl date: Sun Feb 23 08:30:06 2014 +0100 summary: Bump to 3.3.5rc1. files: Include/patchlevel.h | 8 ++++---- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 3 -#define PY_MICRO_VERSION 4 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 5 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.3.4+" +#define PY_VERSION "3.3.5rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.4" +__version__ = "3.3.5rc1" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.4" +IDLE_VERSION = "3.3.5rc1" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.5 release candidate 1? =============================================== -*Release date: XXXX-XX-XX* +*Release date: 23-Feb-2014* Core and Builtins ----------------- diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.4 +%define version 3.3.5rc1 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.3.4 -============================ +This is Python version 3.3.5 release candidate 1 +================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:53 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzIwNzE5OiBEaXNh?= =?utf-8?q?ble_the_robotparser_python=2Eorg_test_until_the_gzip_encoding_i?= =?utf-8?q?ssue?= Message-ID: <3fcHY93mS2z7LkB@mail.python.org> http://hg.python.org/cpython/rev/540ce9bb19e8 changeset: 89452:540ce9bb19e8 branch: 3.3 parent: 89333:d8f48717b74e user: Georg Brandl date: Sun Feb 23 08:45:15 2014 +0100 summary: #20719: Disable the robotparser python.org test until the gzip encoding issue can be sorted. files: Lib/test/test_robotparser.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -275,6 +275,7 @@ self.skipTest('%s is unavailable' % url) self.assertEqual(parser.can_fetch("*", robots_url), False) + @unittest.skip('does not handle the gzip encoding delivered by pydotorg') def testPythonOrg(self): support.requires('network') with support.transient_internet('www.python.org'): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:54 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge?= Message-ID: <3fcHYB56wyz7Ll3@mail.python.org> http://hg.python.org/cpython/rev/9ec811df548e changeset: 89453:9ec811df548e branch: 3.3 tag: v3.3.5rc1 parent: 89452:540ce9bb19e8 parent: 89451:64e58df40786 user: Georg Brandl date: Sun Feb 23 08:46:00 2014 +0100 summary: merge files: Include/patchlevel.h | 8 ++++---- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 3 -#define PY_MICRO_VERSION 4 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 5 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.3.4+" +#define PY_VERSION "3.3.5rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.4" +__version__ = "3.3.5rc1" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.4" +IDLE_VERSION = "3.3.5rc1" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.5 release candidate 1? =============================================== -*Release date: XXXX-XX-XX* +*Release date: 23-Feb-2014* Core and Builtins ----------------- diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.4 +%define version 3.3.5rc1 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.3.4 -============================ +This is Python version 3.3.5 release candidate 1 +================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:55 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Added_tag_v3?= =?utf-8?q?=2E3=2E5rc1_for_changeset_9ec811df548e?= Message-ID: <3fcHYC6RTKz7Lkp@mail.python.org> http://hg.python.org/cpython/rev/7118f103c593 changeset: 89454:7118f103c593 branch: 3.3 user: Georg Brandl date: Sun Feb 23 08:46:11 2014 +0100 summary: Added tag v3.3.5rc1 for changeset 9ec811df548e files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -120,3 +120,4 @@ c3896275c0f61b2510a6c7e6c458a750359a91b8 v3.3.3 fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 +9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:57 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E3=2E5rc1_release_commits_with_3=2E3_branch?= Message-ID: <3fcHYF1BCnz7LkB@mail.python.org> http://hg.python.org/cpython/rev/e56522b1c42d changeset: 89455:e56522b1c42d branch: 3.3 parent: 89448:406851b1814c parent: 89454:7118f103c593 user: Georg Brandl date: Sun Mar 02 08:54:15 2014 +0100 summary: merge 3.3.5rc1 release commits with 3.3 branch files: .hgtags | 1 + Include/patchlevel.h | 8 ++++---- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Lib/test/test_robotparser.py | 1 + Misc/NEWS | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 4 ++-- 8 files changed, 12 insertions(+), 10 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -120,3 +120,4 @@ c3896275c0f61b2510a6c7e6c458a750359a91b8 v3.3.3 fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 +9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 3 -#define PY_MICRO_VERSION 4 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 5 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.3.4+" +#define PY_VERSION "3.3.5rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.4" +__version__ = "3.3.5rc1" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.4" +IDLE_VERSION = "3.3.5rc1" diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -275,6 +275,7 @@ self.skipTest('%s is unavailable' % url) self.assertEqual(parser.can_fetch("*", robots_url), False) + @unittest.skip('does not handle the gzip encoding delivered by pydotorg') def testPythonOrg(self): support.requires('network') with support.transient_internet('www.python.org'): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.5 release candidate 1? =============================================== -*Release date: XXXX-XX-XX* +*Release date: 23-Feb-2014* Core and Builtins ----------------- diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.4 +%define version 3.3.5rc1 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.3.4 -============================ +This is Python version 3.3.5 release candidate 1 +================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:58 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwNDA0?= =?utf-8?q?=3A_reject_non-text_encodings_early_in_TextIOWrapper=2E?= Message-ID: <3fcHYG44ncz7LlS@mail.python.org> http://hg.python.org/cpython/rev/140a69d950eb changeset: 89456:140a69d950eb branch: 3.3 user: Georg Brandl date: Sun Mar 02 09:18:31 2014 +0100 summary: Issue #20404: reject non-text encodings early in TextIOWrapper. files: Include/codecs.h | 20 +++++++ Lib/_pyio.py | 5 + Lib/test/test_io.py | 30 ++++++++-- Modules/_io/textio.c | 34 ++++++++---- Python/codecs.c | 88 +++++++++++++++++++++++-------- 5 files changed, 136 insertions(+), 41 deletions(-) diff --git a/Include/codecs.h b/Include/codecs.h --- a/Include/codecs.h +++ b/Include/codecs.h @@ -104,7 +104,14 @@ Please note that these APIs are internal and should not be used in Python C extensions. + XXX (ncoghlan): should we make these, or something like them, public + in Python 3.5+? + */ +PyAPI_FUNC(PyObject *) _PyCodec_LookupTextEncoding( + const char *encoding, + const char *alternate_command + ); PyAPI_FUNC(PyObject *) _PyCodec_EncodeText( PyObject *object, @@ -117,6 +124,19 @@ const char *encoding, const char *errors ); + +/* These two aren't actually text encoding specific, but _io.TextIOWrapper + * is the only current API consumer. + */ +PyAPI_FUNC(PyObject *) _PyCodecInfo_GetIncrementalDecoder( + PyObject *codec_info, + const char *errors + ); + +PyAPI_FUNC(PyObject *) _PyCodecInfo_GetIncrementalEncoder( + PyObject *codec_info, + const char *errors + ); #endif diff --git a/Lib/_pyio.py b/Lib/_pyio.py --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -1495,6 +1495,11 @@ if not isinstance(encoding, str): raise ValueError("invalid encoding: %r" % encoding) + if not codecs.lookup(encoding)._is_text_encoding: + msg = ("%r is not a text encoding; " + "use codecs.open() to handle arbitrary codecs") + raise LookupError(msg % encoding) + if errors is None: errors = "strict" else: diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -1955,6 +1955,15 @@ self.assertRaises(TypeError, t.__init__, b, newline=42) self.assertRaises(ValueError, t.__init__, b, newline='xyzzy') + def test_non_text_encoding_codecs_are_rejected(self): + # Ensure the constructor complains if passed a codec that isn't + # marked as a text encoding + # http://bugs.python.org/issue20404 + r = self.BytesIO() + b = self.BufferedWriter(r) + with self.assertRaisesRegex(LookupError, "is not a text encoding"): + self.TextIOWrapper(b, encoding="hex_codec") + def test_detach(self): r = self.BytesIO() b = self.BufferedWriter(r) @@ -2607,15 +2616,22 @@ def test_illegal_decoder(self): # Issue #17106 + # Bypass the early encoding check added in issue 20404 + def _make_illegal_wrapper(): + quopri = codecs.lookup("quopri_codec") + quopri._is_text_encoding = True + try: + t = self.TextIOWrapper(self.BytesIO(b'aaaaaa'), + newline='\n', encoding="quopri_codec") + finally: + quopri._is_text_encoding = False + return t # Crash when decoder returns non-string - t = self.TextIOWrapper(self.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') + t = _make_illegal_wrapper() self.assertRaises(TypeError, t.read, 1) - t = self.TextIOWrapper(self.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') + t = _make_illegal_wrapper() self.assertRaises(TypeError, t.readline) - t = self.TextIOWrapper(self.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') + t = _make_illegal_wrapper() self.assertRaises(TypeError, t.read) @@ -3053,6 +3069,7 @@ class CMiscIOTest(MiscIOTest): io = io + shutdown_error = "RuntimeError: could not find io module state" def test_readinto_buffer_overflow(self): # Issue #18025 @@ -3065,6 +3082,7 @@ class PyMiscIOTest(MiscIOTest): io = pyio + shutdown_error = "LookupError: unknown encoding: ascii" @unittest.skipIf(os.name == 'nt', 'POSIX signals required for this test.') diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -836,7 +836,7 @@ char *kwlist[] = {"buffer", "encoding", "errors", "newline", "line_buffering", "write_through", NULL}; - PyObject *buffer, *raw; + PyObject *buffer, *raw, *codec_info = NULL; char *encoding = NULL; char *errors = NULL; char *newline = NULL; @@ -951,6 +951,17 @@ "could not determine default encoding"); } + /* Check we have been asked for a real text encoding */ + codec_info = _PyCodec_LookupTextEncoding(encoding, "codecs.open()"); + if (codec_info == NULL) { + Py_CLEAR(self->encoding); + goto error; + } + + /* XXX: Failures beyond this point have the potential to leak elements + * of the partially constructed object (like self->encoding) + */ + if (errors == NULL) errors = "strict"; self->errors = PyBytes_FromString(errors); @@ -965,7 +976,7 @@ if (newline) { self->readnl = PyUnicode_FromString(newline); if (self->readnl == NULL) - return -1; + goto error; } self->writetranslate = (newline == NULL || newline[0] != '\0'); if (!self->readuniversal && self->readnl) { @@ -989,8 +1000,8 @@ if (r == -1) goto error; if (r == 1) { - self->decoder = PyCodec_IncrementalDecoder( - encoding, errors); + self->decoder = _PyCodecInfo_GetIncrementalDecoder(codec_info, + errors); if (self->decoder == NULL) goto error; @@ -1014,17 +1025,12 @@ if (r == -1) goto error; if (r == 1) { - PyObject *ci; - self->encoder = PyCodec_IncrementalEncoder( - encoding, errors); + self->encoder = _PyCodecInfo_GetIncrementalEncoder(codec_info, + errors); if (self->encoder == NULL) goto error; /* Get the normalized named of the codec */ - ci = _PyCodec_Lookup(encoding); - if (ci == NULL) - goto error; - res = _PyObject_GetAttrId(ci, &PyId_name); - Py_DECREF(ci); + res = _PyObject_GetAttrId(codec_info, &PyId_name); if (res == NULL) { if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear(); @@ -1044,6 +1050,9 @@ Py_XDECREF(res); } + /* Finished sorting out the codec details */ + Py_DECREF(codec_info); + self->buffer = buffer; Py_INCREF(buffer); @@ -1106,6 +1115,7 @@ return 0; error: + Py_XDECREF(codec_info); return -1; } diff --git a/Python/codecs.c b/Python/codecs.c --- a/Python/codecs.c +++ b/Python/codecs.c @@ -243,20 +243,15 @@ return v; } -/* Helper function to create an incremental codec. */ +/* Helper functions to create an incremental codec. */ +static +PyObject *codec_makeincrementalcodec(PyObject *codec_info, + const char *errors, + const char *attrname) +{ + PyObject *ret, *inccodec; -static -PyObject *codec_getincrementalcodec(const char *encoding, - const char *errors, - const char *attrname) -{ - PyObject *codecs, *ret, *inccodec; - - codecs = _PyCodec_Lookup(encoding); - if (codecs == NULL) - return NULL; - inccodec = PyObject_GetAttrString(codecs, attrname); - Py_DECREF(codecs); + inccodec = PyObject_GetAttrString(codec_info, attrname); if (inccodec == NULL) return NULL; if (errors) @@ -267,6 +262,21 @@ return ret; } +static +PyObject *codec_getincrementalcodec(const char *encoding, + const char *errors, + const char *attrname) +{ + PyObject *codec_info, *ret; + + codec_info = _PyCodec_Lookup(encoding); + if (codec_info == NULL) + return NULL; + ret = codec_makeincrementalcodec(codec_info, errors, attrname); + Py_DECREF(codec_info); + return ret; +} + /* Helper function to create a stream codec. */ static @@ -290,6 +300,24 @@ return streamcodec; } +/* Helpers to work with the result of _PyCodec_Lookup + + */ +PyObject *_PyCodecInfo_GetIncrementalDecoder(PyObject *codec_info, + const char *errors) +{ + return codec_makeincrementalcodec(codec_info, errors, + "incrementaldecoder"); +} + +PyObject *_PyCodecInfo_GetIncrementalEncoder(PyObject *codec_info, + const char *errors) +{ + return codec_makeincrementalcodec(codec_info, errors, + "incrementalencoder"); +} + + /* Convenience APIs to query the Codec registry. All APIs return a codec object with incremented refcount. @@ -447,15 +475,12 @@ } /* Text encoding/decoding API */ -static -PyObject *codec_getitem_checked(const char *encoding, - const char *operation_name, - int index) +PyObject * _PyCodec_LookupTextEncoding(const char *encoding, + const char *alternate_command) { _Py_IDENTIFIER(_is_text_encoding); PyObject *codec; PyObject *attr; - PyObject *v; int is_text_codec; codec = _PyCodec_Lookup(encoding); @@ -482,27 +507,44 @@ Py_DECREF(codec); PyErr_Format(PyExc_LookupError, "'%.400s' is not a text encoding; " - "use codecs.%s() to handle arbitrary codecs", - encoding, operation_name); + "use %s to handle arbitrary codecs", + encoding, alternate_command); return NULL; } } } + /* This appears to be a valid text encoding */ + return codec; +} + + +static +PyObject *codec_getitem_checked(const char *encoding, + const char *alternate_command, + int index) +{ + PyObject *codec; + PyObject *v; + + codec = _PyCodec_LookupTextEncoding(encoding, alternate_command); + if (codec == NULL) + return NULL; + v = PyTuple_GET_ITEM(codec, index); + Py_INCREF(v); Py_DECREF(codec); - Py_INCREF(v); return v; } static PyObject * _PyCodec_TextEncoder(const char *encoding) { - return codec_getitem_checked(encoding, "encode", 0); + return codec_getitem_checked(encoding, "codecs.encode()", 0); } static PyObject * _PyCodec_TextDecoder(const char *encoding) { - return codec_getitem_checked(encoding, "decode", 1); + return codec_getitem_checked(encoding, "codecs.decode()", 1); } PyObject *_PyCodec_EncodeText(PyObject *object, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:22:59 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:22:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Update_NEWS_fo?= =?utf-8?b?ciAzLjMuNXJjMi4=?= Message-ID: <3fcHYH5v5hz7LlF@mail.python.org> http://hg.python.org/cpython/rev/69305c61cd5b changeset: 89457:69305c61cd5b branch: 3.3 user: Georg Brandl date: Sun Mar 02 09:18:41 2014 +0100 summary: Update NEWS for 3.3.5rc2. files: Misc/NEWS | 55 +++++++++++++++++++++++++++++------------- 1 files changed, 38 insertions(+), 17 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,10 +2,10 @@ Python News +++++++++++ -What's New in Python 3.3.5 release candidate 1? +What's New in Python 3.3.5 release candidate 2? =============================================== -*Release date: 23-Feb-2014* +*Release date: 02-Mar-2014* Core and Builtins ----------------- @@ -19,13 +19,6 @@ TypeError for an unexpected output type. (The latter mechanism remains in place for third party non-text encodings) -- Issue #20588: Make Python-ast.c C89 compliant. - -- Issue #20437: Fixed 21 potential bugs when deleting objects references. - -- Issue #20538: UTF-7 incremental decoder produced inconsistant string when - input was truncated in BASE64 section. - Library ------- @@ -38,6 +31,42 @@ spurious crashes or SystemErrors when importing modules or packages from a zip file. The change causing the problem was reverted. +- Issue #20404: io.TextIOWrapper (and hence the open() builtin) now uses the + internal codec marking system added for issue #19619 to throw LookupError + for known non-text encodings at stream construction time. The existing + output type checks remain in place to deal with unmarked third party + codecs. + +Tests +----- + +- Issue #20743: Fix a reference leak in test_tcl. + +Tools/Demos +----------- + +- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. + Patch by Arfrever Frehtes Taifersar Arahesis. + + +What's New in Python 3.3.5 release candidate 1? +=============================================== + +*Release date: 23-Feb-2014* + +Core and Builtins +----------------- + +- Issue #20588: Make Python-ast.c C89 compliant. + +- Issue #20437: Fixed 21 potential bugs when deleting objects references. + +- Issue #20538: UTF-7 incremental decoder produced inconsistant string when + input was truncated in BASE64 section. + +Library +------- + - Issue #20635: Fixed grid_columnconfigure() and grid_rowconfigure() methods of Tkinter widgets to work in wantobjects=True mode. @@ -135,8 +164,6 @@ Tests ----- -- Issue #20743: Fix a reference leak in test_tcl. - - Issue #20510: Rewrote test_exit in test_sys to match existing comments, use modern unittest features, and use helpers from test.script_helper instead of using subprocess directly. Patch by Gareth Rees. @@ -167,12 +194,6 @@ - Issue #20609: Restored the ability to build 64-bit Windows binaries on 32-bit Windows, which was broken by the change in issue #19788. -Tools/Demos ------------ - -- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. - Patch by Arfrever Frehtes Taifersar Arahesis. - What's New in Python 3.3.4? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 09:23:01 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:23:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQnVtcCB0byAzLjMu?= =?utf-8?q?5rc2=2E?= Message-ID: <3fcHYK0Q9kz7LlF@mail.python.org> http://hg.python.org/cpython/rev/ca5635efe090 changeset: 89458:ca5635efe090 branch: 3.3 user: Georg Brandl date: Sun Mar 02 09:19:03 2014 +0100 summary: Bump to 3.3.5rc2. files: Include/patchlevel.h | 4 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 3 #define PY_MICRO_VERSION 5 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_SERIAL 2 /* Version as a string */ -#define PY_VERSION "3.3.5rc1" +#define PY_VERSION "3.3.5rc2" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.5rc1" +__version__ = "3.3.5rc2" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.5rc1" +IDLE_VERSION = "3.3.5rc2" diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.5rc1 +%define version 3.3.5rc2 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,4 +1,4 @@ -This is Python version 3.3.5 release candidate 1 +This is Python version 3.3.5 release candidate 2 ================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Mar 2 09:24:01 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 02 Mar 2014 09:24:01 +0100 Subject: [Python-checkins] Daily reference leaks (c07354bbb3c8): sum=4 Message-ID: results for c07354bbb3c8 on branch "default" -------------------------------------------- test_asyncio leaked [0, 4, 0] memory blocks, sum=4 test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogyWCgdI', '-x'] From python-checkins at python.org Sun Mar 2 09:27:31 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 09:27:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3fcHfW2h2xz7LjX@mail.python.org> http://hg.python.org/cpython/rev/ea03b0fa5bd3 changeset: 89459:ea03b0fa5bd3 parent: 89450:c07354bbb3c8 parent: 89458:ca5635efe090 user: Georg Brandl date: Sun Mar 02 09:28:24 2014 +0100 summary: merge with 3.3 files: .hgtags | 1 + Lib/test/test_robotparser.py | 1 + 2 files changed, 2 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -120,6 +120,7 @@ c3896275c0f61b2510a6c7e6c458a750359a91b8 v3.3.3 fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 +9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 46535f65e7f3bcdcf176f36d34bc1fed719ffd2b v3.4.0a1 9265a2168e2cb2a84785d8717792acc661e6b692 v3.4.0a2 dd9cdf90a5073510877e9dd5112f8e6cf20d5e89 v3.4.0a3 diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -275,6 +275,7 @@ self.skipTest('%s is unavailable' % url) self.assertEqual(parser.can_fetch("*", robots_url), False) + @unittest.skip('does not handle the gzip encoding delivered by pydotorg') def testPythonOrg(self): support.requires('network') with support.transient_internet('www.python.org'): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 10:17:28 2014 From: python-checkins at python.org (terry.reedy) Date: Sun, 2 Mar 2014 10:17:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_issue_=2320728=3A_R?= =?utf-8?q?emove_unused_import_added_in_rev_42366e293b7b_and?= Message-ID: <3fcJm833KGz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/d3732760e3d6 changeset: 89460:d3732760e3d6 user: Terry Jan Reedy date: Sun Mar 02 04:17:01 2014 -0500 summary: Closes issue #20728: Remove unused import added in rev 42366e293b7b and noticed by Claudiu Popa. files: Lib/base64.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/base64.py b/Lib/base64.py --- a/Lib/base64.py +++ b/Lib/base64.py @@ -9,7 +9,6 @@ import re import struct import binascii -import itertools __all__ = [ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 18:03:58 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 18:03:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Add_first_version_of_my_sc?= =?utf-8?q?ript_to_automatically_publish_release_files_on?= Message-ID: <3fcW6Q54fcz7Ljd@mail.python.org> http://hg.python.org/release/rev/e2e377d936fd changeset: 77:e2e377d936fd user: Georg Brandl date: Sun Mar 02 18:04:53 2014 +0100 summary: Add first version of my script to automatically publish release files on python.org. files: add-to-pydotorg.py | 194 +++++++++++++++++++++++++++++++++ 1 files changed, 194 insertions(+), 0 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py new file mode 100644 --- /dev/null +++ b/add-to-pydotorg.py @@ -0,0 +1,194 @@ +""" +Script to add ReleaseFile objects for Python releases on the new pydotorg. + +To use: + +* Copy this script to dinsdale (it needs access to all the release files). + You could also download all files, then you need to adapt the "ftp_root" + string below. + +* Create a new Release object via the Django admin (API is currently broken), + the name MUST be "Python RELEASE". + +* Make sure all download files are in place in the /data/ftp.python.org + subdirectory. + +* Put an AUTH_INFO variable containing "username:api_key" in your environment. + +* Call this script as "python add-to-pydotorg.py RELEASE". + + Each call will remove all previous file objects, so you can call the script + multiple times. +""" + +import os +import re +import sys +import json +import hashlib +from os import path + +import requests + +try: + auth_info = os.environ['AUTH_INFO'] +except KeyError: + print 'Please set an environment variable named AUTH_INFO ' \ + 'containing "username:api_key".' + sys.exit() + +base_url = 'http://127.0.0.1:8124/api/v1/' +ftp_root = '/data/ftp.python.org/pub/python/' +download_root = 'http://www.python.org/ftp/python/' + +headers = {'Authorization': 'ApiKey %s' % auth_info, 'Content-Type': 'application/json'} + +rx = re.compile +# value is (file "name", OS id, file "description") +file_descriptions = [ + (rx(r'\.tgz$'), ('Gzipped source tarball', 3, '')), + (rx(r'\.tar\.xz$'), ('XZ compressed source tarball', 3, '')), + (rx(r'\.amd64\.msi$'), ('Windows x86-64 MSI installer', 1, + 'for AMD64/EM64T/x64, not Itanium processors')), + (rx(r'\.msi$'), ('Windows x86 MSI installer', 1, '')), + (rx(r'\.chm$'), ('Windows help file', 1, '')), + (rx(r'amd64-pdb\.zip$'), ('Windows debug information files for 64-bit binaries', 1, '')), + (rx(r'-pdb\.zip$'), ('Windows debug information files', 1, '')), + (rx(r'-macosx10\.5(_rev\d)?\.dmg$'), ('Mac OS X 32-bit i386/PPC installer', 2, + 'for Mac OS X 10.5 and later')), + (rx(r'-macosx10\.6(_rev\d)?\.dmg$'), ('Mac OS X 64-bit/32-bit installer', 2, + 'for Mac OS X 10.6 and later')), +] + +def changelog_for(release): + new_url = 'http://docs.python.org/release/%s/whatsnew/changelog.html' % release + if requests.head(new_url).status_code != 200: + return 'http://hg.python.org/cpython/file/v%s/Misc/NEWS' % release + +def slug_for(release): + return release[0] + '-' + release[2] + '-' + release[4] + \ + ('-' + release[5:] if release[5:] else '') + +def sigfile_for(release, rfile): + return download_root + '%s/%s.asc' % (release, rfile) + +def md5sum_for(release, rfile): + return hashlib.md5(open(ftp_root + release[:5] + '/' + rfile, 'rb').read()).hexdigest() + +def filesize_for(release, rfile): + return path.getsize(ftp_root + release[:5] + '/' + rfile) + +def make_slug(text): + return re.sub('[^a-zA-Z0-9_-]', '', text.replace(' ', '-')) + +def build_release_dict(release, reldate, is_latest, page_pk): + """Return a dictionary with all needed fields for a Release object.""" + return dict( + name = 'Python ' + release, + slug = slug_for(release), + version = int(release[0]), + is_published = True, + release_date = reldate, # in "YYYY-MM-ddTHH:MM:SS" + pre_release = bool(release[5:]), + release_page = '/api/v1/pages/page/%s/' % page_pk, # XXX doesn't work yet + release_notes_url = changelog_for(release), + show_on_download_page = True, + ) + +def build_file_dict(release, rfile, rel_pk, file_desc, os_pk, add_desc): + """Return a dictionary with all needed fields for a ReleaseFile object.""" + return dict( + name = file_desc, + slug = slug_for(release) + '-' + make_slug(file_desc), + os = '/api/v1/downloads/os/%s/' % os_pk, + release = '/api/v1/downloads/release/%s/' % rel_pk, + description = add_desc, + is_source = os_pk == 3, + url = download_root + '%s/%s' % (release, rfile), + gpg_signature_file = sigfile_for(release, rfile), + md5_sum = md5sum_for(release, rfile), + filesize = filesize_for(release, rfile), + download_button = 'tar.xz' in rfile or + 'macosx10.6.dmg' in rfile or + ('.msi' in rfile and not 'amd64' in rfile), + ) + +def list_files(release): + """List all of the release's download files.""" + reldir = release[:5] + for rfile in os.listdir(path.join(ftp_root, reldir)): + if not path.isfile(path.join(ftp_root, reldir, rfile)): + continue + if rfile.endswith('.asc'): + continue + for prefix in ('python', 'Python'): + if rfile.startswith(prefix): + break + else: + print ' File %s/%s has wrong prefix' % (reldir, rfile) + continue + if rfile.endswith('.chm'): + if rfile[:-4] != 'python' + release.replace('.', ''): + print ' File %s/%s has a different version' % (reldir, rfile) + continue + else: + try: + prefix, rest = rfile.split('-', 1) + except: + prefix, rest = rfile, '' + if not rest.startswith((release + '-', release + '.')): + print ' File %s/%s has a different version' % (reldir, rfile) + continue + for rx, info in file_descriptions: + if rx.search(rfile): + file_desc, os_pk, add_desc = info + yield rfile, file_desc, os_pk, add_desc + break + else: + print ' File %s/%s not recognized' % (reldir, rfile) + continue + +def query_object(objtype, **params): + """Find an API object by query parameters.""" + uri = base_url + 'downloads/%s/' % objtype + uri += '?' + '&'.join('%s=%s' % v for v in params.items()) + resp = requests.get(uri, headers=headers) + if resp.status_code != 200 or not json.loads(resp.text)['objects']: + raise RuntimeError('no object for %s params=%r' % (objtype, params)) + obj = json.loads(resp.text)['objects'][0] + return int(obj['resource_uri'].strip('/').split('/')[-1]) + +def post_object(objtype, datadict): + """Create a new API object.""" + resp = requests.post(base_url + 'downloads/' + objtype + '/', + data=json.dumps(datadict), headers=headers) + if resp.status_code != 201: + try: + print resp.json['traceback'] + except: + pass + raise RuntimeError('creating %s failed: %s' % (objtype, resp.status_code)) + newloc = resp.headers['Location'] + pk = int(newloc.strip('/').split('/')[-1]) + return pk + +def main(): + rel = sys.argv[1] + print 'Querying python.org for release', rel + rel_pk = query_object('release', name='Python+' + rel) + print 'Found Release object: id =', rel_pk + n = 0 + print 'Deleting previous release files' + resp = requests.delete(base_url + 'downloads/release_file/?release=%s' % rel_pk, + headers=headers) + if resp.status_code != 204: + raise RuntimeError('deleting previous releases failed: %s' % resp.status_code) + for rfile, file_desc, os_pk, add_desc in list_files(rel): + print 'Creating ReleaseFile object for', rfile + file_dict = build_file_dict(rel, rfile, rel_pk, file_desc, os_pk, add_desc) + file_pk = post_object('release_file', file_dict) + print 'Created as id =', file_pk + n += 1 + print 'Done - %d files added' % n + +main() -- Repository URL: http://hg.python.org/release From python-checkins at python.org Sun Mar 2 18:25:38 2014 From: python-checkins at python.org (yury.selivanov) Date: Sun, 2 Mar 2014 18:25:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320786=3A_Fix_sign?= =?utf-8?q?atures_for_dict=2E=5F=5Fdelitem=5F=5F_and_property=2E=5F=5Fdele?= =?utf-8?q?te=5F=5F?= Message-ID: <3fcWbQ0T48z7LjX@mail.python.org> http://hg.python.org/cpython/rev/c9861ec8754c changeset: 89461:c9861ec8754c user: Yury Selivanov date: Sun Mar 02 12:25:27 2014 -0500 summary: Issue #20786: Fix signatures for dict.__delitem__ and property.__delete__ files: Lib/test/test_inspect.py | 5 +++++ Misc/NEWS | 3 +++ Objects/typeobject.c | 4 ++-- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1764,6 +1764,11 @@ __call__ = type test_callable(ThisWorksNow()) + # Regression test for issue #20786 + test_unbound_method(dict.__delitem__) + test_unbound_method(property.__delete__) + + @cpython_only @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -14,6 +14,9 @@ - Issue #20637: Key-sharing now also works for instance dictionaries of subclasses. Patch by Peter Ingebretson. +- Issue #20786: Fix signatures for dict.__delitem__ and + property.__delete__ builtins. + Library ------- diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -6192,7 +6192,7 @@ "__set__($self, instance, value, /)\n--\n\nSet an attribute of instance to value."), TPSLOT("__delete__", tp_descr_set, slot_tp_descr_set, wrap_descr_delete, - "__delete__(instance, /)\n--\n\nDelete an attribute of instance."), + "__delete__($self, instance, /)\n--\n\nDelete an attribute of instance."), FLSLOT("__init__", tp_init, slot_tp_init, (wrapperfunc)wrap_init, "__init__($self, /, *args, **kwargs)\n--\n\n" "Initialize self. See help(type(self)) for accurate signature.", @@ -6289,7 +6289,7 @@ "__setitem__($self, key, value, /)\n--\n\nSet self[key] to value."), MPSLOT("__delitem__", mp_ass_subscript, slot_mp_ass_subscript, wrap_delitem, - "__delitem__(key)\n--\n\nDelete self[key]."), + "__delitem__($self, key, /)\n--\n\nDelete self[key]."), SQSLOT("__len__", sq_length, slot_sq_length, wrap_lenfunc, "__len__($self, /)\n--\n\nReturn len(self)."), -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 18:29:37 2014 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 2 Mar 2014 18:29:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Make_PEP_8_follow_PEP_257_on_?= =?utf-8?q?docstring_endings=2E?= Message-ID: <3fcWh10z3Kz7Lk6@mail.python.org> http://hg.python.org/peps/rev/380301e300a6 changeset: 5395:380301e300a6 user: Guido van Rossum date: Sun Mar 02 09:29:33 2014 -0800 summary: Make PEP 8 follow PEP 257 on docstring endings. files: pep-0008.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -545,14 +545,14 @@ - PEP 257 describes good docstring conventions. Note that most importantly, the ``"""`` that ends a multiline docstring should be - on a line by itself, and preferably preceded by a blank line, e.g.:: + on a line by itself, e.g.:: """Return a foobang Optional plotz says to frobnicate the bizbaz first. """ -- For one liner docstrings, it's okay to keep the closing ``"""`` on +- For one liner docstrings, please keep the closing ``"""`` on the same line. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 2 19:12:30 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Mar 2014 19:12:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_a_few_fixes?= Message-ID: <3fcXdV0G66z7Ljj@mail.python.org> http://hg.python.org/release/rev/14aa9229450b changeset: 78:14aa9229450b user: Georg Brandl date: Sun Mar 02 19:13:23 2014 +0100 summary: a few fixes files: add-to-pydotorg.py | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -1,17 +1,16 @@ """ Script to add ReleaseFile objects for Python releases on the new pydotorg. - -To use: +To use (RELEASE is something like 3.3.5rc2): * Copy this script to dinsdale (it needs access to all the release files). You could also download all files, then you need to adapt the "ftp_root" string below. -* Create a new Release object via the Django admin (API is currently broken), - the name MUST be "Python RELEASE". +* Make sure all download files are in place in the correct /data/ftp.python.org + subdirectory. -* Make sure all download files are in place in the /data/ftp.python.org - subdirectory. +* Create a new Release object via the Django admin (adding via API is + currently broken), the name MUST be "Python RELEASE". * Put an AUTH_INFO variable containing "username:api_key" in your environment. @@ -19,6 +18,8 @@ Each call will remove all previous file objects, so you can call the script multiple times. + +Georg Brandl, March 2014. """ import os @@ -37,7 +38,7 @@ 'containing "username:api_key".' sys.exit() -base_url = 'http://127.0.0.1:8124/api/v1/' +base_url = 'http://www.python.org/api/v1/' ftp_root = '/data/ftp.python.org/pub/python/' download_root = 'http://www.python.org/ftp/python/' -- Repository URL: http://hg.python.org/release From python-checkins at python.org Sun Mar 2 19:16:35 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 2 Mar 2014 19:16:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320568=3A_Fix_typo?= =?utf-8?q?_in_pip_option=2E?= Message-ID: <3fcXkC3XgMz7LjX@mail.python.org> http://hg.python.org/cpython/rev/16f91d87ff39 changeset: 89462:16f91d87ff39 user: Martin v. L?wis date: Sun Mar 02 19:15:47 2014 +0100 summary: Issue #20568: Fix typo in pip option. files: Tools/msi/msi.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -420,7 +420,7 @@ compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"' lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"' - updatepipargs = r'-m ensurepip -U --default-install' + updatepipargs = r'-m ensurepip -U --default-pip' removepipargs = r'-m ensurepip._uninstall' # See "CustomAction Table" add_data(db, "CustomAction", [ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 19:29:59 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 2 Mar 2014 19:29:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A__Issue_=2320748=3A_Uninsta?= =?utf-8?q?lling_pip_does_not_leave_behind_the_pyc_of?= Message-ID: <3fcY1g1h4Vz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/7b3c40510a08 changeset: 89463:7b3c40510a08 user: Martin v. L?wis date: Sun Mar 02 19:29:19 2014 +0100 summary: Issue #20748: Uninstalling pip does not leave behind the pyc of the uninstaller anymore. files: Misc/NEWS | 3 +++ Tools/msi/msi.py | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,9 @@ Build ----- +- Issue #20748: Uninstalling pip does not leave behind the pyc of + the uninstaller anymore. + - Issue #20568: The Windows installer now installs the unversioned ``pip`` command in addition to the versioned ``pip3`` and ``pip3.4`` commands. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -421,7 +421,7 @@ compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"' lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"' updatepipargs = r'-m ensurepip -U --default-pip' - removepipargs = r'-m ensurepip._uninstall' + removepipargs = r'-B -m ensurepip._uninstall' # See "CustomAction Table" add_data(db, "CustomAction", [ # msidbCustomActionTypeFirstSequence + msidbCustomActionTypeTextData + msidbCustomActionTypeProperty -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 19:43:21 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 2 Mar 2014 19:43:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320465=3A_Update_W?= =?utf-8?q?indows_installer_to_SQLite_3=2E8=2E3=2E1=2E?= Message-ID: <3fcYK552Rfz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/9459f517d854 changeset: 89464:9459f517d854 user: Martin v. L?wis date: Sun Mar 02 19:42:50 2014 +0100 summary: Issue #20465: Update Windows installer to SQLite 3.8.3.1. files: Misc/NEWS | 3 ++- PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/pyproject.props | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 6 +++--- 5 files changed, 8 insertions(+), 7 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -52,7 +52,8 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X installer build to use SQLite 3.8.3.1. +- Issue #20465: Update OS X and Windows installer builds to use + SQLite 3.8.3.1. Tools/Demos ----------- diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -50,7 +50,7 @@ /> $(OutDir)python$(PyDebugExt).exe $(OutDir)kill_python$(PyDebugExt).exe ..\.. - $(externalsDir)\sqlite-3.8.1 + $(externalsDir)\sqlite-3.8.3.1 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.5 $(externalsDir)\openssl-1.0.1e diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -206,7 +206,7 @@ The ssl sub-project does not have the ability to clean the OpenSSL build; if you need to rebuild, you'll have to clean it by hand. _sqlite3 - Wraps SQLite 3.8.1, which is itself built by sqlite3.vcxproj + Wraps SQLite 3.8.3.1, which is itself built by sqlite3.vcxproj Homepage: http://www.sqlite.org/ _tkinter diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -37,9 +37,9 @@ if not exist tk-8.6.1.0 svn export http://svn.python.org/projects/external/tk-8.6.1.0 @rem sqlite3 -if not exist sqlite-3.8.1 ( - rd /s/q sqlite-source-3.7.12 - svn export http://svn.python.org/projects/external/sqlite-3.8.1 +if not exist sqlite-3.8.3.1 ( + rd /s/q sqlite-source-3.8.1 + svn export http://svn.python.org/projects/external/sqlite-3.8.3.1 ) @rem lzma -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 20:29:50 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 2 Mar 2014 20:29:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314512=3A_Launch_p?= =?utf-8?q?ydoc_-b_instead_of_pydocgui=2Epyw_on_Windows=2E?= Message-ID: <3fcZLk4wmnz7LkL@mail.python.org> http://hg.python.org/cpython/rev/de81e0fe4905 changeset: 89465:de81e0fe4905 user: Martin v. L?wis date: Sun Mar 02 20:29:18 2014 +0100 summary: Issue #14512: Launch pydoc -b instead of pydocgui.pyw on Windows. files: Misc/NEWS | 2 ++ Tools/msi/msi.py | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,8 @@ Build ----- +- Issue #14512: Launch pydoc -b instead of pydocgui.pyw on Windows. + - Issue #20748: Uninstalling pip does not leave behind the pyc of the uninstaller anymore. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -1324,8 +1324,6 @@ tcltkshortcuts = [ ("IDLE", "MenuDir", "IDLE|IDLE (Python GUI)", "pythonw.exe", tcltk.id, r'"[TARGETDIR]Lib\idlelib\idle.pyw"', None, None, "python_icon.exe", 0, None, "TARGETDIR"), - ("PyDoc", "MenuDir", "MODDOCS|Module Docs", "pythonw.exe", - tcltk.id, r'"[TARGETDIR]Tools\scripts\pydocgui.pyw"', None, None, "python_icon.exe", 0, None, "TARGETDIR"), ] add_data(db, "Shortcut", tcltkshortcuts + @@ -1340,6 +1338,8 @@ ("Manual", "MenuDir", "MANUAL|Python Manuals", "REGISTRY.doc", "[#%s]" % docfile, None, None, None, None, None, None, None), + ("PyDoc", "MenuDir", "MODDOCS|Module Docs", "python.exe", + default_feature.id, r'-m pydoc -b', None, None, "python_icon.exe", 0, None, "TARGETDIR"), ("Uninstall", "MenuDir", "UNINST|Uninstall Python", "REGISTRY", SystemFolderName+"msiexec", "/x%s" % product_code, None, None, None, None, None, None), -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 2 23:42:07 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 2 Mar 2014 23:42:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_add_some_issue?= =?utf-8?q?_number_annotations=2E?= Message-ID: <3fcfcb5bH9z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/fceec0368cfa changeset: 89466:fceec0368cfa user: R David Murray date: Sun Mar 02 17:14:33 2014 -0500 summary: whatsnew: add some issue number annotations. Add issue number to tick_counter, PyOS_ReadlineFunctionPointer, marshal optimizations (add Antoine's contribution to the entry under the marshal module), and absolute paths in __file__. files: Doc/whatsnew/3.4.rst | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -926,7 +926,8 @@ this "one copy" ability to any object type (including handling recursive references). This reduces both the size of ``.pyc`` files and the amount of memory a module occupies in memory when it is loaded from a ``.pyc`` (or -``.pyo``) file. (Contributed by Kristj?n Valur J?nsson in :issue:`16475`.) +``.pyo``) file. (Contributed by Kristj?n Valur J?nsson in :issue:`16475`, +with additional speedups by Antoine Pitrou in :issue:`19219`.) mmap @@ -1782,7 +1783,8 @@ ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). * The ``PyThreadState.tick_counter`` field has been removed; its value has - been meaningless since Python 3.2, when the "new GIL" was introduced. + been meaningless since Python 3.2, when the "new GIL" was introduced + (:issue:`19199`). * ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. (Contributed by Taras Lyapun in :issue:`15641`.) @@ -1912,7 +1914,7 @@ will not find all keys. A module's ``__file__`` when imported in the current working directory will also now have an absolute path, including when using ``-m`` with the interpreter (this does not influence when the path to a file - is specified on the command-line). + is specified on the command-line) (:issue:`18416`). * The removal of the *strict* argument to :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` changes the meaning of the @@ -2032,7 +2034,8 @@ * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must now be a string allocated by :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a - string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. + string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` + (:issue:`16742`) * :c:func:`PyThread_set_key_value` now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Mar 3 10:01:23 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 03 Mar 2014 10:01:23 +0100 Subject: [Python-checkins] Daily reference leaks (fceec0368cfa): sum=0 Message-ID: results for fceec0368cfa on branch "default" -------------------------------------------- test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog7IoOXy', '-x'] From python-checkins at python.org Mon Mar 3 10:51:20 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 3 Mar 2014 10:51:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_3=2E3=2E5_schedule_update=2E?= Message-ID: <3fcxSm1GHXz7Ll4@mail.python.org> http://hg.python.org/peps/rev/a7ccffe9cd9e changeset: 5396:a7ccffe9cd9e user: Georg Brandl date: Mon Mar 03 10:52:18 2014 +0100 summary: 3.3.5 schedule update. files: pep-0398.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -95,7 +95,8 @@ -------------- - 3.3.5 candidate 1: February 22, 2014 -- 3.3.5 final: March 1, 2014 +- 3.3.5 candidate 2: March 1, 2014 +- 3.3.5 final: March 8, 2014 3.3.6 schedule -------------- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 3 11:22:08 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 3 Mar 2014 11:22:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Fix_errors_in_the_download?= =?utf-8?q?_file_URLs=2E?= Message-ID: <3fcy8J6sWTz7LjM@mail.python.org> http://hg.python.org/release/rev/3688da9b5212 changeset: 79:3688da9b5212 user: Georg Brandl date: Mon Mar 03 11:23:06 2014 +0100 summary: Fix errors in the download file URLs. files: add-to-pydotorg.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -71,7 +71,7 @@ ('-' + release[5:] if release[5:] else '') def sigfile_for(release, rfile): - return download_root + '%s/%s.asc' % (release, rfile) + return download_root + '%s/%s.asc' % (release[:5], rfile) def md5sum_for(release, rfile): return hashlib.md5(open(ftp_root + release[:5] + '/' + rfile, 'rb').read()).hexdigest() @@ -105,7 +105,7 @@ release = '/api/v1/downloads/release/%s/' % rel_pk, description = add_desc, is_source = os_pk == 3, - url = download_root + '%s/%s' % (release, rfile), + url = download_root + '%s/%s' % (release[:5], rfile), gpg_signature_file = sigfile_for(release, rfile), md5_sum = md5sum_for(release, rfile), filesize = filesize_for(release, rfile), -- Repository URL: http://hg.python.org/release From python-checkins at python.org Mon Mar 3 11:58:52 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 3 Mar 2014 11:58:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2320814=3A_doc=3A_F?= =?utf-8?q?ix_=22Pretty_top=22_example_of_tracemalloc?= Message-ID: <3fcyyh6v46z7LjQ@mail.python.org> http://hg.python.org/cpython/rev/a9058b772807 changeset: 89467:a9058b772807 user: Victor Stinner date: Mon Mar 03 11:57:57 2014 +0100 summary: Close #20814: doc: Fix "Pretty top" example of tracemalloc files: Doc/library/tracemalloc.rst | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -200,8 +200,7 @@ # replace "/path/to/module/file.py" with "module/file.py" filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" - % (index, filename, frame.lineno, - stat.size / 1024)) + % (index, filename, frame.lineno, stat.size / 1024)) other = top_stats[limit:] if other: @@ -215,7 +214,7 @@ # ... run your application ... snapshot = tracemalloc.take_snapshot() - display_top(snapshot, 10) + display_top(snapshot) Example of output of the Python test suite:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 3 20:21:11 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 3 Mar 2014 20:21:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Correct_and_im?= =?utf-8?q?prove_comments_in_test=5Ffileinput_=28closes_=2320501=29=2E?= Message-ID: <3fdB6H68C6z7Ljy@mail.python.org> http://hg.python.org/cpython/rev/d37f963394aa changeset: 89468:d37f963394aa branch: 2.7 parent: 89449:893c9e441cd7 user: Serhiy Storchaka date: Mon Mar 03 21:16:27 2014 +0200 summary: Correct and improve comments in test_fileinput (closes #20501). Thanks Vajrasky Kok and Zachary Ware. files: Lib/test/test_fileinput.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py --- a/Lib/test/test_fileinput.py +++ b/Lib/test/test_fileinput.py @@ -228,6 +228,8 @@ self.addCleanup(safe_unlink, TESTFN) fi = FileInput(files=TESTFN, openhook=hook_encoded('ascii'), bufsize=8) + # The most likely failure is a UnicodeDecodeError due to the entire + # file being read when it shouldn't have been. self.assertEqual(fi.readline(), u'A\n') self.assertEqual(fi.readline(), u'B\r\n') self.assertEqual(fi.readline(), u'C\r') @@ -240,11 +242,9 @@ """Unit tests for fileinput.hook_encoded()""" def test_modes(self): - # Unlikely UTF-7 is locale encoding with open(TESTFN, 'wb') as f: + # UTF-7 is a convenient, seldom used encoding f.write('A\nB\r\nC\rD+IKw-') - t1 = TESTFN - #t1 = writeTmp(1, ['A\nB\r\nC\rD+IKw-'], mode='wb') self.addCleanup(safe_unlink, TESTFN) def check(mode, expected_lines): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 3 20:21:13 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 3 Mar 2014 20:21:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Correct_commen?= =?utf-8?q?ts_and_improve_failure_reports_in_test=5Ffileinput_=28closes_?= =?utf-8?b?IzIwNTAxKS4=?= Message-ID: <3fdB6K0v0Nz7Lln@mail.python.org> http://hg.python.org/cpython/rev/204ef3bca9c8 changeset: 89469:204ef3bca9c8 branch: 3.3 parent: 89458:ca5635efe090 user: Serhiy Storchaka date: Mon Mar 03 21:17:17 2014 +0200 summary: Correct comments and improve failure reports in test_fileinput (closes #20501). Thanks Vajrasky Kok and Zachary Ware. files: Lib/test/test_fileinput.py | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py --- a/Lib/test/test_fileinput.py +++ b/Lib/test/test_fileinput.py @@ -269,9 +269,12 @@ with FileInput(files=TESTFN, openhook=hook_encoded('ascii'), bufsize=8) as fi: - self.assertEqual(fi.readline(), 'A\n') - self.assertEqual(fi.readline(), 'B\n') - self.assertEqual(fi.readline(), 'C\n') + try: + self.assertEqual(fi.readline(), 'A\n') + self.assertEqual(fi.readline(), 'B\n') + self.assertEqual(fi.readline(), 'C\n') + except UnicodeDecodeError: + self.fail('Read to end of file') with self.assertRaises(UnicodeDecodeError): # Read to the end of file. list(fi) @@ -854,8 +857,8 @@ self.assertFalse(kwargs) def test_modes(self): - # Unlikely UTF-7 is locale encoding with open(TESTFN, 'wb') as f: + # UTF-7 is a convenient, seldom used encoding f.write(b'A\nB\r\nC\rD+IKw-') self.addCleanup(safe_unlink, TESTFN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 3 20:21:14 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 3 Mar 2014 20:21:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Correct_comments_and_improve_failure_reports_in_test=5Ff?= =?utf-8?q?ileinput_=28closes_=2320501=29=2E?= Message-ID: <3fdB6L2mVMz7LkH@mail.python.org> http://hg.python.org/cpython/rev/c47cc6351ce7 changeset: 89470:c47cc6351ce7 parent: 89467:a9058b772807 parent: 89469:204ef3bca9c8 user: Serhiy Storchaka date: Mon Mar 03 21:19:19 2014 +0200 summary: Correct comments and improve failure reports in test_fileinput (closes #20501). Thanks Vajrasky Kok and Zachary Ware. files: Lib/test/test_fileinput.py | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py --- a/Lib/test/test_fileinput.py +++ b/Lib/test/test_fileinput.py @@ -271,9 +271,12 @@ with FileInput(files=TESTFN, openhook=hook_encoded('ascii'), bufsize=8) as fi: - self.assertEqual(fi.readline(), 'A\n') - self.assertEqual(fi.readline(), 'B\n') - self.assertEqual(fi.readline(), 'C\n') + try: + self.assertEqual(fi.readline(), 'A\n') + self.assertEqual(fi.readline(), 'B\n') + self.assertEqual(fi.readline(), 'C\n') + except UnicodeDecodeError: + self.fail('Read to end of file') with self.assertRaises(UnicodeDecodeError): # Read to the end of file. list(fi) @@ -856,8 +859,8 @@ self.assertFalse(kwargs) def test_modes(self): - # Unlikely UTF-7 is locale encoding with open(TESTFN, 'wb') as f: + # UTF-7 is a convenient, seldom used encoding f.write(b'A\nB\r\nC\rD+IKw-') self.addCleanup(safe_unlink, TESTFN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 3 21:43:08 2014 From: python-checkins at python.org (ethan.furman) Date: Mon, 3 Mar 2014 21:43:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_issue20653=3A_improv?= =?utf-8?q?e_functional_API_docs=3B_minor_code_changes?= Message-ID: <3fdCwr61JYz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/b637064cc696 changeset: 89471:b637064cc696 user: Ethan Furman date: Mon Mar 03 12:42:52 2014 -0800 summary: Close issue20653: improve functional API docs; minor code changes files: Doc/library/enum.rst | 34 +++++++++++++++++++++++++++ Lib/enum.py | 39 ++++++++++++++++++++++--------- 2 files changed, 61 insertions(+), 12 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -374,6 +374,9 @@ With pickle protocol version 4 it is possible to easily pickle enums nested in other classes. +It is possible to modify how Enum members are pickled/unpickled by defining +:meth:`__reduce_ex__` in the enumeration class. + Functional API -------------- @@ -420,6 +423,12 @@ >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__) +.. warning:: + + If :param module: is not supplied, and Enum cannot determine what it is, + the new Enum members will not be unpicklable; to keep errors closer to + the source, pickling will be disabled. + The new pickle protocol 4 also, in some circumstances, relies on :attr:`__qualname__` being set to the location where pickle will be able to find the class. For example, if the class was made available in class @@ -427,6 +436,31 @@ >>> Animals = Enum('Animals', 'ant bee cat dog', qualname='SomeData.Animals') +The complete signature is:: + + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=) + +:param value: What the new Enum class will record as its name. + +:param names: The Enum members. This can be a whitespace or comma seperated +string:: + + 'red green blue', 'red,green,blue', 'red, green, blue' + +(values will start at 1), or an iterator of name, value pairs:: + + [('cyan', 4), ('magenta', 5), ('yellow', 6)] + +or a mapping:: + + {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + +:param module: name of module where new Enum class can be found. + +:param qualname: where in module new Enum class can be found. + +:param type: type to mix in to new Enum class. + Derived Enumerations -------------------- diff --git a/Lib/enum.py b/Lib/enum.py --- a/Lib/enum.py +++ b/Lib/enum.py @@ -115,14 +115,21 @@ # Reverse value->name map for hashable values. enum_class._value2member_map_ = {} - # check for a supported pickle protocols, and if not present sabotage - # pickling, since it won't work anyway. - # if new class implements its own __reduce_ex__, do not sabotage - if classdict.get('__reduce_ex__') is None: + # If a custom type is mixed into the Enum, and it does not know how + # to pickle itself, pickle.dumps will succeed but pickle.loads will + # fail. Rather than have the error show up later and possibly far + # from the source, sabotage the pickle protocol for this class so + # that pickle.dumps also fails. + # + # However, if the new class implements its own __reduce_ex__, do not + # sabotage -- it's on them to make sure it works correctly. We use + # __reduce_ex__ instead of any of the others as it is preferred by + # pickle over __reduce__, and it handles all pickle protocols. + if '__reduce_ex__' not in classdict: if member_type is not object: methods = ('__getnewargs_ex__', '__getnewargs__', '__reduce_ex__', '__reduce__') - if not any(map(member_type.__dict__.get, methods)): + if not any(m in member_type.__dict__ for m in methods): _make_class_unpicklable(enum_class) # instantiate them, checking for duplicates as we go @@ -193,14 +200,22 @@ to an enumeration member (i.e. Color(3)) and for the functional API (i.e. Color = Enum('Color', names='red green blue')). - When used for the functional API: `module`, if set, will be stored in - the new class' __module__ attribute; `qualname`, if set, will be stored - in the new class' __qualname__ attribute; `type`, if set, will be mixed - in as the first base class. + When used for the functional API: - Note: if `module` is not set this routine will attempt to discover the - calling module by walking the frame stack; if this is unsuccessful - the resulting class will not be pickleable. + `value` will be the name of the new class. + + `names` should be either a string of white-space/comma delimited names + (values will start at 1), or an iterator/mapping of name, value pairs. + + `module` should be set to the module this class is being created in; + if it is not set, an attempt to find that module will be made, but if + it fails the class will not be picklable. + + `qualname` should be set to the actual location this class can be found + at in its module; by default it is set to the global scope. If this is + not correct, unpickling will fail in some circumstances. + + `type`, if set, will be mixed in as the first base class. """ if names is None: # simple value lookup -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 4 00:02:05 2014 From: python-checkins at python.org (ethan.furman) Date: Tue, 4 Mar 2014 00:02:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue20653=3A_fix_ReST_for?= =?utf-8?q?_Enum?= Message-ID: <3fdH194TlXz7Lr8@mail.python.org> http://hg.python.org/cpython/rev/54ab95407288 changeset: 89472:54ab95407288 user: Ethan Furman date: Mon Mar 03 15:02:04 2014 -0800 summary: Issue20653: fix ReST for Enum files: Doc/library/enum.rst | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -425,7 +425,7 @@ .. warning:: - If :param module: is not supplied, and Enum cannot determine what it is, + If ``module`` is not supplied, and Enum cannot determine what it is, the new Enum members will not be unpicklable; to keep errors closer to the source, pickling will be disabled. @@ -440,26 +440,26 @@ Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=) -:param value: What the new Enum class will record as its name. +:value: What the new Enum class will record as its name. -:param names: The Enum members. This can be a whitespace or comma seperated -string:: +:names: The Enum members. This can be a whitespace or comma seperated string + (values will start at 1):: - 'red green blue', 'red,green,blue', 'red, green, blue' + 'red green blue' | 'red,green,blue' | 'red, green, blue' -(values will start at 1), or an iterator of name, value pairs:: + or an iterator of (name, value) pairs:: [('cyan', 4), ('magenta', 5), ('yellow', 6)] -or a mapping:: + or a mapping:: {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} -:param module: name of module where new Enum class can be found. +:module: name of module where new Enum class can be found. -:param qualname: where in module new Enum class can be found. +:qualname: where in module new Enum class can be found. -:param type: type to mix in to new Enum class. +:type: type to mix in to new Enum class. Derived Enumerations -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Mar 4 09:51:58 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 04 Mar 2014 09:51:58 +0100 Subject: [Python-checkins] Daily reference leaks (54ab95407288): sum=0 Message-ID: results for 54ab95407288 on branch "default" -------------------------------------------- test_site leaked [0, -2, 2] references, sum=0 test_site leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogINlA7F', '-x'] From python-checkins at python.org Tue Mar 4 11:39:56 2014 From: python-checkins at python.org (nick.coghlan) Date: Tue, 4 Mar 2014 11:39:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2320839=3A_pkgutil?= =?utf-8?q?=2Efind=5Floader_now_uses_importlib=2Eutil=2Efind=5Fspec?= Message-ID: <3fdZVN6xh3z7LrG@mail.python.org> http://hg.python.org/cpython/rev/ea827c809765 changeset: 89473:ea827c809765 user: Nick Coghlan date: Tue Mar 04 20:39:42 2014 +1000 summary: Close #20839: pkgutil.find_loader now uses importlib.util.find_spec files: Doc/library/pkgutil.rst | 17 +++++++++-------- Lib/pkgutil.py | 17 +++++------------ Lib/test/test_pkgutil.py | 19 +++++++++++++++++++ Misc/NEWS | 4 ++++ 4 files changed, 37 insertions(+), 20 deletions(-) diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -74,15 +74,17 @@ Retrieve a :pep:`302` module loader for the given *fullname*. - This is a convenience wrapper around :func:`importlib.find_loader` that - sets the *path* argument correctly when searching for submodules, and - also ensures parent packages (if any) are imported before searching for - submodules. + This is a backwards compatibility wrapper around + :func:`importlib.util.find_spec` that converts most failures to + :exc:`ImportError` and only returns the loader rather than the full + :class:`ModuleSpec`. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. + .. versionchanged:: 3.4 + Updated to be based on :pep:`451` .. function:: get_importer(path_item) @@ -109,14 +111,13 @@ not already imported, its containing package (if any) is imported, in order to establish the package ``__path__``. - This function uses :func:`iter_importers`, and is thus subject to the same - limitations regarding platform-specific special import locations such as the - Windows registry. - .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. + .. versionchanged:: 3.4 + Updated to be based on :pep:`451` + .. function:: iter_importers(fullname='') diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py --- a/Lib/pkgutil.py +++ b/Lib/pkgutil.py @@ -470,29 +470,22 @@ def find_loader(fullname): """Find a PEP 302 "loader" object for fullname - This is s convenience wrapper around :func:`importlib.find_loader` that - sets the *path* argument correctly when searching for submodules, and - also ensures parent packages (if any) are imported before searching for - submodules. + This is a backwards compatibility wrapper around + importlib.util.find_spec that converts most failures to ImportError + and only returns the loader rather than the full spec """ if fullname.startswith('.'): msg = "Relative module name {!r} not supported".format(fullname) raise ImportError(msg) - path = None - pkg_name = fullname.rpartition(".")[0] - if pkg_name: - pkg = importlib.import_module(pkg_name) - path = getattr(pkg, "__path__", None) - if path is None: - return None try: - return importlib.find_loader(fullname, path) + spec = importlib.util.find_spec(fullname) except (ImportError, AttributeError, TypeError, ValueError) as ex: # This hack fixes an impedance mismatch between pkgutil and # importlib, where the latter raises other errors for cases where # pkgutil previously raised ImportError msg = "Error while finding loader for {!r} ({}: {})" raise ImportError(msg.format(fullname, type(ex), ex)) from ex + return spec.loader def extend_path(path, name): diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -334,6 +334,25 @@ self.assertIsNotNone(pkgutil.get_loader("test.support")) self.assertEqual(len(w.warnings), 0) + def test_get_loader_handles_missing_loader_attribute(self): + global __loader__ + this_loader = __loader__ + del __loader__ + try: + with check_warnings() as w: + self.assertIsNotNone(pkgutil.get_loader(__name__)) + self.assertEqual(len(w.warnings), 0) + finally: + __loader__ = this_loader + + + def test_find_loader_avoids_emulation(self): + with check_warnings() as w: + self.assertIsNotNone(pkgutil.find_loader("sys")) + self.assertIsNotNone(pkgutil.find_loader("os")) + self.assertIsNotNone(pkgutil.find_loader("test.support")) + self.assertEqual(len(w.warnings), 0) + def test_get_importer_avoids_emulation(self): # We use an illegal path so *none* of the path hooks should fire with check_warnings() as w: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,10 @@ Library ------- +- Issue #20839: Don't trigger a DeprecationWarning in the still supported + pkgutil.get_loader() API when __loader__ isn't set on a module (nor + when pkgutil.find_loader() is called directly). + - Issue #20778: Fix modulefinder to work with bytecode-only modules. - Issue #20791: copy.copy() now doesn't make a copy when the input is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 4 11:50:40 2014 From: python-checkins at python.org (nick.coghlan) Date: Tue, 4 Mar 2014 11:50:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_464=3A_drop_the_legacy_mi?= =?utf-8?q?rror_checking_API?= Message-ID: <3fdZkm4SFYz7LnS@mail.python.org> http://hg.python.org/peps/rev/8daf4ec34acb changeset: 5397:8daf4ec34acb user: Nick Coghlan date: Tue Mar 04 20:50:03 2014 +1000 summary: PEP 464: drop the legacy mirror checking API files: pep-0464.txt | 85 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 85 insertions(+), 0 deletions(-) diff --git a/pep-0464.txt b/pep-0464.txt new file mode 100644 --- /dev/null +++ b/pep-0464.txt @@ -0,0 +1,85 @@ +PEP: 464 +Title: Removal of the PyPI Mirror Authenticity API +Version: $Revision$ +Last-Modified: $Date$ +Author: Donald Stufft +BDFL-Delegate: Richard Jones +Discussions-To: distutils-sig at python.org +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 02-Mar-2014 +Post-History: 03-Mar-2014 +Replaces: 381 + + +Abstract +======== + +This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity +API, this includes the /serverkey URL and all of the URLs under /serversig. + + +Rationale +========= + +The PyPI mirroring infrastructure (defined in PEP 381) provides a means to +mirror the content of PyPI used by the automatic installers, and as a component +of that, it provides a method for verifying the authenticity of the mirrored +content. + +This PEP proposal the removal of this API due to: + +* No known implementations that utilize this API are known, this includes + `pip `_ and + `setuptools `_. +* Because this API uses DSA it is vulnerable to leaking the private key if + there is *any* bias in the random nonce. +* This API solves one small corner of the trust problem, however the problem + itself is much larger and it would be better to have a fully fledged system, + such as `The Update Framework `_, + instead. + +Due to the issues it has and the lack of use it is the opinion of this PEP +that it does not provide any practical benefit to justify the additional +complexity. + + +Plan for Deprecation & Removal +============================== + +Immediately upon the acceptance of this PEP the Mirror Authenticity API will +be considered deprecated and mirroring agents and installation tools should +stop accessing it. + +Instead of actually removing it from the current code base (PyPI 1.0) the +current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply +not implement this API. This would cause the API to be "removed" when the +switch from 1.0 to 2.0 occurs. + +If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then +this PEP will be implemented in the PyPI 1.0 code base instead (by removing +the associated code). + +No changes will be required in the installers, however PEP 381 compliant +mirroring clients, such as +`bandersnatch `_ and +`pep381client `_ will need to be +updated to no longer attempt to mirror the /serversig URLs. + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 4 12:15:12 2014 From: python-checkins at python.org (nick.coghlan) Date: Tue, 4 Mar 2014 12:15:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Workflow_automation_proposal_?= =?utf-8?q?updates?= Message-ID: <3fdbH46KPRz7Lrw@mail.python.org> http://hg.python.org/peps/rev/bb1aa7f8efdc changeset: 5398:bb1aa7f8efdc user: Nick Coghlan date: Tue Mar 04 21:15:03 2014 +1000 summary: Workflow automation proposal updates - merge gating makes it easier for others to do continuous integration against the CPython development branches - interesting IRC chat with the CTO of RhodeCode last week, updated accordingly files: pep-0462.txt | 36 ++++++++++++++++++++++++++++++++---- 1 files changed, 32 insertions(+), 4 deletions(-) diff --git a/pep-0462.txt b/pep-0462.txt --- a/pep-0462.txt +++ b/pep-0462.txt @@ -232,6 +232,9 @@ (in the case of ``hg.python.org``, the relevant category is "public open source project"). +The RhodeCode developers have also expressed interest in helping out with +ensuring a python.org RhodeCode deployment is successful. + .. _RhodeCode: https://rhodecode.com/ .. _business source license: https://rhodecode.com/licenses @@ -411,13 +414,21 @@ the other sprint participants than it is on things that are sufficiently mechanical that a computer can (and should) handle them. -Finally, with most of the ways to make a mistake when committing a change -automated out of existence, there are substantially fewer new things to +With most of the ways to make a mistake when committing a change +automated out of existence, there are also substantially fewer new things to learn when a contributor is nominated to become a core developer. This should have a dual benefit, both in making the existing core developers more comfortable with granting that additional level of responsibility, and in making new contributors more comfortable with exercising it. +Finally, a more stable default branch in CPython makes it easier for +other Python projects to conduct continuous integration directly against the +main repo, rather than having to wait until we get into the release +candidate phase. At the moment, setting up such a system isn't particularly +attractive, as it would need to include an additional mechanism to wait +until CPython's own Buildbot fleet had indicate that the build was in a +usable state. + Technical Challenges ==================== @@ -434,8 +445,18 @@ User account management ----------------------- -Ideally, RhodeCode's user account management would be integrated with -the existing account management in Roundup. +Ideally we'd be able to offer a single account that spans all python.org +services, including RhodeCode, Roundup/Rietveld, PyPI and the back end for +the new python.org site, but actually implementing that would be a distinct +infrastructure project, independent of this PEP. + +A potentially simpler near term solution would be if RhodeCode's user +account management could be integrated with the existing account management +in Roundup, similar to what was done for Rietveld. However, if that also +turns out to be impractical in the near term, we would merely end up with +another identity silo to be integrated at a later date, suggesting that +this doesn't need to be considered a blocker for an initial RhodeCode +deployment. Preserving existing SSH access and links for Mercurial repositories @@ -784,6 +805,9 @@ workflow changes are not expected to require any significant changes in Roundup or Buildbot). +Unfortunately, the lead RhodeCode developers aren't able to attend PyCon US +this year, or we would have invited them, too. + Acknowledgements ================ @@ -793,6 +817,10 @@ Taylor for additional technical feedback following publication of the initial draft. +Thanks also to Marcin Kuzminski (CTO of RhodeCode) for getting in touch to +express their interest in helping to ensure the success of a RhodeCode +deployment if we choose to proceed down that path. + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 4 18:20:00 2014 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 4 Mar 2014 18:20:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Attempt_to_clean_up_PEP_257_f?= =?utf-8?q?ootnotes=2E?= Message-ID: <3fdlN01Nxlz7Lk8@mail.python.org> http://hg.python.org/peps/rev/4b5a1100c7af changeset: 5399:4b5a1100c7af user: Guido van Rossum date: Tue Mar 04 09:19:58 2014 -0800 summary: Attempt to clean up PEP 257 footnotes. files: pep-0257.txt | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/pep-0257.txt b/pep-0257.txt --- a/pep-0257.txt +++ b/pep-0257.txt @@ -290,13 +290,10 @@ .. [2] PEP 258, Docutils Design Specification, Goodger (http://www.python.org/dev/peps/pep-0258/) -.. [3] Guido van Rossum, Python's creator and Benevolent Dictator For - Life. - .. _Docutils: http://docutils.sourceforge.net/ .. _Python Style Guide: - (http://www.python.org/dev/peps/pep-0008/) + http://www.python.org/dev/peps/pep-0008/ .. _Doc-SIG: http://www.python.org/sigs/doc-sig/ -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 4 20:49:52 2014 From: python-checkins at python.org (donald.stufft) Date: Tue, 4 Mar 2014 20:49:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_the_Post-History_date_of_?= =?utf-8?q?PEP464?= Message-ID: <3fdphw2LYSz7LmR@mail.python.org> http://hg.python.org/peps/rev/ac37e8e47178 changeset: 5400:ac37e8e47178 user: Donald Stufft date: Tue Mar 04 14:49:47 2014 -0500 summary: Fix the Post-History date of PEP464 files: pep-0464.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0464.txt b/pep-0464.txt --- a/pep-0464.txt +++ b/pep-0464.txt @@ -9,7 +9,7 @@ Type: Process Content-Type: text/x-rst Created: 02-Mar-2014 -Post-History: 03-Mar-2014 +Post-History: 04-Mar-2014 Replaces: 381 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 4 21:24:47 2014 From: python-checkins at python.org (donald.stufft) Date: Tue, 4 Mar 2014 21:24:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Slight_rewording_of_PEP464?= Message-ID: <3fdqTC3vfvz7Ljn@mail.python.org> http://hg.python.org/peps/rev/08150dafffd5 changeset: 5401:08150dafffd5 user: Donald Stufft date: Tue Mar 04 15:24:43 2014 -0500 summary: Slight rewording of PEP464 files: pep-0464.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0464.txt b/pep-0464.txt --- a/pep-0464.txt +++ b/pep-0464.txt @@ -28,9 +28,9 @@ of that, it provides a method for verifying the authenticity of the mirrored content. -This PEP proposal the removal of this API due to: +This PEP proposes the removal of this API due to: -* No known implementations that utilize this API are known, this includes +* There are no known implementations that utilize this API, this includes `pip `_ and `setuptools `_. * Because this API uses DSA it is vulnerable to leaking the private key if -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 4 23:08:26 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 4 Mar 2014 23:08:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=2C_Tulip_issue_158?= =?utf-8?q?=3A_Task=2E=5Fstep=28=29_now_also_sets_self_to_None_if_an?= Message-ID: <3fdsmp11VJz7LjN@mail.python.org> http://hg.python.org/cpython/rev/f4a2d0e4e0be changeset: 89474:f4a2d0e4e0be user: Victor Stinner date: Tue Mar 04 23:07:08 2014 +0100 summary: asyncio, Tulip issue 158: Task._step() now also sets self to None if an exception is raised. self is set to None to break a reference cycle. files: Lib/asyncio/tasks.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -325,7 +325,7 @@ 'Task got bad yield: {!r}'.format(result))) finally: self.__class__._current_tasks.pop(self._loop) - self = None + self = None # Needed to break cycles when an exception occurs. def _wakeup(self, future): try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 5 00:26:45 2014 From: python-checkins at python.org (kristjan.jonsson) Date: Wed, 5 Mar 2014 00:26:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_pickling_o?= =?utf-8?q?f_rangeiter=2E__rangeiter=5Fsetstate_would_not_allow_setting_it?= Message-ID: <3fdvW955psz7LnD@mail.python.org> http://hg.python.org/cpython/rev/24d4e52f4f87 changeset: 89475:24d4e52f4f87 branch: 3.3 parent: 89469:204ef3bca9c8 user: Kristj?n Valur J?nsson date: Tue Mar 04 23:19:24 2014 +0000 summary: Fix pickling of rangeiter. rangeiter_setstate would not allow setting it to the exhausted state. files: Lib/test/test_range.py | 12 ++++++++++++ Objects/rangeobject.c | 2 +- 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py --- a/Lib/test/test_range.py +++ b/Lib/test/test_range.py @@ -379,6 +379,18 @@ it = pickle.loads(d) self.assertEqual(list(it), data[1:]) + def test_exhausted_iterator_pickling(self): + r = range(20) + i = iter(r) + while True: + r = next(i) + if r == 19: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + def test_odd_bug(self): # This used to raise a "SystemError: NULL result without error" # because the range validation step was eating the exception diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -1000,7 +1000,7 @@ long index = PyLong_AsLong(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0 || index >= r->len) { + if (index < 0 || index > r->len) { PyErr_SetString(PyExc_ValueError, "index out of range"); return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 5 00:26:46 2014 From: python-checkins at python.org (kristjan.jonsson) Date: Wed, 5 Mar 2014 00:26:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E3?= Message-ID: <3fdvWB6dH9z7LnD@mail.python.org> http://hg.python.org/cpython/rev/23d9daed4b14 changeset: 89476:23d9daed4b14 parent: 89474:f4a2d0e4e0be parent: 89475:24d4e52f4f87 user: Kristj?n Valur J?nsson date: Tue Mar 04 23:22:15 2014 +0000 summary: Merge with 3.3 files: Lib/test/test_range.py | 12 ++++++++++++ Objects/rangeobject.c | 2 +- 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py --- a/Lib/test/test_range.py +++ b/Lib/test/test_range.py @@ -380,6 +380,18 @@ it = pickle.loads(d) self.assertEqual(list(it), data[1:]) + def test_exhausted_iterator_pickling(self): + r = range(20) + i = iter(r) + while True: + r = next(i) + if r == 19: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + def test_odd_bug(self): # This used to raise a "SystemError: NULL result without error" # because the range validation step was eating the exception diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -807,7 +807,7 @@ long index = PyLong_AsLong(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0 || index >= r->len) { + if (index < 0 || index > r->len) { PyErr_SetString(PyExc_ValueError, "index out of range"); return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 5 03:47:46 2014 From: python-checkins at python.org (larry.hastings) Date: Wed, 5 Mar 2014 03:47:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Updated_429_=283=2E4_release_?= =?utf-8?q?schedule=29=2C_including_adding_a_third_RC=2E?= Message-ID: <3fdzz66NF5z7Lw1@mail.python.org> http://hg.python.org/peps/rev/2cf89e9e50a3 changeset: 5402:2cf89e9e50a3 user: Larry Hastings date: Tue Mar 04 18:47:44 2014 -0800 summary: Updated 429 (3.4 release schedule), including adding a third RC. files: pep-0429.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0429.txt b/pep-0429.txt --- a/pep-0429.txt +++ b/pep-0429.txt @@ -44,12 +44,13 @@ - 3.4.0 beta 2: January 5, 2014 - 3.4.0 beta 3: January 26, 2014 - 3.4.0 candidate 1: February 10, 2014 +- 3.4.0 candidate 2: February 23, 2014 (Beta 1 is also "feature freeze"--no new features beyond this point.) The anticipated schedule for future releases: -- 3.4.0 candidate 2: February 23, 2014 +- 3.4.0 candidate 3: March 9, 2014 - 3.4.0 final: March 16, 2014 .. don't forget to update final date above as well -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Wed Mar 5 09:55:23 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 05 Mar 2014 09:55:23 +0100 Subject: [Python-checkins] Daily reference leaks (23d9daed4b14): sum=0 Message-ID: results for 23d9daed4b14 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloga65uPA', '-x'] From python-checkins at python.org Wed Mar 5 16:24:30 2014 From: python-checkins at python.org (kristjan.jonsson) Date: Wed, 5 Mar 2014 16:24:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Make_the_vario?= =?utf-8?q?us_iterators=27_=22setstate=22_sliently_and_consistently_clip_t?= =?utf-8?q?he?= Message-ID: <3ffJmG2K2Yz7Llf@mail.python.org> http://hg.python.org/cpython/rev/3b2c28061184 changeset: 89477:3b2c28061184 branch: 3.3 parent: 89475:24d4e52f4f87 user: Kristj?n Valur J?nsson date: Wed Mar 05 13:47:57 2014 +0000 summary: Make the various iterators' "setstate" sliently and consistently clip the index. This avoids the possibility of setting an iterator to an invalid state. files: Lib/test/test_range.py | 12 ++++++++++ Modules/arraymodule.c | 2 + Objects/bytearrayobject.c | 10 ++++++-- Objects/bytesobject.c | 10 ++++++-- Objects/listobject.c | 2 + Objects/rangeobject.c | 31 +++++++++++++++++++++++--- Objects/tupleobject.c | 4 +- Objects/unicodeobject.c | 10 ++++++-- 8 files changed, 66 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py --- a/Lib/test/test_range.py +++ b/Lib/test/test_range.py @@ -380,6 +380,18 @@ self.assertEqual(list(it), data[1:]) def test_exhausted_iterator_pickling(self): + r = range(2**65, 2**65+2) + i = iter(r) + while True: + r = next(i) + if r == 2**65+1: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + + def test_large_exhausted_iterator_pickling(self): r = range(20) i = iter(r) while True: diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2798,6 +2798,8 @@ return NULL; if (index < 0) index = 0; + else if (index > Py_SIZE(it->ao)) + index = Py_SIZE(it->ao); /* iterator exhausted */ it->index = index; Py_RETURN_NONE; } diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -3043,9 +3043,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyByteArray_GET_SIZE(it->it_seq)) + index = PyByteArray_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2996,9 +2996,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyBytes_GET_SIZE(it->it_seq)) + index = PyBytes_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2803,6 +2803,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; + else if (index > PyList_GET_SIZE(it->it_seq)) + index = PyList_GET_SIZE(it->it_seq); /* iterator exhausted */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -1000,10 +1000,11 @@ long index = PyLong_AsLong(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0 || index > r->len) { - PyErr_SetString(PyExc_ValueError, "index out of range"); - return NULL; - } + /* silently clip the index value */ + if (index < 0) + index = 0; + else if (index > r->len) + index = r->len; /* exhausted iterator */ r->index = index; Py_RETURN_NONE; } @@ -1178,6 +1179,28 @@ static PyObject * longrangeiter_setstate(longrangeiterobject *r, PyObject *state) { + int cmp; + + /* clip the value */ + PyObject *zero = PyLong_FromLong(0); + if (zero == NULL) + return NULL; + cmp = PyObject_RichCompareBool(state, zero, Py_LT); + if (cmp > 0) { + Py_CLEAR(r->index); + r->index = zero; + Py_RETURN_NONE; + } + Py_DECREF(zero); + if (cmp < 0) + return NULL; + + cmp = PyObject_RichCompareBool(r->len, state, Py_LT); + if (cmp < 0) + return NULL; + if (cmp > 0) + state = r->len; + Py_CLEAR(r->index); r->index = state; Py_INCREF(r->index); diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -998,8 +998,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; - else if (it->it_seq != NULL && index > PyTuple_GET_SIZE(it->it_seq)) - index = PyTuple_GET_SIZE(it->it_seq); + else if (index > PyTuple_GET_SIZE(it->it_seq)) + index = PyTuple_GET_SIZE(it->it_seq); /* exhausted iterator */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -14420,9 +14420,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyUnicode_GET_LENGTH(it->it_seq)) + index = PyUnicode_GET_LENGTH(it->it_seq); /* iterator truncated */ + it->it_index = index; + } Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 5 16:24:31 2014 From: python-checkins at python.org (kristjan.jonsson) Date: Wed, 5 Mar 2014 16:24:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Make_the_various_iterators=27_=22setstate=22_sliently_an?= =?utf-8?q?d_consistently_clip_the?= Message-ID: <3ffJmH59Sqz7Lts@mail.python.org> http://hg.python.org/cpython/rev/08cf35d4ef49 changeset: 89478:08cf35d4ef49 parent: 89476:23d9daed4b14 parent: 89477:3b2c28061184 user: Kristj?n Valur J?nsson date: Wed Mar 05 15:23:07 2014 +0000 summary: Make the various iterators' "setstate" sliently and consistently clip the index. This avoids the possibility of setting an iterator to an invalid state. files: Lib/test/test_range.py | 12 ++++++++++ Modules/arraymodule.c | 2 + Objects/bytearrayobject.c | 10 ++++++-- Objects/bytesobject.c | 10 ++++++-- Objects/listobject.c | 2 + Objects/rangeobject.c | 31 +++++++++++++++++++++++--- Objects/tupleobject.c | 4 +- Objects/unicodeobject.c | 10 ++++++-- 8 files changed, 66 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py --- a/Lib/test/test_range.py +++ b/Lib/test/test_range.py @@ -381,6 +381,18 @@ self.assertEqual(list(it), data[1:]) def test_exhausted_iterator_pickling(self): + r = range(2**65, 2**65+2) + i = iter(r) + while True: + r = next(i) + if r == 2**65+1: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + + def test_large_exhausted_iterator_pickling(self): r = range(20) i = iter(r) while True: diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2838,6 +2838,8 @@ return NULL; if (index < 0) index = 0; + else if (index > Py_SIZE(it->ao)) + index = Py_SIZE(it->ao); /* iterator exhausted */ it->index = index; Py_RETURN_NONE; } diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -3025,9 +3025,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyByteArray_GET_SIZE(it->it_seq)) + index = PyByteArray_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2937,9 +2937,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyBytes_GET_SIZE(it->it_seq)) + index = PyBytes_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2811,6 +2811,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; + else if (index > PyList_GET_SIZE(it->it_seq)) + index = PyList_GET_SIZE(it->it_seq); /* iterator exhausted */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -807,10 +807,11 @@ long index = PyLong_AsLong(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0 || index > r->len) { - PyErr_SetString(PyExc_ValueError, "index out of range"); - return NULL; - } + /* silently clip the index value */ + if (index < 0) + index = 0; + else if (index > r->len) + index = r->len; /* exhausted iterator */ r->index = index; Py_RETURN_NONE; } @@ -985,6 +986,28 @@ static PyObject * longrangeiter_setstate(longrangeiterobject *r, PyObject *state) { + int cmp; + + /* clip the value */ + PyObject *zero = PyLong_FromLong(0); + if (zero == NULL) + return NULL; + cmp = PyObject_RichCompareBool(state, zero, Py_LT); + if (cmp > 0) { + Py_CLEAR(r->index); + r->index = zero; + Py_RETURN_NONE; + } + Py_DECREF(zero); + if (cmp < 0) + return NULL; + + cmp = PyObject_RichCompareBool(r->len, state, Py_LT); + if (cmp < 0) + return NULL; + if (cmp > 0) + state = r->len; + Py_CLEAR(r->index); r->index = state; Py_INCREF(r->index); diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -1011,8 +1011,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; - else if (it->it_seq != NULL && index > PyTuple_GET_SIZE(it->it_seq)) - index = PyTuple_GET_SIZE(it->it_seq); + else if (index > PyTuple_GET_SIZE(it->it_seq)) + index = PyTuple_GET_SIZE(it->it_seq); /* exhausted iterator */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -15196,9 +15196,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyUnicode_GET_LENGTH(it->it_seq)) + index = PyUnicode_GET_LENGTH(it->it_seq); /* iterator truncated */ + it->it_index = index; + } Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 00:54:30 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 6 Mar 2014 00:54:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Synchronize_wit?= =?utf-8?q?h_Tulip?= Message-ID: <3ffX4k67LWz7Lkm@mail.python.org> http://hg.python.org/cpython/rev/bf94d79cecfe changeset: 89479:bf94d79cecfe user: Victor Stinner date: Thu Mar 06 00:52:53 2014 +0100 summary: asyncio: Synchronize with Tulip * Issue #159: Fix windows_utils.socketpair() - Use "127.0.0.1" (IPv4) or "::1" (IPv6) host instead of "localhost", because "localhost" may be a different IP address - Reject also invalid arguments: only AF_INET/AF_INET6 with SOCK_STREAM (and proto=0) are supported * Reject add/remove reader/writer when event loop is closed. * Fix ResourceWarning warnings files: Lib/asyncio/selector_events.py | 8 ++ Lib/asyncio/windows_utils.py | 17 +++++- Lib/test/test_asyncio/test_events.py | 27 ++++++++++ Lib/test/test_asyncio/test_windows_utils.py | 27 ++++++++- 4 files changed, 73 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -136,6 +136,8 @@ def add_reader(self, fd, callback, *args): """Add a reader callback.""" + if self._selector is None: + raise RuntimeError('Event loop is closed') handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) @@ -151,6 +153,8 @@ def remove_reader(self, fd): """Remove a reader callback.""" + if self._selector is None: + return False try: key = self._selector.get_key(fd) except KeyError: @@ -171,6 +175,8 @@ def add_writer(self, fd, callback, *args): """Add a writer callback..""" + if self._selector is None: + raise RuntimeError('Event loop is closed') handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) @@ -186,6 +192,8 @@ def remove_writer(self, fd): """Remove a writer callback.""" + if self._selector is None: + return False try: key = self._selector.get_key(fd) except KeyError: diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -36,12 +36,25 @@ Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain. """ + if family == socket.AF_INET: + host = '127.0.0.1' + elif family == socket.AF_INET6: + host = '::1' + else: + raise ValueError("Ony AF_INET and AF_INET6 socket address families " + "are supported") + if type != socket.SOCK_STREAM: + raise ValueError("Only SOCK_STREAM socket type is supported") + if proto != 0: + raise ValueError("Only protocol zero is supported") + # We create a connected TCP socket. Note the trick with setblocking(0) # that prevents us from having to create a thread. lsock = socket.socket(family, type, proto) - lsock.bind(('localhost', 0)) + lsock.bind((host, 0)) lsock.listen(1) - addr, port = lsock.getsockname() + # On IPv6, ignore flow_info and scope_id + addr, port = lsock.getsockname()[:2] csock = socket.socket(family, type, proto) csock.setblocking(False) try: diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1326,6 +1326,30 @@ self.assertIn('address must be resolved', str(cm.exception)) + def test_remove_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = test_utils.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.add_reader(r, callback) + loop.add_writer(w, callback) + loop.close() + self.assertFalse(loop.remove_reader(r)) + self.assertFalse(loop.remove_writer(w)) + + def test_add_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = test_utils.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.close() + with self.assertRaises(RuntimeError): + loop.add_reader(r, callback) + with self.assertRaises(RuntimeError): + loop.add_writer(w, callback) + class SubprocessTestsMixin: @@ -1632,6 +1656,9 @@ def test_create_datagram_endpoint(self): raise unittest.SkipTest( "IocpEventLoop does not have create_datagram_endpoint()") + + def test_remove_fds_after_closing(self): + raise unittest.SkipTest("IocpEventLoop does not have add_reader()") else: from asyncio import selectors diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py --- a/Lib/test/test_asyncio/test_windows_utils.py +++ b/Lib/test/test_asyncio/test_windows_utils.py @@ -1,8 +1,10 @@ """Tests for window_utils""" +import socket import sys import test.support import unittest +from test.support import IPV6_ENABLED from unittest import mock if sys.platform != 'win32': @@ -16,23 +18,40 @@ class WinsocketpairTests(unittest.TestCase): - def test_winsocketpair(self): - ssock, csock = windows_utils.socketpair() - + def check_winsocketpair(self, ssock, csock): csock.send(b'xxx') self.assertEqual(b'xxx', ssock.recv(1024)) - csock.close() ssock.close() + def test_winsocketpair(self): + ssock, csock = windows_utils.socketpair() + self.check_winsocketpair(ssock, csock) + + @unittest.skipUnless(IPV6_ENABLED, 'IPv6 not supported or enabled') + def test_winsocketpair_ipv6(self): + ssock, csock = windows_utils.socketpair(family=socket.AF_INET6) + self.check_winsocketpair(ssock, csock) + @mock.patch('asyncio.windows_utils.socket') def test_winsocketpair_exc(self, m_socket): + m_socket.AF_INET = socket.AF_INET + m_socket.SOCK_STREAM = socket.SOCK_STREAM m_socket.socket.return_value.getsockname.return_value = ('', 12345) m_socket.socket.return_value.accept.return_value = object(), object() m_socket.socket.return_value.connect.side_effect = OSError() self.assertRaises(OSError, windows_utils.socketpair) + def test_winsocketpair_invalid_args(self): + self.assertRaises(ValueError, + windows_utils.socketpair, family=socket.AF_UNSPEC) + self.assertRaises(ValueError, + windows_utils.socketpair, type=socket.SOCK_DGRAM) + self.assertRaises(ValueError, + windows_utils.socketpair, proto=1) + + class PipeTests(unittest.TestCase): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 01:02:35 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 6 Mar 2014 01:02:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=2C_Tulip_issue_157?= =?utf-8?q?=3A_Improve_test=5Fevents=2Epy=2C_avoid_run=5Fbriefly=28=29_whi?= =?utf-8?q?ch_is?= Message-ID: <3ffXG30bCYz7Lkm@mail.python.org> http://hg.python.org/cpython/rev/56c346e9ae4d changeset: 89480:56c346e9ae4d user: Victor Stinner date: Thu Mar 06 01:00:36 2014 +0100 summary: asyncio, Tulip issue 157: Improve test_events.py, avoid run_briefly() which is not reliable files: Lib/asyncio/test_utils.py | 15 +- Lib/test/test_asyncio/test_events.py | 129 +++++++------- 2 files changed, 71 insertions(+), 73 deletions(-) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -21,10 +21,11 @@ except ImportError: # pragma: no cover ssl = None -from . import tasks from . import base_events from . import events +from . import futures from . import selectors +from . import tasks if sys.platform == 'win32': # pragma: no cover @@ -52,18 +53,14 @@ gen.close() -def run_until(loop, pred, timeout=None): - if timeout is not None: - deadline = time.time() + timeout +def run_until(loop, pred, timeout=30): + deadline = time.time() + timeout while not pred(): if timeout is not None: timeout = deadline - time.time() if timeout <= 0: - return False - loop.run_until_complete(tasks.sleep(timeout, loop=loop)) - else: - run_briefly(loop) - return True + raise futures.TimeoutError() + loop.run_until_complete(tasks.sleep(0.001, loop=loop)) def run_once(loop): diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -56,6 +56,7 @@ class MyBaseProto(asyncio.Protocol): + connected = None done = None def __init__(self, loop=None): @@ -63,12 +64,15 @@ self.state = 'INITIAL' self.nbytes = 0 if loop is not None: + self.connected = asyncio.Future(loop=loop) self.done = asyncio.Future(loop=loop) def connection_made(self, transport): self.transport = transport assert self.state == 'INITIAL', self.state self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) def data_received(self, data): assert self.state == 'CONNECTED', self.state @@ -330,7 +334,8 @@ def test_reader_callback(self): r, w = test_utils.socketpair() - bytes_read = [] + r.setblocking(False) + bytes_read = bytearray() def reader(): try: @@ -340,37 +345,40 @@ # at least on Linux -- see man select. return if data: - bytes_read.append(data) + bytes_read.extend(data) else: self.assertTrue(self.loop.remove_reader(r.fileno())) r.close() self.loop.add_reader(r.fileno(), reader) self.loop.call_soon(w.send, b'abc') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 3) self.loop.call_soon(w.send, b'def') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 6) self.loop.call_soon(w.close) self.loop.call_soon(self.loop.stop) self.loop.run_forever() - self.assertEqual(b''.join(bytes_read), b'abcdef') + self.assertEqual(bytes_read, b'abcdef') def test_writer_callback(self): r, w = test_utils.socketpair() w.setblocking(False) - self.loop.add_writer(w.fileno(), w.send, b'x'*(256*1024)) - test_utils.run_briefly(self.loop) - def remove_writer(): - self.assertTrue(self.loop.remove_writer(w.fileno())) + def writer(data): + w.send(data) + self.loop.stop() - self.loop.call_soon(remove_writer) - self.loop.call_soon(self.loop.stop) + data = b'x' * 1024 + self.loop.add_writer(w.fileno(), writer, data) self.loop.run_forever() + + self.assertTrue(self.loop.remove_writer(w.fileno())) + self.assertFalse(self.loop.remove_writer(w.fileno())) + w.close() - data = r.recv(256*1024) + read = r.recv(len(data) * 2) r.close() - self.assertGreaterEqual(len(data), 200) + self.assertEqual(read, data) def _basetest_sock_client_ops(self, httpd, sock): sock.setblocking(False) @@ -464,10 +472,10 @@ self.assertFalse(self.loop.remove_signal_handler(signal.SIGKILL)) # Now set a handler and handle it. self.loop.add_signal_handler(signal.SIGINT, my_handler) - test_utils.run_briefly(self.loop) + os.kill(os.getpid(), signal.SIGINT) - test_utils.run_briefly(self.loop) - self.assertEqual(caught, 1) + test_utils.run_until(self.loop, lambda: caught) + # Removing it should restore the default handler. self.assertTrue(self.loop.remove_signal_handler(signal.SIGINT)) self.assertEqual(signal.getsignal(signal.SIGINT), @@ -623,7 +631,7 @@ self.assertIn(str(httpd.address), cm.exception.strerror) def test_create_server(self): - proto = MyProto() + proto = MyProto(self.loop) f = self.loop.create_server(lambda: proto, '0.0.0.0', 0) server = self.loop.run_until_complete(f) self.assertEqual(len(server.sockets), 1) @@ -633,14 +641,11 @@ client = socket.socket() client.connect(('127.0.0.1', port)) client.sendall(b'xxx') - test_utils.run_briefly(self.loop) - test_utils.run_until(self.loop, lambda: proto is not None, 10) - self.assertIsInstance(proto, MyProto) - self.assertEqual('INITIAL', proto.state) - test_utils.run_briefly(self.loop) + + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # extra info is available @@ -650,7 +655,7 @@ # close connection proto.transport.close() - test_utils.run_briefly(self.loop) # windows iocp + self.loop.run_until_complete(proto.done) self.assertEqual('CLOSED', proto.state) @@ -672,27 +677,22 @@ @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server(self): - proto = MyProto() + proto = MyProto(loop=self.loop) server, path = self._make_unix_server(lambda: proto) self.assertEqual(len(server.sockets), 1) client = socket.socket(socket.AF_UNIX) client.connect(path) client.sendall(b'xxx') - test_utils.run_briefly(self.loop) - test_utils.run_until(self.loop, lambda: proto is not None, 10) - self.assertIsInstance(proto, MyProto) - self.assertEqual('INITIAL', proto.state) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # close connection proto.transport.close() - test_utils.run_briefly(self.loop) # windows iocp + self.loop.run_until_complete(proto.done) self.assertEqual('CLOSED', proto.state) @@ -735,12 +735,10 @@ client, pr = self.loop.run_until_complete(f_c) client.write(b'xxx') - test_utils.run_briefly(self.loop) - self.assertIsInstance(proto, MyProto) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # extra info is available @@ -774,12 +772,9 @@ client, pr = self.loop.run_until_complete(f_c) client.write(b'xxx') - test_utils.run_briefly(self.loop) - self.assertIsInstance(proto, MyProto) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # close connection @@ -1044,15 +1039,9 @@ self.assertEqual('INITIALIZED', client.state) transport.sendto(b'xxx') - for _ in range(1000): - if server.nbytes: - break - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: server.nbytes) self.assertEqual(3, server.nbytes) - for _ in range(1000): - if client.nbytes: - break - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: client.nbytes) # received self.assertEqual(8, client.nbytes) @@ -1097,11 +1086,11 @@ self.loop.run_until_complete(connect()) os.write(wpipe, b'1') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: proto.nbytes >= 1) self.assertEqual(1, proto.nbytes) os.write(wpipe, b'2345') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: proto.nbytes >= 5) self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) self.assertEqual(5, proto.nbytes) @@ -1166,14 +1155,19 @@ self.assertEqual('CONNECTED', proto.state) transport.write(b'1') - test_utils.run_briefly(self.loop) - data = os.read(rpipe, 1024) + + data = bytearray() + def reader(data): + chunk = os.read(rpipe, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1) self.assertEqual(b'1', data) transport.write(b'2345') - test_utils.run_briefly(self.loop) - data = os.read(rpipe, 1024) - self.assertEqual(b'2345', data) + test_utils.run_until(self.loop, lambda: reader(data) >= 5) + self.assertEqual(b'12345', data) self.assertEqual('CONNECTED', proto.state) os.close(rpipe) @@ -1225,14 +1219,21 @@ self.assertEqual('CONNECTED', proto.state) transport.write(b'1') - test_utils.run_briefly(self.loop) - data = os.read(master, 1024) + + data = bytearray() + def reader(data): + chunk = os.read(master, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1, + timeout=10) self.assertEqual(b'1', data) transport.write(b'2345') - test_utils.run_briefly(self.loop) - data = os.read(master, 1024) - self.assertEqual(b'2345', data) + test_utils.run_until(self.loop, lambda: reader(data) >= 5, + timeout=10) + self.assertEqual(b'12345', data) self.assertEqual('CONNECTED', proto.state) os.close(master) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 02:02:17 2014 From: python-checkins at python.org (thomas.wouters) Date: Thu, 6 Mar 2014 02:02:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_463_to_Chris_Angel?= =?utf-8?q?ico=27s_latest_=28and_supposedly_final=29_version=2E?= Message-ID: <3ffYZx3Gzyz7Lkj@mail.python.org> http://hg.python.org/peps/rev/59653081cdf6 changeset: 5403:59653081cdf6 user: Thomas Wouters date: Thu Mar 06 02:02:00 2014 +0100 summary: Update PEP 463 to Chris Angelico's latest (and supposedly final) version. files: pep-0463.txt | 41 +++++++++++++++++++++++++++++++++++++++- 1 files changed, 40 insertions(+), 1 deletions(-) diff --git a/pep-0463.txt b/pep-0463.txt --- a/pep-0463.txt +++ b/pep-0463.txt @@ -250,7 +250,8 @@ alternatives listed above must (by the nature of functions) evaluate their default values eagerly. The preferred form, using the colon, parallels try/except by using "except exception_list:", and parallels lambda by having -"keyword name_list: subexpression". Using the arrow introduces a token many +"keyword name_list: subexpression"; it also can be read as mapping Exception +to the default value, dict-style. Using the arrow introduces a token many programmers will not be familiar with, and which currently has no similar meaning, but is otherwise quite readable. The English word "pass" has a vaguely similar meaning (consider the common usage "pass by value/reference" @@ -271,6 +272,18 @@ Using the preferred order, subexpressions will always be evaluated from left to right, no matter how the syntax is nested. +Keeping the existing notation, but shifting the mandatory parentheses, we +have the following suggestion:: + + value = expr except (Exception: default) + value = expr except(Exception: default) + +This is reminiscent of a function call, or a dict initializer. The colon +cannot be confused with introducing a suite, but on the other hand, the new +syntax guarantees lazy evaluation, which a dict does not. The potential +to reduce confusion is considered unjustified by the corresponding potential +to increase it. + Example usage ============= @@ -854,6 +867,32 @@ expression to achieve this. +Common objections +================= + +Colons always introduce suites +------------------------------ + +While it is true that many of Python's syntactic elements use the colon to +introduce a statement suite (if, while, with, for, etcetera), this is not +by any means the sole use of the colon. Currently, Python syntax includes +four cases where a colon introduces a subexpression: + +* dict display - { ... key:value ... } +* slice notation - [start:stop:step] +* function definition - parameter : annotation +* lambda - arg list: return value + +This proposal simply adds a fifth: + +* except-expression - exception list: result + +Style guides and PEP 8 should recommend not having the colon at the end of +a wrapped line, which could potentially look like the introduction of a +suite, but instead advocate wrapping before the exception list, keeping the +colon clearly between two expressions. + + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 6 05:16:30 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 6 Mar 2014 05:16:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE1NjE4?= =?utf-8?q?=3A_Make_turtle=2Epy_itself_work_when_run_from_a_module_with?= Message-ID: <3ffdv20dk0z7M2k@mail.python.org> http://hg.python.org/cpython/rev/1be39dc4409a changeset: 89481:1be39dc4409a branch: 2.7 parent: 89468:d37f963394aa user: Terry Jan Reedy date: Wed Mar 05 23:15:57 2014 -0500 summary: Issue #15618: Make turtle.py itself work when run from a module with from __future__ import unicode_literals. Initial patch by Juancarlo A?ez. The demos at the end of turtle.py appear to be the only test, so I changed some of the strings to unicode with a u prefix. If os.path.isfile or the Tk image function have problems with Unicode input, that would be different issue. files: Lib/lib-tk/turtle.py | 18 +++++++++--------- Misc/ACKS | 1 + Misc/NEWS | 5 ++++- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/Lib/lib-tk/turtle.py b/Lib/lib-tk/turtle.py --- a/Lib/lib-tk/turtle.py +++ b/Lib/lib-tk/turtle.py @@ -835,7 +835,7 @@ if isinstance(data, list): data = tuple(data) elif type_ == "image": - if isinstance(data, str): + if isinstance(data, basestring): if data.lower().endswith(".gif") and isfile(data): data = TurtleScreen._image(data) # else data assumed to be Photoimage @@ -1098,7 +1098,7 @@ """ if len(color) == 1: color = color[0] - if isinstance(color, str): + if isinstance(color, basestring): if self._iscolorstring(color) or color == "": return color else: @@ -2602,7 +2602,7 @@ def _cc(self, args): """Convert colortriples to hexstrings. """ - if isinstance(args, str): + if isinstance(args, basestring): return args try: r, g, b = args @@ -3228,7 +3228,7 @@ """ #print "dot-1:", size, color if not color: - if isinstance(size, (str, tuple)): + if isinstance(size, (basestring, tuple)): color = self._colorstr(size) size = self._pensize + max(self._pensize, 4) else: @@ -3913,7 +3913,7 @@ down() # some text write("startstart", 1) - write("start", 1) + write(u"start", 1) color("red") # staircase for i in range(5): @@ -3988,7 +3988,7 @@ tri = getturtle() tri.resizemode("auto") turtle = Turtle() - turtle.resizemode("auto") + turtle.resizemode(u"auto") turtle.shape("turtle") turtle.reset() turtle.left(90) @@ -3998,7 +3998,7 @@ turtle.lt(30) turtle.down() turtle.speed(6) - turtle.color("blue","orange") + turtle.color("blue",u"orange") turtle.pensize(2) tri.speed(6) setheading(towards(turtle)) @@ -4013,9 +4013,9 @@ tri.stamp() switchpen() count += 1 - tri.write("CAUGHT! ", font=("Arial", 16, "bold"), align="right") + tri.write("CAUGHT! ", font=("Arial", 16, "bold"), align=u"right") tri.pencolor("black") - tri.pencolor("red") + tri.pencolor(u"red") def baba(xdummy, ydummy): clearscreen() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -33,6 +33,7 @@ Erik Anders?n Oliver Andrich Ross Andrus +Juancarlo A?ez Chris Angelico Ankur Ankan Heidi Annexstad diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -40,6 +40,9 @@ Library ------- +- Issue #15618: Make turtle.py compatible with 'from __future__ import + unicode_literals'. Initial patch by Juancarlo A?ez. + - Issue #20501: fileinput module no longer reads whole file into memory when using fileinput.hook_encoded. @@ -53,7 +56,7 @@ Based on patch by Stephen Tu. - Issue #8478: Untokenizer.compat processes first token from iterator input. - Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees. + Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees. - Issue #20594: Avoid name clash with the libc function posix_close. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Mar 6 10:03:05 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 06 Mar 2014 10:03:05 +0100 Subject: [Python-checkins] Daily reference leaks (56c346e9ae4d): sum=0 Message-ID: results for 56c346e9ae4d on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogfXg3mv', '-x'] From python-checkins at python.org Thu Mar 6 10:37:28 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 6 Mar 2014 10:37:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMjgz?= =?utf-8?q?=3A_RE_pattern_methods_now_accept_the_string_keyword_parameters?= Message-ID: <3ffn1N07fgz7M0f@mail.python.org> http://hg.python.org/cpython/rev/52743dc788e6 changeset: 89482:52743dc788e6 branch: 3.3 parent: 89477:3b2c28061184 user: Serhiy Storchaka date: Thu Mar 06 11:28:32 2014 +0200 summary: Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. files: Lib/test/test_re.py | 16 ++++++ Misc/NEWS | 4 + Modules/_sre.c | 84 +++++++++++++++++++++++++------- 3 files changed, 84 insertions(+), 20 deletions(-) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1076,6 +1076,22 @@ self.assertEqual(out.getvalue().splitlines(), ['literal 102 ', 'literal 111 ', 'literal 111 ']) + def test_keyword_parameters(self): + # Issue #20283: Accepting the string keyword parameter. + pat = re.compile(r'(ab)') + self.assertEqual( + pat.match(string='abracadabra', pos=7, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.search(string='abracadabra', pos=3, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.findall(string='abracadabra', pos=3, endpos=10), ['ab']) + self.assertEqual( + pat.split(string='abracadabra', maxsplit=1), + ['', 'ab', 'racadabra']) + self.assertEqual( + pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(), + (7, 9)) + def run_re_tests(): from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -22,6 +22,10 @@ Library ------- +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + - Issue #20778: Fix modulefinder to work with bytecode-only modules. - Issue #20791: copy.copy() now doesn't make a copy when the input is diff --git a/Modules/_sre.c b/Modules/_sre.c --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -1877,17 +1877,45 @@ } static PyObject* +fix_string_param(PyObject *string, PyObject *string2, const char *oldname) +{ + if (string2 != NULL) { + if (string != NULL) { + PyErr_Format(PyExc_TypeError, + "Argument given by name ('%s') and position (1)", + oldname); + return NULL; + } + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "The '%s' keyword parameter name is deprecated. " + "Use 'string' instead.", oldname) < 0) + return NULL; + return string2; + } + if (string == NULL) { + PyErr_SetString(PyExc_TypeError, + "Required argument 'string' (pos 1) not found"); + return NULL; + } + return string; +} + +static PyObject* pattern_match(PatternObject* self, PyObject* args, PyObject* kw) { SRE_STATE state; int status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:match", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:match", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -1919,12 +1947,16 @@ SRE_STATE state; int status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:search", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:search", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -2052,12 +2084,16 @@ int status; Py_ssize_t i, b, e; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:findall", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:findall", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -2180,11 +2216,15 @@ Py_ssize_t i; void* last; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t maxsplit = 0; - static char* kwlist[] = { "source", "maxsplit", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|n:split", kwlist, - &string, &maxsplit)) + static char* kwlist[] = { "string", "maxsplit", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|On$O:split", kwlist, + &string, &maxsplit, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, 0, PY_SSIZE_T_MAX); @@ -3882,12 +3922,16 @@ ScannerObject* self; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:scanner", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:scanner", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; /* create scanner object */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 10:37:29 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 6 Mar 2014 10:37:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320283=3A_RE_pattern_methods_now_accept_the_stri?= =?utf-8?q?ng_keyword_parameters?= Message-ID: <3ffn1P345Nz7M0F@mail.python.org> http://hg.python.org/cpython/rev/f4d7abcf8080 changeset: 89483:f4d7abcf8080 parent: 89480:56c346e9ae4d parent: 89482:52743dc788e6 user: Serhiy Storchaka date: Thu Mar 06 11:36:15 2014 +0200 summary: Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. files: Lib/test/test_re.py | 18 +++ Misc/NEWS | 4 + Modules/_sre.c | 145 +++++++++++++++++-------------- 3 files changed, 102 insertions(+), 65 deletions(-) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1205,6 +1205,24 @@ self.assertEqual(out.getvalue().splitlines(), ['literal 102 ', 'literal 111 ', 'literal 111 ']) + def test_keyword_parameters(self): + # Issue #20283: Accepting the string keyword parameter. + pat = re.compile(r'(ab)') + self.assertEqual( + pat.match(string='abracadabra', pos=7, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.fullmatch(string='abracadabra', pos=7, endpos=9).span(), (7, 9)) + self.assertEqual( + pat.search(string='abracadabra', pos=3, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.findall(string='abracadabra', pos=3, endpos=10), ['ab']) + self.assertEqual( + pat.split(string='abracadabra', maxsplit=1), + ['', 'ab', 'racadabra']) + self.assertEqual( + pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(), + (7, 9)) + class PatternReprTests(unittest.TestCase): def check(self, pattern, expected): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,10 @@ Library ------- +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + - Issue #20839: Don't trigger a DeprecationWarning in the still supported pkgutil.get_loader() API when __loader__ isn't set on a module (nor when pkgutil.find_loader() is called directly). diff --git a/Modules/_sre.c b/Modules/_sre.c --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -526,59 +526,49 @@ return sre_ucs4_search(state, pattern); } -/*[clinic input] -module _sre -class _sre.SRE_Pattern "PatternObject *" "&Pattern_Type" - -_sre.SRE_Pattern.match as pattern_match - - pattern: object - pos: Py_ssize_t = 0 - endpos: Py_ssize_t(c_default="PY_SSIZE_T_MAX") = sys.maxsize - -Matches zero or more characters at the beginning of the string. -[clinic start generated code]*/ - -PyDoc_STRVAR(pattern_match__doc__, -"match($self, /, pattern, pos=0, endpos=sys.maxsize)\n" -"--\n" -"\n" -"Matches zero or more characters at the beginning of the string."); - -#define PATTERN_MATCH_METHODDEF \ - {"match", (PyCFunction)pattern_match, METH_VARARGS|METH_KEYWORDS, pattern_match__doc__}, - static PyObject * -pattern_match_impl(PatternObject *self, PyObject *pattern, Py_ssize_t pos, Py_ssize_t endpos); +fix_string_param(PyObject *string, PyObject *string2, const char *oldname) +{ + if (string2 != NULL) { + if (string != NULL) { + PyErr_Format(PyExc_TypeError, + "Argument given by name ('%s') and position (1)", + oldname); + return NULL; + } + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "The '%s' keyword parameter name is deprecated. " + "Use 'string' instead.", oldname) < 0) + return NULL; + return string2; + } + if (string == NULL) { + PyErr_SetString(PyExc_TypeError, + "Required argument 'string' (pos 1) not found"); + return NULL; + } + return string; +} static PyObject * pattern_match(PatternObject *self, PyObject *args, PyObject *kwargs) { - PyObject *return_value = NULL; - static char *_keywords[] = {"pattern", "pos", "endpos", NULL}; - PyObject *pattern; + static char *_keywords[] = {"string", "pos", "endpos", "pattern", NULL}; + PyObject *string = NULL; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - - if (!PyArg_ParseTupleAndKeywords(args, kwargs, - "O|nn:match", _keywords, - &pattern, &pos, &endpos)) - goto exit; - return_value = pattern_match_impl(self, pattern, pos, endpos); - -exit: - return return_value; -} - -static PyObject * -pattern_match_impl(PatternObject *self, PyObject *pattern, Py_ssize_t pos, Py_ssize_t endpos) -/*[clinic end generated code: output=1528eafdb8b025ad input=26f9fd31befe46b9]*/ -{ + PyObject *pattern = NULL; SRE_STATE state; Py_ssize_t status; - PyObject *string; - - string = state_init(&state, (PatternObject *)self, pattern, pos, endpos); + + if (!PyArg_ParseTupleAndKeywords(args, kwargs, + "|Onn$O:match", _keywords, + &string, &pos, &endpos, &pattern)) + return NULL; + string = fix_string_param(string, pattern, "pattern"); + if (!string) + return NULL; + string = state_init(&state, (PatternObject *)self, string, pos, endpos); if (!string) return NULL; @@ -603,12 +593,16 @@ SRE_STATE state; Py_ssize_t status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:fullmatch", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:fullmatch", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -637,12 +631,16 @@ SRE_STATE state; Py_ssize_t status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:search", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:search", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -718,12 +716,16 @@ Py_ssize_t status; Py_ssize_t i, b, e; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:findall", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:findall", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -840,11 +842,15 @@ Py_ssize_t i; void* last; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t maxsplit = 0; - static char* kwlist[] = { "source", "maxsplit", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|n:split", kwlist, - &string, &maxsplit)) + static char* kwlist[] = { "string", "maxsplit", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|On$O:split", kwlist, + &string, &maxsplit, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, 0, PY_SSIZE_T_MAX); @@ -1292,6 +1298,10 @@ return result; } +PyDoc_STRVAR(pattern_match_doc, +"match(string[, pos[, endpos]]) -> match object or None.\n\ + Matches zero or more characters at the beginning of the string"); + PyDoc_STRVAR(pattern_fullmatch_doc, "fullmatch(string[, pos[, endpos]]) -> match object or None.\n\ Matches against all of the string"); @@ -1329,7 +1339,8 @@ PyDoc_STRVAR(pattern_doc, "Compiled regular expression objects"); static PyMethodDef pattern_methods[] = { - PATTERN_MATCH_METHODDEF + {"match", (PyCFunction) pattern_match, METH_VARARGS|METH_KEYWORDS, + pattern_match_doc}, {"fullmatch", (PyCFunction) pattern_fullmatch, METH_VARARGS|METH_KEYWORDS, pattern_fullmatch_doc}, {"search", (PyCFunction) pattern_search, METH_VARARGS|METH_KEYWORDS, @@ -2654,12 +2665,16 @@ ScannerObject* self; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:scanner", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:scanner", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; /* create scanner object */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 11:25:56 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 6 Mar 2014 11:25:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMjgz?= =?utf-8?q?=3A_RE_pattern_methods_now_accept_the_string_keyword_parameters?= Message-ID: <3ffp5J3w3lz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/52256a5861fa changeset: 89484:52256a5861fa branch: 2.7 parent: 89481:1be39dc4409a user: Serhiy Storchaka date: Thu Mar 06 12:24:29 2014 +0200 summary: Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. files: Lib/test/test_re.py | 13 ++++ Misc/NEWS | 4 + Modules/_sre.c | 97 +++++++++++++++++++++++++++----- 3 files changed, 98 insertions(+), 16 deletions(-) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -933,6 +933,19 @@ self.assertEqual(out.getvalue().splitlines(), ['literal 102', 'literal 111', 'literal 111']) + def test_keyword_parameters(self): + # Issue #20283: Accepting the string keyword parameter. + pat = re.compile(r'(ab)') + self.assertEqual( + pat.match(string='abracadabra', pos=7, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.search(string='abracadabra', pos=3, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.findall(string='abracadabra', pos=3, endpos=10), ['ab']) + self.assertEqual( + pat.split(string='abracadabra', maxsplit=1), + ['', 'ab', 'racadabra']) + def run_re_tests(): from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -40,6 +40,10 @@ Library ------- +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + - Issue #15618: Make turtle.py compatible with 'from __future__ import unicode_literals'. Initial patch by Juancarlo A?ez. diff --git a/Modules/_sre.c b/Modules/_sre.c --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -1875,18 +1875,62 @@ PyObject_DEL(self); } +static int +check_args_size(const char *name, PyObject* args, PyObject* kw, int n) +{ + Py_ssize_t m = PyTuple_GET_SIZE(args) + (kw ? PyDict_Size(kw) : 0); + if (m <= n) + return 1; + PyErr_Format(PyExc_TypeError, + "%s() takes at most %d positional arguments (%zd given)", + name, n, m); + return 0; +} + +static PyObject* +fix_string_param(PyObject *string, PyObject *string2, const char *oldname) +{ + if (string2 != NULL) { + char buf[100]; + if (string != NULL) { + PyErr_Format(PyExc_TypeError, + "Argument given by name ('%s') and position (1)", + oldname); + return NULL; + } + sprintf(buf, "The '%s' keyword parameter name is deprecated. " + "Use 'string' instead.", oldname); + if (PyErr_Warn(PyExc_DeprecationWarning, buf) < 0) + return NULL; + return string2; + } + if (string == NULL) { + PyErr_SetString(PyExc_TypeError, + "Required argument 'string' (pos 1) not found"); + return NULL; + } + return string; +} + static PyObject* pattern_match(PatternObject* self, PyObject* args, PyObject* kw) { SRE_STATE state; int status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:match", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!check_args_size("match", args, kw, 3)) + return NULL; + + if (!PyArg_ParseTupleAndKeywords(args, kw, "|OnnO:match", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -1920,12 +1964,19 @@ SRE_STATE state; int status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:search", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!check_args_size("search", args, kw, 3)) + return NULL; + + if (!PyArg_ParseTupleAndKeywords(args, kw, "|OnnO:search", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -2055,12 +2106,19 @@ int status; Py_ssize_t i, b, e; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:findall", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!check_args_size("findall", args, kw, 3)) + return NULL; + + if (!PyArg_ParseTupleAndKeywords(args, kw, "|OnnO:findall", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -2185,11 +2243,18 @@ Py_ssize_t i; void* last; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t maxsplit = 0; - static char* kwlist[] = { "source", "maxsplit", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|n:split", kwlist, - &string, &maxsplit)) + static char* kwlist[] = { "string", "maxsplit", "source", NULL }; + if (!check_args_size("split", args, kw, 2)) + return NULL; + + if (!PyArg_ParseTupleAndKeywords(args, kw, "|OnO:split", kwlist, + &string, &maxsplit, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, 0, PY_SSIZE_T_MAX); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 16:09:34 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 16:09:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_Windows_build_?= =?utf-8?q?uses_ASLR_and_DEP=2E?= Message-ID: <3ffwNZ5Kysz7LjX@mail.python.org> http://hg.python.org/cpython/rev/285ae56e4713 changeset: 89485:285ae56e4713 parent: 89483:f4d7abcf8080 user: R David Murray date: Mon Mar 03 08:26:49 2014 -0500 summary: whatsnew: Windows build uses ASLR and DEP. files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1547,6 +1547,11 @@ values from :envvar:`PYTHONIOENCODING` if it exists. (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`.) +* The Windows build now uses `Address Space Layout Randomization + `_ and `Data Execution Prevention + `_. (Contributed by + Christian Heimes in :issue:`16632`.) + .. _other-improvements-3.4: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 16:09:36 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 16:09:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_tweak_a_couple?= =?utf-8?q?_importlib_porting_notes=2E?= Message-ID: <3ffwNc0D59z7Lk2@mail.python.org> http://hg.python.org/cpython/rev/78f063002259 changeset: 89486:78f063002259 user: R David Murray date: Mon Mar 03 08:56:43 2014 -0500 summary: whatsnew: tweak a couple importlib porting notes. I had missed that there was already an entry for the Frozen package change, so I combined mine with the existing one. Also added the info about the reload change to the entry for the other changes to reload, but I'm missing an issue number for the rest of the attribute behavior changes so I left myself an XXX as a reminder to find it. files: Doc/whatsnew/3.4.rst | 15 +++++++-------- Misc/NEWS | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1883,10 +1883,15 @@ * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. + Note that this restores a pre-3.3 behavior in that it means a module is + re-found when re-loaded (:issue:`19413` and XXX). * Frozen packages no longer set ``__path__`` to a list containing the package - name but an empty list instead. Determing if a module is a package should be - done using ``hasattr(module, '__path__')``. + name, they now set it to an empty list. The previous behavior could cause + the import system to do the wrong thing on submodule imports if there was + also a directory with the same name as the frozen package. The correct way + to determine if a module is a package or not is to use``hasattr(module, + '__path__')`` (:issue:`18065`). * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path it would write to is a symlink or a non-regular file. This is to act as a @@ -1984,12 +1989,6 @@ similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in :issue:`20625`). -* Frozen packages used to have their ``__path__`` set to the package name, - but this could cause the import system to do the wrong thing on submodule - imports if there was also a directory with the same name as the frozen - package. The ``__path__`` for frozen packages is now set to ``[]`` - (:issue:`18065`). - * :attr:`hashlib.hash.name` now always returns the identifier in lower case. Previously some builtin hashes had uppercase names, but now that it is a formal public interface the naming has been made consistent (:issue:`18532`). diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1463,7 +1463,7 @@ - Issue #19085: Added basic tests for all tkinter widget options. -- Issue 19384: Fix test_py_compile for root user, patch by Claudiu Popa. +- Issue #19384: Fix test_py_compile for root user, patch by Claudiu Popa. Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 16:09:37 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 16:09:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_improve_PyZipF?= =?utf-8?q?ile_filterfuc_entry=2C_and_its_docs_=28=2319274=29=2E?= Message-ID: <3ffwNd2hS6z7Lkj@mail.python.org> http://hg.python.org/cpython/rev/064ee489982e changeset: 89487:064ee489982e user: R David Murray date: Wed Mar 05 10:48:01 2014 -0500 summary: whatsnew: improve PyZipFile filterfuc entry, and its docs (#19274). files: Doc/library/zipfile.rst | 26 ++++++++++++++++++++------ Doc/whatsnew/3.4.rst | 12 +++++++----- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -401,18 +401,32 @@ ``2``, only files with that optimization level (see :func:`compile`) are added to the archive, compiling if necessary. - If the pathname is a file, the filename must end with :file:`.py`, and + If *pathname* is a file, the filename must end with :file:`.py`, and just the (corresponding :file:`\*.py[co]`) file is added at the top level - (no path information). If the pathname is a file that does not end with + (no path information). If *pathname* is a file that does not end with :file:`.py`, a :exc:`RuntimeError` will be raised. If it is a directory, and the directory is not a package directory, then all the files :file:`\*.py[co]` are added at the top level. If the directory is a package directory, then all :file:`\*.py[co]` are added under the package name as a file path, and if any subdirectories are package directories, - all of these are added recursively. *basename* is intended for internal - use only. When *filterfunc(pathname)* is given, it will be called for every - invocation. When it returns a false value, that path and its subpaths will - be ignored. + all of these are added recursively. + + *basename* is intended for internal use only. + + *filterfunc*, if given, must be a function taking a single string + argument. It will be passed each path (including each individual full + file path) before it is added to the archive. If *filterfunc* returns a + false value, the path will not be added, and if it is a directory its + contents will be ignored. For example, if our test files are all either + in ``test`` directories or start with the string ``test_``, we can use a + *filterfunc* to exclude them:: + + >>> zf = PyZipFile('myprog.zip') + >>> def notests(s): + ... fn = os.path.basename(s) + ... return (not (fn == 'test' or fn.startswith('test_'))) + >>> zf.writepy('myprog', filterfunc=notests) + The :meth:`writepy` method makes archives with file names like this:: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1415,11 +1415,13 @@ Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) -zipfile.PyZipfile ------------------ - -Add a filter function to ignore some packages (tests for instance), -:meth:`~zipfile.PyZipFile.writepy`. +zipfile +------- + +The :meth:`~zipfile.PyZipFile.writepy` method of the +:class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be +used to control which directories and files are added to the archive. For +example, this could be used to exclude test files from the archive. (Contributed by Christian Tismer in :issue:`19274`.) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 17:07:02 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 6 Mar 2014 17:07:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_tracemalloc=2Epy=3A_fix_in?= =?utf-8?q?dentation?= Message-ID: <3ffxft34mtz7LjW@mail.python.org> http://hg.python.org/cpython/rev/56fbf7a7173a changeset: 89488:56fbf7a7173a user: Victor Stinner date: Thu Mar 06 17:06:04 2014 +0100 summary: tracemalloc.py: fix indentation files: Lib/tracemalloc.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py --- a/Lib/tracemalloc.py +++ b/Lib/tracemalloc.py @@ -119,12 +119,12 @@ previous = old_group.pop(traceback, None) if previous is not None: stat = StatisticDiff(traceback, - stat.size, stat.size - previous.size, - stat.count, stat.count - previous.count) + stat.size, stat.size - previous.size, + stat.count, stat.count - previous.count) else: stat = StatisticDiff(traceback, - stat.size, stat.size, - stat.count, stat.count) + stat.size, stat.size, + stat.count, stat.count) statistics.append(stat) for traceback, stat in old_group.items(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 17:07:03 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 6 Mar 2014 17:07:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_tracemalloc=2Epy=3A_add_co?= =?utf-8?q?mments_to_show_the_format_of_tuples?= Message-ID: <3ffxfv4y0Zz7LkD@mail.python.org> http://hg.python.org/cpython/rev/b07400659dba changeset: 89489:b07400659dba user: Victor Stinner date: Thu Mar 06 17:06:41 2014 +0100 summary: tracemalloc.py: add comments to show the format of tuples files: Lib/tracemalloc.py | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py --- a/Lib/tracemalloc.py +++ b/Lib/tracemalloc.py @@ -141,6 +141,7 @@ __slots__ = ("_frame",) def __init__(self, frame): + # frame is a tuple: (filename: str, lineno: int) self._frame = frame @property @@ -177,6 +178,8 @@ def __init__(self, frames): Sequence.__init__(self) + # frames is a tuple of frame tuples: see Frame constructor for the + # format of a frame tuple self._frames = frames def __len__(self): @@ -241,6 +244,8 @@ __slots__ = ("_trace",) def __init__(self, trace): + # trace is a tuple: (size, traceback), see Traceback constructor + # for the format of the traceback tuple self._trace = trace @property @@ -268,6 +273,7 @@ class _Traces(Sequence): def __init__(self, traces): Sequence.__init__(self) + # traces is a tuple of trace tuples: see Trace constructor self._traces = traces def __len__(self): @@ -338,6 +344,8 @@ """ def __init__(self, traces, traceback_limit): + # traces is a tuple of trace tuples: see _Traces constructor for + # the exact format self.traces = _Traces(traces) self.traceback_limit = traceback_limit -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 17:44:33 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 17:44:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2311558=3A_Better_message?= =?utf-8?q?_if_attach_called_on_non-multipart=2E?= Message-ID: <3ffyV920zGz7LjX@mail.python.org> http://hg.python.org/cpython/rev/302c8fdb17e3 changeset: 89490:302c8fdb17e3 user: R David Murray date: Thu Mar 06 11:44:17 2014 -0500 summary: #11558: Better message if attach called on non-multipart. Original patch by Varun Sharma. files: Lib/email/message.py | 6 +++++- Lib/test/test_email/test_email.py | 8 ++++++++ Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 18 insertions(+), 1 deletions(-) diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -203,7 +203,11 @@ if self._payload is None: self._payload = [payload] else: - self._payload.append(payload) + try: + self._payload.append(payload) + except AttributeError: + raise TypeError("Attach is not valid on a message with a" + " non-multipart payload") def get_payload(self, i=None, decode=False): """Return a reference to the payload. diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -124,6 +124,14 @@ msg.set_payload([]) self.assertEqual(msg.get_payload(), []) + def test_attach_when_payload_is_string(self): + msg = Message() + msg['Content-Type'] = 'multipart/mixed' + msg.set_payload('string payload') + sub_msg = MIMEMessage(Message()) + self.assertRaisesRegex(TypeError, "[Aa]ttach.*non-multipart", + msg.attach, sub_msg) + def test_get_charsets(self): eq = self.assertEqual diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1188,6 +1188,7 @@ Ha Shao Mark Shannon Richard Shapiro +Varun Sharma Vlad Shcherbina Justin Sheehy Charlie Shepherd diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,10 @@ Library ------- +- Issue #11558: ``email.message.Message.attach`` now returns a more + useful error message if ``attach`` is called on a message for which + ``is_multipart`` is False. + - Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 19:41:11 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 19:41:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_ipaddress_is?= =?utf-8?q?=5Fglobal?= Message-ID: <3fg14l6Lk2z7LjR@mail.python.org> http://hg.python.org/cpython/rev/fa3efccb9888 changeset: 89491:fa3efccb9888 user: R David Murray date: Thu Mar 06 11:51:37 2014 -0500 summary: whatsnew: ipaddress is_global Also tweaked the docs a bit and added the missing entry for the ipv6 is_global. files: Doc/library/ipaddress.rst | 18 +++++++++++++----- Doc/whatsnew/3.4.rst | 4 ++++ 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -154,16 +154,16 @@ .. attribute:: is_private ``True`` if the address is allocated for private networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). .. attribute:: is_global ``True`` if the address is allocated for public networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). - .. versionadded:: 3.4 + .. versionadded:: 3.4 .. attribute:: is_unspecified @@ -184,6 +184,9 @@ ``True`` if the address is reserved for link-local usage. See :RFC:`3927`. +.. _iana-ipv4-special-registry: http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml +.. _iana-ipv6-special-registry: http://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml + .. class:: IPv6Address(address) @@ -218,18 +221,23 @@ The long form of the address representation, with all leading zeroes and groups consisting entirely of zeroes included. + + For the following attributes, see the corresponding documention of the + :class:`IPv4Address` class: + .. attribute:: packed .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast .. attribute:: is_private + .. attribute:: is_global .. attribute:: is_unspecified .. attribute:: is_reserved .. attribute:: is_loopback .. attribute:: is_link_local - Refer to the corresponding attribute documentation in - :class:`IPv4Address` + .. versionadded:: 3.4 + is_global .. attribute:: is_site_local diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -901,6 +901,10 @@ by the normal standard library requirements to maintain backwards compatibility. +A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if +an address is globally routeable. (Contributed by Peter Moody in +:issue:`17400`.) + logging ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 19:41:13 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 19:41:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_rewrite_resour?= =?utf-8?q?ce_entry=3A_prlimit_and_new_constants=2E?= Message-ID: <3fg14n0sLDz7LjR@mail.python.org> http://hg.python.org/cpython/rev/4f946e19f016 changeset: 89492:4f946e19f016 user: R David Murray date: Thu Mar 06 12:47:31 2014 -0500 summary: whatsnew: rewrite resource entry: prlimit and new constants. files: Doc/whatsnew/3.4.rst | 17 +++++++++++++++-- 1 files changed, 15 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1101,8 +1101,21 @@ resource -------- -New :func:`resource.prlimit` function and Linux specific constants. -(Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) +New :func:`~resource.prlimit` function, available on Linux platforms with a +kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the +ability to query or set the resource limits for processes other than the one +making the call. (Contributed by Christian Heimes in :issue:`16595`.) + +On Linux kernel version 2.6.36 or later, there are there are also some new +Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, +:attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, +:attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. +(Contributed by Christian Heimes in :issue:`19324`.) + +On FreeBSD version 9 and later, there some new FreeBSD specific constants: +:attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and +:attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in +:issue:`19343`.) select -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 19:41:14 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 19:41:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_redundant_NEWS_entr?= =?utf-8?q?y=2E?= Message-ID: <3fg14p31jYz7Llq@mail.python.org> http://hg.python.org/cpython/rev/fe2042d02d85 changeset: 89493:fe2042d02d85 user: R David Murray date: Thu Mar 06 12:58:25 2014 -0500 summary: Remove redundant NEWS entry. files: Misc/NEWS | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1419,8 +1419,6 @@ - Issue #19324: Expose Linux-specific constants in resource module. -- Issue #17400: ipaddress should make it easy to identify rfc6598 addresses. - - Load SSL's error strings in hashlib. - Issue #18527: Upgrade internal copy of zlib to 1.2.8. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 19:41:15 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 6 Mar 2014 19:41:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_note_in_deprec?= =?utf-8?q?ation_section_about_html=2Eparser_strict=2E?= Message-ID: <3fg14q5J0Pz7LkY@mail.python.org> http://hg.python.org/cpython/rev/964c4899efc4 changeset: 89494:964c4899efc4 user: R David Murray date: Thu Mar 06 13:40:53 2014 -0500 summary: whatsnew: note in deprecation section about html.parser strict. files: Doc/whatsnew/3.4.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1759,6 +1759,8 @@ not have been exposed and is hopefully not in use; it is deprecated and will mostly likely be removed in Python 3.5. +* The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. + Deprecated Functions and Types in the C API ------------------------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 20:28:34 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 6 Mar 2014 20:28:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Make_pywlaunch?= =?utf-8?q?er=2Evcxproj_depend_on_make=5Fversioninfo=2Evcxproj=2E?= Message-ID: <3fg27Q6N92z7Ljh@mail.python.org> http://hg.python.org/cpython/rev/7a7ff2015099 changeset: 89495:7a7ff2015099 branch: 3.3 parent: 89482:52743dc788e6 user: Zachary Ware date: Thu Mar 06 13:21:45 2014 -0600 summary: Make pywlauncher.vcxproj depend on make_versioninfo.vcxproj. pylauncher.vcxproj already has the same dependency, and adding it to the pywlauncher project eliminates the possibility of the pywlauncher build failing when building the solution in parallel. In particular, pywlauncher may fail to build when doing a "Rebuild Solution" from the VS Build menu. files: PCbuild/pywlauncher.vcxproj | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/PCbuild/pywlauncher.vcxproj b/PCbuild/pywlauncher.vcxproj --- a/PCbuild/pywlauncher.vcxproj +++ b/PCbuild/pywlauncher.vcxproj @@ -240,6 +240,11 @@ + + + {f0e0541e-f17d-430b-97c4-93adf0dd284e} + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 6 20:28:36 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 6 Mar 2014 20:28:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Make_pywlauncher=2Evcxproj_depend_on_make=5Fversioninfo?= =?utf-8?b?LnZjeHByb2ou?= Message-ID: <3fg27S19fHz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/47f37a688c4c changeset: 89496:47f37a688c4c parent: 89494:964c4899efc4 parent: 89495:7a7ff2015099 user: Zachary Ware date: Thu Mar 06 13:28:08 2014 -0600 summary: Make pywlauncher.vcxproj depend on make_versioninfo.vcxproj. pylauncher.vcxproj already has the same dependency, and adding it to the pywlauncher project eliminates the possibility of the pywlauncher build failing when building the solution in parallel. In particular, pywlauncher may fail to build when doing a "Rebuild Solution" from the VS Build menu. Merge with 3.3. files: PCbuild/pywlauncher.vcxproj | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/PCbuild/pywlauncher.vcxproj b/PCbuild/pywlauncher.vcxproj --- a/PCbuild/pywlauncher.vcxproj +++ b/PCbuild/pywlauncher.vcxproj @@ -240,6 +240,11 @@ + + + {f0e0541e-f17d-430b-97c4-93adf0dd284e} + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 7 09:37:18 2014 From: python-checkins at python.org (larry.hastings) Date: Fri, 7 Mar 2014 09:37:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_New_hacks_for_threefourtoo?= =?utf-8?q?l!__Now_it_has=2E=2E=2E_*auto-graft*!?= Message-ID: <3fgMdV1M3rz7Ljh@mail.python.org> http://hg.python.org/release/rev/8e3ce80c9f24 changeset: 80:8e3ce80c9f24 user: Larry Hastings date: Fri Mar 07 00:37:13 2014 -0800 summary: New hacks for threefourtool! Now it has... *auto-graft*! files: 3.4/threefourtool.py | 86 ++++++++++++++++++++++++------- 1 files changed, 65 insertions(+), 21 deletions(-) diff --git a/3.4/threefourtool.py b/3.4/threefourtool.py --- a/3.4/threefourtool.py +++ b/3.4/threefourtool.py @@ -28,7 +28,7 @@ def system(s): print(s) - os.system(s) + return os.system(s) def yes_no(): while True: @@ -124,7 +124,7 @@ current_directory = os.getcwd() try: os.chdir("/home/larry/src/python/3.4") - p = subprocess.Popen(["/usr/bin/hg", "log", "-v"], stdout=subprocess.PIPE) + p = subprocess.Popen(["/home/larry/src/hg/hg", "log", "-v"], stdout=subprocess.PIPE) # with open("/home/larry/src/python/logtxt", "rt", encoding="utf-8") as f: # output = f.read() @@ -565,7 +565,7 @@ read_changesets() os.chdir("/home/larry/src/python/3.4") print("Updating to 3.4 branch:") - os.system("hg update -r 3.4") + os.system("/home/larry/src/hg/hg update -r 3.4") print() u = self.unfinished.get print("Picking revisions ", u('picked revisions')) @@ -604,7 +604,7 @@ atexit.register(delete_files, patch_path, commit_message_path) def detect_new_revision(): - output = subprocess.check_output(['/usr/bin/hg', 'summary']).decode('utf-8').split('\n') + output = subprocess.check_output(['/home/larry/src/hg/hg', 'summary']).decode('utf-8').split('\n') line = output[0] assert line.startswith('parent:') line = line[len('parent:'):] @@ -613,6 +613,11 @@ r = r.split()[0].strip() u['threefour picked revision'] = r + show_graft = False + def toggle_graft(): + nonlocal show_graft + show_graft = not show_graft + show_patch = False def toggle_patch(): nonlocal show_patch @@ -625,33 +630,72 @@ print("_" * 79) print() + def remove_orig_and_rej(): + for dirpath, dirnames, filenames in os.walk("."): + for filename in filenames: + if not filename.endswith((".orig", ".rej")): + continue + path = os.path.join(dirpath, filename) + os.unlink(path) + + changes = subprocess.check_output(['/home/larry/src/hg/hg', 'stat', '-mard']).decode('utf-8') + try_auto_graft = not bool(changes) + if try_auto_graft: + print("Attempting auto-graft:") + else: + print("Skipping auto-graft, there are outstanding changes.") + while u.get('default picked revision'): commands = [] - commands.append(("Update to appropriate revision in 3.4 branch", "hg update -r {threefour graft here}")) - commands.append(("Graft revision", "hg graft {default picked revision}")) + commands.append(("Update to appropriate revision in 3.4 branch", "/home/larry/src/hg/hg update -r {threefour graft here}")) + commands.append(("Graft revision", "/home/larry/src/hg/hg graft --tool internal:merge {default picked revision}")) + commands.append(("Handle graft conflict", toggle_graft)) + + if show_graft: + commands.append(("Graft #1: List conflicts", "/home/larry/src/hg/hg diff resolve -l")) + commands.append(("Graft #2: Mark all conflicts as resolved", "/home/larry/src/hg/hg diff resolve -m")) + commands.append(("Graft #3: Resume the graft", "/home/larry/src/hg/hg diff graft --continue")) + commands.append(("Graft #4: Clean up .orig and .rej files", remove_orig_and_rej)) + commands.append(("Patch revision (only if graft fails)", toggle_patch)) - if show_patch: - commands.append(("[graft failed step 1] Generate patch", "/usr/bin/hg diff -r {default diff from} -r {default picked revision} > {patch path}")) - commands.append(("[graft failed step 2] Inspect patch", "{EDITOR} {patch path}")) - commands.append(("[graft failed step 3] Apply patch", "/usr/bin/patch -p1 < {patch path}")) - commands.append(("[graft failed step 4] Check in patch", "/usr/bin/hg ci --user '{user}' --date '{date}' --logfile '{commit message path}'")) + commands.append(("Patch #1: Generate patch", "/home/larry/src/hg/hg diff -r {default diff from} -r {default picked revision} > {patch path}")) + commands.append(("Patch #2: Inspect patch", "{EDITOR} {patch path}")) + commands.append(("Patch #3: Apply patch", "/usr/bin/patch -p1 < {patch path}")) + commands.append(("Patch #4: Check in patch", "/home/larry/src/hg/hg ci --user '{user}' --date '{date}' --logfile '{commit message path}'")) if u.get('threefour rebase from'): commands.append(("Detect new revision", detect_new_revision)) - c = "hg rebase --source {threefour rebase from} --dest {threefour picked revision}" + c = "/home/larry/src/hg/hg rebase --source {threefour rebase from} --dest {threefour picked revision}" commands.append(("Rebase subsequent revisions after grafted revision", c)) - commands.append(("Update to head of 3.4 branch", "hg update -r 3.4")) + commands.append(("Update to head of 3.4 branch", "/home/larry/src/hg/hg update -r 3.4")) commands.append(("Mark revision as picked", mark_as_picked)) - commands.append(("Print details of picked revision", "hg log -r {default picked revision}")) + commands.append(("Print details of picked revision", "/home/larry/src/hg/hg log -v -r {default picked revision}")) print() total = len(u['original picked revisions']) current = len(u['picked revisions']) print("Picking revision {default picked revision}".format_map(u) + " ({}/{}):".format(total-current, total)) + + if try_auto_graft: + for i in (0, 1): + text, cmd = commands[i] + return_code = system(cmd.format_map(u)) + u['commands run'].add(text) + if return_code: + print("*" * 80) + print("*" * 80) + print("** Process return code:", return_code) + print("*" * 80) + print("*" * 80) + break + else: + mark_as_picked() + return + self._run_command(commands, u) def finish(self): @@ -678,7 +722,7 @@ """ Recreate the 3.4 branch from scratch. """ - if (not force) and (yes_no() == 'n'): + if (not force) and (yes_no() != 'y'): sys.exit() os.chdir("/home/larry/src/python") while True: @@ -689,11 +733,11 @@ os.rename("/home/larry/src/python/3.4", "/home/larry/src/python/bad3.4") continue break - os.system("hg clone trunk 3.4") + os.system("/home/larry/src/hg/hg clone trunk 3.4") os.chdir("/home/larry/src/python/3.4") - os.system("hg update -r e64ae8b82672") - os.system("hg branch 3.4") - os.system("hg commit -m 'Created release branch for 3.4.'") + os.system("/home/larry/src/hg/hg update -r e64ae8b82672") + os.system("/home/larry/src/hg/hg branch 3.4") + os.system("/home/larry/src/hg/hg commit -m 'Created release branch for 3.4.'") def tar(self): time = now() @@ -707,10 +751,10 @@ remove_dir(tardir) os.chdir("/home/larry/src/python") - system("hg clone 3.4 " + tardir) + system("/home/larry/src/hg/hg clone 3.4 " + tardir) os.chdir(tardir) - system("hg update -r 3.4") + system("/home/larry/src/hg/hg update -r 3.4") remove_dir(".hg") for prefix in ('.hg', '.bzr', '.git'): for filename in glob.glob(prefix + '*'): -- Repository URL: http://hg.python.org/release From solipsis at pitrou.net Fri Mar 7 09:53:47 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 07 Mar 2014 09:53:47 +0100 Subject: [Python-checkins] Daily reference leaks (47f37a688c4c): sum=0 Message-ID: results for 47f37a688c4c on branch "default" -------------------------------------------- test_site leaked [0, -2, 2] references, sum=0 test_site leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6i6sHX', '-x'] From python-checkins at python.org Fri Mar 7 12:05:26 2014 From: python-checkins at python.org (larry.hastings) Date: Fri, 7 Mar 2014 12:05:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Even_more_updates_for_my_t?= =?utf-8?q?hreefour_tool=2E?= Message-ID: <3fgQwQ0V72z7Ll1@mail.python.org> http://hg.python.org/release/rev/da4d4ef62db4 changeset: 81:da4d4ef62db4 user: Larry Hastings date: Fri Mar 07 03:05:26 2014 -0800 summary: Even more updates for my threefour tool. files: 3.4/dryparse/dryparse.py | 10 ++- 3.4/threefourtool.py | 116 +++++++++++++++++++------- 2 files changed, 92 insertions(+), 34 deletions(-) diff --git a/3.4/dryparse/dryparse.py b/3.4/dryparse/dryparse.py --- a/3.4/dryparse/dryparse.py +++ b/3.4/dryparse/dryparse.py @@ -373,6 +373,7 @@ if is_option and (self.type is not bool) and (not self.value_usage): self.value_usage = option_value_usage_formatter(self.type.__name__) + # print("DryArgument name", repr(name), "type", self.type, "default", repr(default), "annotations", repr(annotations), "is_option", is_option ) class OptionError(RuntimeError): pass @@ -604,7 +605,14 @@ nonlocal needs_value argument, is_bool = analyze_option(option) if is_bool: - argument.value = not argument.value + if value is unspecified: + if argument.value is not unspecified: + value = not argument.value + elif argument.default is not unspecified: + value = not argument.default + else: + value = True + argument.set_value(value) else: if value is unspecified: needs_value = argument diff --git a/3.4/threefourtool.py b/3.4/threefourtool.py --- a/3.4/threefourtool.py +++ b/3.4/threefourtool.py @@ -14,6 +14,7 @@ from dryparse import dryparse import glob import os +import os.path import pprint import pickle import shutil @@ -23,7 +24,22 @@ import time +def which(cmd, path="PATH"): + """Find cmd on PATH.""" + if os.path.exists(cmd): + return cmd + if cmd[0] == '/': + return None + for segment in os.getenv(path).split(":"): + program = os.path.normpath(os.path.join(segment, cmd)) + if os.path.exists(program): + return program + return None + + outgoing = "/home/larry/src/python/34outgoing" +# path_to_hg = "/home/larry/src/hg/hg" +hg = which('hg') def system(s): @@ -124,7 +140,7 @@ current_directory = os.getcwd() try: os.chdir("/home/larry/src/python/3.4") - p = subprocess.Popen(["/home/larry/src/hg/hg", "log", "-v"], stdout=subprocess.PIPE) + p = subprocess.Popen([hg, "log", "-v"], stdout=subprocess.PIPE) # with open("/home/larry/src/python/logtxt", "rt", encoding="utf-8") as f: # output = f.read() @@ -276,6 +292,7 @@ # print(" r", rev, "p", parents, "rev to print", rev_to_print) if rev_to_print in seen: continue + print(fields.get('branch'), "printing r", rev, "rev_to_print", rev_to_print) seen.add(rev_to_print) print_rev(rev_to_print, printer) @@ -301,14 +318,8 @@ branch = fields.get('branch') if branch != '3.4': return False - revs = get_user_date_to_revs(fields) - assert (rev, branch) in revs - # print("is 34?", rev, revs) - for r2, branch in revs: - if not branch: - # print(rev, "->", r2) - return r2 - return False + + return default_from_34.get(rev, False) class Tool: unfinished_filename = "/home/larry/.34unfinished" @@ -340,10 +351,13 @@ def status(self): """ Regenerate the status webpage. + + The status webpage is written to the "34outgoing" directory. """ f = open(outgoing + "/merge.status.html", "wt") read_changesets() + print("changesets in 3.4", default_from_34) def printer(*a): print(*a, file=f) @@ -455,9 +469,15 @@ else: cmd() - def pick(self, picked_revision, *picked_revisions): + def pick(self, picked_revision, *picked_revisions, + rebase:('-r', 'Rebase so that all revisions are in chronological order')=True): """ - Cherry-pick a revision from default to 3.4. + Cherry-pick one or more revisions from default to 3.4. + + You can stop in the middle of a 'pick' session with Ctrl-C. The current + state will be pickled, and you can resume it with the 'finish' command, + or abandon it with the 'abandon' command. You can also resume a pick + session by running 'pick' with exactly the same arguments. """ pr = [picked_revision] pr.extend(picked_revisions) @@ -477,7 +497,8 @@ picked_revisions.sort(key=to_default_index) if self.unfinished: - if self.unfinished['original picked revisions'] == picked_revisions: + if ((self.unfinished['original picked revisions'] == picked_revisions) and + (self.unfinished['rebase'] == rebase)): return self.finish() sys.exit("You have unfinished business!\n\nUse the 'finish' command to finish it,\nor the 'abandon' command to abandon it.") @@ -485,6 +506,7 @@ 'function': '_pick', 'picked revisions': picked_revisions, 'original picked revisions': list(picked_revisions), + 'rebase': rebase, } self.finish() @@ -515,7 +537,9 @@ # print("r", r) if r == r_34_first_revision: break - r_default = default_from_34[r] + r_default = default_from_34.get(r) + if r_default is None: + continue i = branches[None].index(r_default) if i >= index: break @@ -565,7 +589,7 @@ read_changesets() os.chdir("/home/larry/src/python/3.4") print("Updating to 3.4 branch:") - os.system("/home/larry/src/hg/hg update -r 3.4") + os.system(hg + " update -r 3.4") print() u = self.unfinished.get print("Picking revisions ", u('picked revisions')) @@ -584,6 +608,7 @@ u = self.unfinished u['EDITOR'] = os.getenv('EDITOR') + u['hg'] = hg patch_path = "/tmp/patch.{default diff from}.to.{default picked revision}.diff".format_map(u) u['patch path'] = patch_path @@ -604,7 +629,7 @@ atexit.register(delete_files, patch_path, commit_message_path) def detect_new_revision(): - output = subprocess.check_output(['/home/larry/src/hg/hg', 'summary']).decode('utf-8').split('\n') + output = subprocess.check_output([hg, 'summary']).decode('utf-8').split('\n') line = output[0] assert line.startswith('parent:') line = line[len('parent:'):] @@ -638,42 +663,47 @@ path = os.path.join(dirpath, filename) os.unlink(path) - changes = subprocess.check_output(['/home/larry/src/hg/hg', 'stat', '-mard']).decode('utf-8') + changes = subprocess.check_output([hg, 'stat', '-mard']).decode('utf-8') try_auto_graft = not bool(changes) if try_auto_graft: print("Attempting auto-graft:") else: print("Skipping auto-graft, there are outstanding changes.") + rebase = u['rebase'] + while u.get('default picked revision'): commands = [] - commands.append(("Update to appropriate revision in 3.4 branch", "/home/larry/src/hg/hg update -r {threefour graft here}")) - commands.append(("Graft revision", "/home/larry/src/hg/hg graft --tool internal:merge {default picked revision}")) + if rebase: + commands.append(("Update to appropriate revision in 3.4 branch", "{hg} update -r {threefour graft here}")) + else: + commands.append(("Update to current revision in 3.4 branch", "{hg} update -r 3.4")) + commands.append(("Graft revision", "{hg} graft --tool internal:merge {default picked revision}")) commands.append(("Handle graft conflict", toggle_graft)) if show_graft: - commands.append(("Graft #1: List conflicts", "/home/larry/src/hg/hg diff resolve -l")) - commands.append(("Graft #2: Mark all conflicts as resolved", "/home/larry/src/hg/hg diff resolve -m")) - commands.append(("Graft #3: Resume the graft", "/home/larry/src/hg/hg diff graft --continue")) + commands.append(("Graft #1: List conflicts", "{hg} resolve -l")) + commands.append(("Graft #2: Mark all conflicts as resolved", "{hg} resolve -m")) + commands.append(("Graft #3: Resume the graft", "{hg} graft --continue")) commands.append(("Graft #4: Clean up .orig and .rej files", remove_orig_and_rej)) - commands.append(("Patch revision (only if graft fails)", toggle_patch)) + commands.append(("Patch revision (for ungraftable merge revisions)", toggle_patch)) if show_patch: - commands.append(("Patch #1: Generate patch", "/home/larry/src/hg/hg diff -r {default diff from} -r {default picked revision} > {patch path}")) + commands.append(("Patch #1: Generate patch", "{hg} diff -r {default diff from} -r {default picked revision} > {patch path}")) commands.append(("Patch #2: Inspect patch", "{EDITOR} {patch path}")) commands.append(("Patch #3: Apply patch", "/usr/bin/patch -p1 < {patch path}")) - commands.append(("Patch #4: Check in patch", "/home/larry/src/hg/hg ci --user '{user}' --date '{date}' --logfile '{commit message path}'")) + commands.append(("Patch #4: Check in patch", "{hg} ci --user '{user}' --date '{date}' --logfile '{commit message path}'")) if u.get('threefour rebase from'): commands.append(("Detect new revision", detect_new_revision)) - c = "/home/larry/src/hg/hg rebase --source {threefour rebase from} --dest {threefour picked revision}" + c = "{hg} rebase --source {threefour rebase from} --dest {threefour picked revision}" commands.append(("Rebase subsequent revisions after grafted revision", c)) - commands.append(("Update to head of 3.4 branch", "/home/larry/src/hg/hg update -r 3.4")) + commands.append(("Update to head of 3.4 branch", "{hg} update -r 3.4")) commands.append(("Mark revision as picked", mark_as_picked)) - commands.append(("Print details of picked revision", "/home/larry/src/hg/hg log -v -r {default picked revision}")) + commands.append(("Print details of picked revision", "{hg} log -v -r {default picked revision}")) print() total = len(u['original picked revisions']) @@ -691,6 +721,7 @@ print("** Process return code:", return_code) print("*" * 80) print("*" * 80) + try_auto_graft = False break else: mark_as_picked() @@ -699,6 +730,9 @@ self._run_command(commands, u) def finish(self): + """ + Continue unfinished business (e.g. pick). + """ try: if not self.unfinished: sys.exit("No unfinished business!") @@ -713,6 +747,9 @@ self._save() def abandon(self, *, force:('-f',)=False): + """ + Abandon unfinished business (e.g. pick). + """ if not self.unfinished: sys.exit("No unfinished business!") if force or yes_no() == 'y': @@ -733,13 +770,19 @@ os.rename("/home/larry/src/python/3.4", "/home/larry/src/python/bad3.4") continue break - os.system("/home/larry/src/hg/hg clone trunk 3.4") + os.system(hg + " clone trunk 3.4") os.chdir("/home/larry/src/python/3.4") - os.system("/home/larry/src/hg/hg update -r e64ae8b82672") - os.system("/home/larry/src/hg/hg branch 3.4") - os.system("/home/larry/src/hg/hg commit -m 'Created release branch for 3.4.'") + os.system(hg + " update -r e64ae8b82672") + os.system(hg + " branch 3.4") + os.system(hg + " commit -m 'Created release branch for 3.4.'") def tar(self): + """ + Create a tarball from the current 3.4 tree. + + The tarball is written to the "34outgoing" directory. + Its filename is constructed based on the current time. + """ time = now() tarbase = "python_3.4.0rc2_" + time tardir = "/home/larry/src/python/" + tarbase @@ -751,10 +794,10 @@ remove_dir(tardir) os.chdir("/home/larry/src/python") - system("/home/larry/src/hg/hg clone 3.4 " + tardir) + system(hg + " clone 3.4 " + tardir) os.chdir(tardir) - system("/home/larry/src/hg/hg update -r 3.4") + system(hg + " update -r 3.4") remove_dir(".hg") for prefix in ('.hg', '.bzr', '.git'): for filename in glob.glob(prefix + '*'): @@ -765,11 +808,18 @@ remove_dir(tardir) def rsync(self): + """ + Sync the current "34outgoing" directory to the public server. + """ os.chdir(outgoing) system("rsync -av * midwinter.com:public_html/3.4.status") def asyncio(self): + """ + Check that the 'asyncio' files in 3.4 are exactly the same + as the equivalent ones in trunk. + """ for dir in ("Lib/asyncio", "Lib/test/test_asyncio"): os.chdir("/home/larry/src/python/3.4/" + dir) os.system("diff . /home/larry/src/python/trunk/" + dir) -- Repository URL: http://hg.python.org/release From python-checkins at python.org Fri Mar 7 18:27:19 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 7 Mar 2014 18:27:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320812=3A_Add_a_sh?= =?utf-8?q?ort_opener_to_the_Python_2/3_porting_HOWTO=2E?= Message-ID: <3fgbP33DXNz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/2a922153463e changeset: 89497:2a922153463e user: Brett Cannon date: Fri Mar 07 12:27:10 2014 -0500 summary: Issue #20812: Add a short opener to the Python 2/3 porting HOWTO. Thanks to Nick Coghlan for the suggestion. files: Doc/howto/pyporting.rst | 31 ++++++++++++++++++++++++++++- 1 files changed, 30 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -26,6 +26,32 @@ For help with porting, you can email the python-porting_ mailing list with questions. +The Short Version +================= + +* Decide what's the oldest version of Python 2 you want to support (if at all) +* Make sure you have a thorough test suite and use continuous integration + testing to make sure you stay compatible with the versions of Python you care + about +* If you have dependencies, check their Python 3 status using caniusepython3 + (`command-line tool `__, + `web app `__) + +With that done, your options are: + +* If you are dropping Python 2 support, use 2to3_ to port to Python 3 +* If you are keeping Python 2 support, then start writing Python 2/3-compatible + code starting **TODAY** + + + If you have dependencies that have not been ported, reach out to them to port + their project while working to make your code compatible with Python 3 so + you're ready when your dependencies are all ported + + If all your dependencies have been ported (or you have none), go ahead and + port to Python 3 + +* If you are creating a new project that wants to have 2/3 compatibility, + code in Python 3 and then backport to Python 2 + Before You Begin ================ @@ -548,7 +574,10 @@ If you have Python 3 code and have little interest in supporting Python 2 you can use 3to2_ to translate from Python 3 code to Python 2 code. This is only -recommended if you don't plan to heavily support Python 2 users. +recommended if you don't plan to heavily support Python 2 users. Otherwise +write your code for Python 3 and then backport as far back as you want. This +is typically easier than going from Python 2 to 3 as you will have worked out +any difficulties with e.g. bytes/strings, etc. Other Resources -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 7 18:29:35 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 7 Mar 2014 18:29:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwODEz?= =?utf-8?q?=3A_Backport_Python_2/3_HOWTO_updates?= Message-ID: <3fgbRg3Fr3z7Ljv@mail.python.org> http://hg.python.org/cpython/rev/a24085e1b1f5 changeset: 89498:a24085e1b1f5 branch: 3.3 parent: 89495:7a7ff2015099 user: Brett Cannon date: Fri Mar 07 12:28:35 2014 -0500 summary: Issue #20813: Backport Python 2/3 HOWTO updates files: Doc/howto/pyporting.rst | 611 +++++++++++---------------- 1 files changed, 245 insertions(+), 366 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -10,258 +10,211 @@ With Python 3 being the future of Python while Python 2 is still in active use, it is good to have your project available for both major releases of - Python. This guide is meant to help you choose which strategy works best - for your project to support both Python 2 & 3 along with how to execute - that strategy. + Python. This guide is meant to help you figure out how best to support both + Python 2 & 3 simultaneously. If you are looking to port an extension module instead of pure Python code, please see :ref:`cporting-howto`. + If you would like to read one core Python developer's take on why Python 3 + came into existence, you can read Nick Coghlan's `Python 3 Q & A`_. -Choosing a Strategy -=================== + If you prefer to read a (free) book on porting a project to Python 3, + consider reading `Porting to Python 3`_ by Lennart Regebro which should cover + much of what is discussed in this HOWTO. -When a project chooses to support both Python 2 & 3, -a decision needs to be made as to how to go about accomplishing that goal. -The chosen strategy will depend on how large the project's existing -codebase is and how much divergence you want from your current Python 2 codebase -(e.g., changing your code to work simultaneously with Python 2 and 3). + For help with porting, you can email the python-porting_ mailing list with + questions. -If you would prefer to maintain a codebase which is semantically **and** -syntactically compatible with Python 2 & 3 simultaneously, you can write -:ref:`use_same_source`. While this tends to lead to somewhat non-idiomatic -code, it does mean you keep a rapid development process for you, the developer. +The Short Version +================= -If your project is brand-new or does not have a large codebase, then you may -want to consider writing/porting :ref:`all of your code for Python 3 -and use 3to2 ` to port your code for Python 2. +* Decide what's the oldest version of Python 2 you want to support (if at all) +* Make sure you have a thorough test suite and use continuous integration + testing to make sure you stay compatible with the versions of Python you care + about +* If you have dependencies, check their Python 3 status using caniusepython3 + (`command-line tool `__, + `web app `__) -Finally, you do have the option of :ref:`using 2to3 ` to translate -Python 2 code into Python 3 code (with some manual help). This can take the -form of branching your code and using 2to3 to start a Python 3 branch. You can -also have users perform the translation at installation time automatically so -that you only have to maintain a Python 2 codebase. +With that done, your options are: -Regardless of which approach you choose, porting is not as hard or -time-consuming as you might initially think. You can also tackle the problem -piece-meal as a good portion of porting is simply updating your code to follow -current best practices in a Python 2/3 compatible way. +* If you are dropping Python 2 support, use 2to3_ to port to Python 3 +* If you are keeping Python 2 support, then start writing Python 2/3-compatible + code starting **TODAY** + + If you have dependencies that have not been ported, reach out to them to port + their project while working to make your code compatible with Python 3 so + you're ready when your dependencies are all ported + + If all your dependencies have been ported (or you have none), go ahead and + port to Python 3 -Universal Bits of Advice ------------------------- +* If you are creating a new project that wants to have 2/3 compatibility, + code in Python 3 and then backport to Python 2 -Regardless of what strategy you pick, there are a few things you should -consider. -One is make sure you have a robust test suite. You need to make sure everything -continues to work, just like when you support a new minor/feature release of -Python. This means making sure your test suite is thorough and is ported -properly between Python 2 & 3. You will also most likely want to use something -like tox_ to automate testing between both a Python 2 and Python 3 interpreter. +Before You Begin +================ -Two, once your project has Python 3 support, make sure to add the proper -classifier on the Cheeseshop_ (PyPI_). To have your project listed as Python 3 -compatible it must have the -`Python 3 classifier `_ -(from -http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/):: +If your project is on the Cheeseshop_/PyPI_, make sure it has the proper +`trove classifiers`_ to signify what versions of Python it **currently** +supports. At minimum you should specify the major version(s), e.g. +``Programming Language :: Python :: 2`` if your project currently only supports +Python 2. It is preferrable that you be as specific as possible by listing every +major/minor version of Python that you support, e.g. if your project supports +Python 2.6 and 2.7, then you want the classifiers of:: - setup( - name='Your Library', - version='1.0', - classifiers=[ - # make sure to use :: Python *and* :: Python :: 3 so - # that pypi can list the package on the python 3 page - 'Programming Language :: Python', - 'Programming Language :: Python :: 3' - ], - packages=['yourlibrary'], - # make sure to add custom_fixers to the MANIFEST.in - include_package_data=True, - # ... - ) + Programming Language :: Python :: 2 + Programming Language :: Python :: 2.6 + Programming Language :: Python :: 2.7 +Once your project supports Python 3 you will want to go back and add the +appropriate classifiers for Python 3 as well. This is important as setting the +``Programming Language :: Python :: 3`` classifier will lead to your project +being listed under the `Python 3 Packages`_ section of PyPI. -Doing so will cause your project to show up in the -`Python 3 packages list -`_. You will know -you set the classifier properly as visiting your project page on the Cheeseshop -will show a Python 3 logo in the upper-left corner of the page. +Make sure you have a robust test suite. You need to +make sure everything continues to work, just like when you support a new +minor/feature release of Python. This means making sure your test suite is +thorough and is ported properly between Python 2 & 3 (consider using coverage_ +to measure that you have effective test coverage). You will also most likely +want to use something like tox_ to automate testing between all of your +supported versions of Python. You will also want to **port your tests first** so +that you can make sure that you detect breakage during the transition. Tests also +tend to be simpler than the code they are testing so it gives you an idea of how +easy it can be to port code. -Three, the six_ project provides a library which helps iron out differences -between Python 2 & 3. If you find there is a sticky point that is a continual -point of contention in your translation or maintenance of code, consider using -a source-compatible solution relying on six. If you have to create your own -Python 2/3 compatible solution, you can use ``sys.version_info[0] >= 3`` as a -guard. - -Four, read all the approaches. Just because some bit of advice applies to one -approach more than another doesn't mean that some advice doesn't apply to other -strategies. This is especially true of whether you decide to use 2to3 or be -source-compatible; tips for one approach almost always apply to the other. - -Five, drop support for older Python versions if possible. `Python 2.5`_ +Drop support for older Python versions if possible. `Python 2.5`_ introduced a lot of useful syntax and libraries which have become idiomatic in Python 3. `Python 2.6`_ introduced future statements which makes compatibility much easier if you are going from Python 2 to 3. -`Python 2.7`_ continues the trend in the stdlib. So choose the newest version +`Python 2.7`_ continues the trend in the stdlib. Choose the newest version of Python which you believe can be your minimum support version and work from there. -Six, target the newest version of Python 3 that you can. Beyond just the usual +Target the newest version of Python 3 that you can. Beyond just the usual bugfixes, compatibility has continued to improve between Python 2 and 3 as time -has passed. This is especially true for Python 3.3 where the ``u`` prefix for -strings is allowed, making source-compatible Python code easier. +has passed. E.g. Python 3.3 added back the ``u`` prefix for +strings, making source-compatible Python code easier to write. -Seven, make sure to look at the `Other Resources`_ for tips from other people -which may help you out. +Writing Source-Compatible Python 2/3 Code +========================================= -.. _tox: http://codespeak.net/tox/ -.. _Cheeseshop: -.. _PyPI: http://pypi.python.org/ -.. _six: http://packages.python.org/six -.. _Python 2.7: http://www.python.org/2.7.x -.. _Python 2.6: http://www.python.org/2.6.x -.. _Python 2.5: http://www.python.org/2.5.x -.. _Python 2.4: http://www.python.org/2.4.x -.. _Python 2.3: http://www.python.org/2.3.x -.. _Python 2.2: http://www.python.org/2.2.x +Over the years the Python community has discovered that the easiest way to +support both Python 2 and 3 in parallel is to write Python code that works in +either version. While this might sound counter-intuitive at first, it actually +is not difficult and typically only requires following some select +(non-idiomatic) practices and using some key projects to help make bridging +between Python 2 and 3 easier. +Projects to Consider +-------------------- -.. _use_3to2: +The lowest level library for suppoting Python 2 & 3 simultaneously is six_. +Reading through its documentation will give you an idea of where exactly the +Python language changed between versions 2 & 3 and thus what you will want the +library to help you continue to support. -Python 3 and 3to2 -================= +To help automate porting your code over to using six, you can use +modernize_. This project will attempt to rewrite your code to be as modern as +possible while using six to smooth out any differences between Python 2 & 3. -If you are starting a new project or your codebase is small enough, you may -want to consider writing your code for Python 3 and backporting to Python 2 -using 3to2_. Thanks to Python 3 being more strict about things than Python 2 -(e.g., bytes vs. strings), the source translation can be easier and more -straightforward than from Python 2 to 3. Plus it gives you more direct -experience developing in Python 3 which, since it is the future of Python, is a -good thing long-term. +If you want to write your compatible code to feel more like Python 3 there is +the future_ project. It tries to provide backports of objects from Python 3 so +that you can use them from Python 2-compatible code, e.g. replacing the +``bytes`` type from Python 2 with the one from Python 3. +It also provides a translation script like modernize (its translation code is +actually partially based on it) to help start working with a pre-existing code +base. It is also unique in that its translation script will also port Python 3 +code backwards as well as Python 2 code forwards. -A drawback of this approach is that 3to2 is a third-party project. This means -that the Python core developers (and thus this guide) can make no promises -about how well 3to2 works at any time. There is nothing to suggest, though, -that 3to2 is not a high-quality project. +Tips & Tricks +------------- -.. _3to2: https://bitbucket.org/amentajo/lib3to2/overview - - -.. _use_2to3: - -Python 2 and 2to3 -================= - -Included with Python since 2.6, the 2to3_ tool (and :mod:`lib2to3` module) -helps with porting Python 2 to Python 3 by performing various source -translations. This is a perfect solution for projects which wish to branch -their Python 3 code from their Python 2 codebase and maintain them as -independent codebases. You can even begin preparing to use this approach -today by writing future-compatible Python code which works cleanly in -Python 2 in conjunction with 2to3; all steps outlined below will work -with Python 2 code up to the point when the actual use of 2to3 occurs. - -Use of 2to3 as an on-demand translation step at install time is also possible, -preventing the need to maintain a separate Python 3 codebase, but this approach -does come with some drawbacks. While users will only have to pay the -translation cost once at installation, you as a developer will need to pay the -cost regularly during development. If your codebase is sufficiently large -enough then the translation step ends up acting like a compilation step, -robbing you of the rapid development process you are used to with Python. -Obviously the time required to translate a project will vary, so do an -experimental translation just to see how long it takes to evaluate whether you -prefer this approach compared to using :ref:`use_same_source` or simply keeping -a separate Python 3 codebase. - -Below are the typical steps taken by a project which tries to support -Python 2 & 3 while keeping the code directly executable by Python 2. - +To help with writing source-compatible code using one of the projects mentioned +in `Projects to Consider`_, consider following the below suggestions. Some of +them are handled by the suggested projects, so if you do use one of them then +read their documentation first to see which suggestions below will taken care of +for you. Support Python 2.7 ------------------- +////////////////// As a first step, make sure that your project is compatible with `Python 2.7`_. This is just good to do as Python 2.7 is the last release of Python 2 and thus will be used for a rather long time. It also allows for use of the ``-3`` flag -to Python to help discover places in your code which 2to3 cannot handle but are -known to cause issues. +to Python to help discover places in your code where compatibility might be an +issue (the ``-3`` flag is in Python 2.6 but Python 2.7 adds more warnings). Try to Support `Python 2.6`_ and Newer Only -------------------------------------------- +/////////////////////////////////////////// While not possible for all projects, if you can support `Python 2.6`_ and newer **only**, your life will be much easier. Various future statements, stdlib additions, etc. exist only in Python 2.6 and later which greatly assist in -porting to Python 3. But if you project must keep support for `Python 2.5`_ (or -even `Python 2.4`_) then it is still possible to port to Python 3. +supporting Python 3. But if you project must keep support for `Python 2.5`_ then +it is still possible to simultaneously support Python 3. Below are the benefits you gain if you only have to support Python 2.6 and newer. Some of these options are personal choice while others are **strongly** recommended (the ones that are more for personal choice are labeled as such). If you continue to support older versions of Python then you -at least need to watch out for situations that these solutions fix. +at least need to watch out for situations that these solutions fix and handle +them appropriately (which is where library help from e.g. six_ comes in handy). ``from __future__ import print_function`` ''''''''''''''''''''''''''''''''''''''''' -This is a personal choice. 2to3 handles the translation from the print -statement to the print function rather well so this is an optional step. This -future statement does help, though, with getting used to typing -``print('Hello, World')`` instead of ``print 'Hello, World'``. +It will not only get you used to typing ``print()`` as a function instead of a +statement, but it will also give you the various benefits the function has over +the Python 2 statement (six_ provides a function if you support Python 2.5 or +older). ``from __future__ import unicode_literals`` ''''''''''''''''''''''''''''''''''''''''''' -Another personal choice. You can always mark what you want to be a (unicode) -string with a ``u`` prefix to get the same effect. But regardless of whether -you use this future statement or not, you **must** make sure you know exactly -which Python 2 strings you want to be bytes, and which are to be strings. This -means you should, **at minimum** mark all strings that are meant to be text -strings with a ``u`` prefix if you do not use this future statement. Python 3.3 -allows strings to continue to have the ``u`` prefix (it's a no-op in that case) -to make it easier for code to be source-compatible between Python 2 & 3. +If you choose to use this future statement then all string literals in +Python 2 will be assumed to be Unicode (as is already the case in Python 3). +If you choose not to use this future statement then you should mark all of your +text strings with a ``u`` prefix and only support Python 3.3 or newer. But you +are **strongly** advised to do one or the other (six_ provides a function in +case you don't want to use the future statement **and** you want to support +Python 3.2 or older). -Bytes literals -'''''''''''''' +Bytes/string literals +''''''''''''''''''''' -This is a **very** important one. The ability to prefix Python 2 strings that -are meant to contain bytes with a ``b`` prefix help to very clearly delineate -what is and is not a Python 3 string. When you run 2to3 on code, all Python 2 -strings become Python 3 strings **unless** they are prefixed with ``b``. +This is a **very** important one. Prefix Python 2 strings that +are meant to contain bytes with a ``b`` prefix to very clearly delineate +what is and is not a Python 3 text string (six_ provides a function to use for +Python 2.5 compatibility). This point cannot be stressed enough: make sure you know what all of your string -literals in Python 2 are meant to become in Python 3. Any string literal that +literals in Python 2 are meant to be in Python 3. Any string literal that should be treated as bytes should have the ``b`` prefix. Any string literal that should be Unicode/text in Python 2 should either have the ``u`` literal (supported, but ignored, in Python 3.3 and later) or you should have ``from __future__ import unicode_literals`` at the top of the file. But the key -point is you should know how Python 3 will treat everyone one of your string +point is you should know how Python 3 will treat every one one of your string literals and you should mark them as appropriate. There are some differences between byte literals in Python 2 and those in Python 3 thanks to the bytes type just being an alias to ``str`` in Python 2. -Probably the biggest "gotcha" is that indexing results in different values. In -Python 2, the value of ``b'py'[1]`` is ``'y'``, while in Python 3 it's ``121``. -You can avoid this disparity by always slicing at the size of a single element: -``b'py'[1:2]`` is ``'y'`` in Python 2 and ``b'y'`` in Python 3 (i.e., close -enough). +See the `Handle Common "Gotchas"`_ section for what to watch out for. -You cannot concatenate bytes and strings in Python 3. But since Python -2 has bytes aliased to ``str``, it will succeed: ``b'a' + u'b'`` works in -Python 2, but ``b'a' + 'b'`` in Python 3 is a :exc:`TypeError`. A similar issue -also comes about when doing comparisons between bytes and strings. +``from __future__ import absolute_import`` +'''''''''''''''''''''''''''''''''''''''''' +Discussed in more detail below, but you should use this future statement to +prevent yourself from accidentally using implicit relative imports. Supporting `Python 2.5`_ and Newer Only ---------------------------------------- +/////////////////////////////////////// If you are supporting `Python 2.5`_ and newer there are still some features of Python that you can utilize. @@ -271,7 +224,7 @@ '''''''''''''''''''''''''''''''''''''''''' Implicit relative imports (e.g., importing ``spam.bacon`` from within -``spam.eggs`` with the statement ``import bacon``) does not work in Python 3. +``spam.eggs`` with the statement ``import bacon``) do not work in Python 3. This future statement moves away from that and allows the use of explicit relative imports (e.g., ``from . import bacon``). @@ -281,7 +234,7 @@ the statement, but you still want the __future__ statement to prevent implicit relative imports. In `Python 2.7`_ the __future__ statement is not needed. In other words, unless you are only supporting Python 2.7 or a version earlier -than Python 2.5, use the __future__ statement. +than Python 2.5, use this __future__ statement. Mark all Unicode strings with a ``u`` prefix @@ -290,17 +243,65 @@ While Python 2.6 has a ``__future__`` statement to automatically cause Python 2 to treat all string literals as Unicode, Python 2.5 does not have that shortcut. This means you should go through and mark all string literals with a ``u`` -prefix to turn them explicitly into Unicode strings where appropriate. That -leaves all unmarked string literals to be considered byte literals in Python 3. +prefix to turn them explicitly into text strings where appropriate and only +support Python 3.3 or newer. Otherwise use a project like six_ which provides a +function to pass all text string literals through. +Capturing the Currently Raised Exception +'''''''''''''''''''''''''''''''''''''''' + +In Python 2.5 and earlier the syntax to access the current exception is:: + + try: + raise Exception() + except Exception, exc: + # Current exception is 'exc'. + pass + +This syntax changed in Python 3 (and backported to `Python 2.6`_ and later) +to:: + + try: + raise Exception() + except Exception as exc: + # Current exception is 'exc'. + # In Python 3, 'exc' is restricted to the block; in Python 2.6/2.7 it will "leak". + pass + +Because of this syntax change you must change how you capture the current +exception in Python 2.5 and earlier to:: + + try: + raise Exception() + except Exception: + import sys + exc = sys.exc_info()[1] + # Current exception is 'exc'. + pass + +You can get more information about the raised exception from +:func:`sys.exc_info` than simply the current exception instance, but you most +likely don't need it. + +.. note:: + In Python 3, the traceback is attached to the exception instance + through the ``__traceback__`` attribute. If the instance is saved in + a local variable that persists outside of the ``except`` block, the + traceback will create a reference cycle with the current frame and its + dictionary of local variables. This will delay reclaiming dead + resources until the next cyclic :term:`garbage collection` pass. + + In Python 2, this problem only occurs if you save the traceback itself + (e.g. the third element of the tuple returned by :func:`sys.exc_info`) + in a variable. + Handle Common "Gotchas" ------------------------ +/////////////////////// -There are a few things that just consistently come up as sticking points for -people which 2to3 cannot handle automatically or can easily be done in Python 2 -to help modernize your code. +These are things to watch out for no matter what version of Python 2 you are +supporting which are not syntactic considerations. ``from __future__ import division`` @@ -357,9 +358,9 @@ the bytes/string dichotomy. Because Python 2 allowed the ``str`` type to hold textual data, people have over the years been rather loose in their delineation of what ``str`` instances held text compared to bytes. In Python 3 you cannot -be so care-free anymore and need to properly handle the difference. The key +be so care-free anymore and need to properly handle the difference. The key to handling this issue is to make sure that **every** string literal in your -Python 2 code is either syntactically of functionally marked as either bytes or +Python 2 code is either syntactically or functionally marked as either bytes or text data. After this is done you then need to make sure your APIs are designed to either handle a specific type or made to be properly polymorphic. @@ -466,14 +467,7 @@ happen to use the ``unicode(self).encode('utf8')`` idiom as the body of your ``__str__()`` method). -There are two ways to solve this issue. One is to use a custom 2to3 fixer. The -blog post at http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ -specifies how to do this. That will allow 2to3 to change all instances of ``def -__unicode(self): ...`` to ``def __str__(self): ...``. This does require that you -define your ``__str__()`` method in Python 2 before your ``__unicode__()`` -method. - -The other option is to use a mixin class. This allows you to only define a +You can use a mixin class to work around this. This allows you to only define a ``__unicode__()`` method for your class and let the mixin derive ``__str__()`` for you (code from http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/):: @@ -516,6 +510,7 @@ Even better is to use the documented attributes the exception provides. + Don't use ``__getslice__`` & Friends '''''''''''''''''''''''''''''''''''' @@ -527,23 +522,23 @@ Updating doctests ''''''''''''''''' -2to3_ will attempt to generate fixes for doctests that it comes across. It's -not perfect, though. If you wrote a monolithic set of doctests (e.g., a single -docstring containing all of your doctests), you should at least consider -breaking the doctests up into smaller pieces to make it more manageable to fix. -Otherwise it might very well be worth your time and effort to port your tests -to :mod:`unittest`. +Don't forget to make them Python 2/3 compatible as well. If you wrote a +monolithic set of doctests (e.g., a single docstring containing all of your +doctests), you should at least consider breaking the doctests up into smaller +pieces to make it more manageable to fix. Otherwise it might very well be worth +your time and effort to port your tests to :mod:`unittest`. -Update `map` for imbalanced input sequences -''''''''''''''''''''''''''''''''''''''''''' +Update ``map`` for imbalanced input sequences +''''''''''''''''''''''''''''''''''''''''''''' -With Python 2, `map` would pad input sequences of unequal length with -`None` values, returning a sequence as long as the longest input sequence. +With Python 2, when ``map`` was given more than one input sequence it would pad +the shorter sequences with `None` values, returning a sequence as long as the +longest input sequence. -With Python 3, if the input sequences to `map` are of unequal length, `map` +With Python 3, if the input sequences to ``map`` are of unequal length, ``map`` will stop at the termination of the shortest of the sequences. For full -compatibility with `map` from Python 2.x, also wrap the sequences in +compatibility with ``map`` from Python 2.x, wrap the sequence arguments in :func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes ``list(map(func, itertools.zip_longest(*sequences)))``. @@ -552,176 +547,37 @@ When you run your application's test suite, run it using the ``-3`` flag passed to Python. This will cause various warnings to be raised during execution about -things that 2to3 cannot handle automatically (e.g., modules that have been -removed). Try to eliminate those warnings to make your code even more portable -to Python 3. +things that are semantic changes between Python 2 and 3. Try to eliminate those +warnings to make your code even more portable to Python 3. -Run 2to3 --------- +Alternative Approaches +====================== -Once you have made your Python 2 code future-compatible with Python 3, it's -time to use 2to3_ to actually port your code. +While supporting Python 2 & 3 simultaneously is typically the preferred choice +by people so that they can continue to improve code and have it work for the +most number of users, your life may be easier if you only have to support one +major version of Python going forward. +Supporting Only Python 3 Going Forward From Python 2 Code +--------------------------------------------------------- -Manually -'''''''' +If you have Python 2 code but going forward only want to improve it as Python 3 +code, then you can use 2to3_ to translate your Python 2 code to Python 3 code. +This is only recommended, though, if your current version of your project is +going into maintenance mode and you want all new features to be exclusive to +Python 3. -To manually convert source code using 2to3_, you use the ``2to3`` script that -is installed with Python 2.6 and later.:: - 2to3 +Backporting Python 3 code to Python 2 +------------------------------------- -This will cause 2to3 to write out a diff with all of the fixers applied for the -converted source code. If you would like 2to3 to go ahead and apply the changes -you can pass it the ``-w`` flag:: - - 2to3 -w - -There are other flags available to control exactly which fixers are applied, -etc. - - -During Installation -''''''''''''''''''' - -When a user installs your project for Python 3, you can have either -:mod:`distutils` or Distribute_ run 2to3_ on your behalf. -For distutils, use the following idiom:: - - try: # Python 3 - from distutils.command.build_py import build_py_2to3 as build_py - except ImportError: # Python 2 - from distutils.command.build_py import build_py - - setup(cmdclass = {'build_py': build_py}, - # ... - ) - -For Distribute:: - - setup(use_2to3=True, - # ... - ) - -This will allow you to not have to distribute a separate Python 3 version of -your project. It does require, though, that when you perform development that -you at least build your project and use the built Python 3 source for testing. - - -Verify & Test -------------- - -At this point you should (hopefully) have your project converted in such a way -that it works in Python 3. Verify it by running your unit tests and making sure -nothing has gone awry. If you miss something then figure out how to fix it in -Python 3, backport to your Python 2 code, and run your code through 2to3 again -to verify the fix transforms properly. - - -.. _2to3: http://docs.python.org/py3k/library/2to3.html -.. _Distribute: http://packages.python.org/distribute/ - - -.. _use_same_source: - -Python 2/3 Compatible Source -============================ - -While it may seem counter-intuitive, you can write Python code which is -source-compatible between Python 2 & 3. It does lead to code that is not -entirely idiomatic Python (e.g., having to extract the currently raised -exception from ``sys.exc_info()[1]``), but it can be run under Python 2 -**and** Python 3 without using 2to3_ as a translation step (although the tool -should be used to help find potential portability problems). This allows you to -continue to have a rapid development process regardless of whether you are -developing under Python 2 or Python 3. Whether this approach or using -:ref:`use_2to3` works best for you will be a per-project decision. - -To get a complete idea of what issues you will need to deal with, see the -`What's New in Python 3.0`_. Others have reorganized the data in other formats -such as http://docs.pythonsprints.com/python3_porting/py-porting.html\ . - -The following are some steps to take to try to support both Python 2 & 3 from -the same source code. - - -.. _What's New in Python 3.0: http://docs.python.org/release/3.0/whatsnew/3.0.html - - -Follow The Steps for Using 2to3_ --------------------------------- - -All of the steps outlined in how to -:ref:`port Python 2 code with 2to3 ` apply -to creating a Python 2/3 codebase. This includes trying only support Python 2.6 -or newer (the :mod:`__future__` statements work in Python 3 without issue), -eliminating warnings that are triggered by ``-3``, etc. - -You should even consider running 2to3_ over your code (without committing the -changes). This will let you know where potential pain points are within your -code so that you can fix them properly before they become an issue. - - -Use six_ --------- - -The six_ project contains many things to help you write portable Python code. -You should make sure to read its documentation from beginning to end and use -any and all features it provides. That way you will minimize any mistakes you -might make in writing cross-version code. - - -Capturing the Currently Raised Exception ----------------------------------------- - -One change between Python 2 and 3 that will require changing how you code (if -you support `Python 2.5`_ and earlier) is -accessing the currently raised exception. In Python 2.5 and earlier the syntax -to access the current exception is:: - - try: - raise Exception() - except Exception, exc: - # Current exception is 'exc' - pass - -This syntax changed in Python 3 (and backported to `Python 2.6`_ and later) -to:: - - try: - raise Exception() - except Exception as exc: - # Current exception is 'exc' - # In Python 3, 'exc' is restricted to the block; Python 2.6 will "leak" - pass - -Because of this syntax change you must change to capturing the current -exception to:: - - try: - raise Exception() - except Exception: - import sys - exc = sys.exc_info()[1] - # Current exception is 'exc' - pass - -You can get more information about the raised exception from -:func:`sys.exc_info` than simply the current exception instance, but you most -likely don't need it. - -.. note:: - In Python 3, the traceback is attached to the exception instance - through the ``__traceback__`` attribute. If the instance is saved in - a local variable that persists outside of the ``except`` block, the - traceback will create a reference cycle with the current frame and its - dictionary of local variables. This will delay reclaiming dead - resources until the next cyclic :term:`garbage collection` pass. - - In Python 2, this problem only occurs if you save the traceback itself - (e.g. the third element of the tuple returned by :func:`sys.exc_info`) - in a variable. +If you have Python 3 code and have little interest in supporting Python 2 you +can use 3to2_ to translate from Python 3 code to Python 2 code. This is only +recommended if you don't plan to heavily support Python 2 users. Otherwise +write your code for Python 3 and then backport as far back as you want. This +is typically easier than going from Python 2 to 3 as you will have worked out +any difficulties with e.g. bytes/strings, etc. Other Resources @@ -729,18 +585,41 @@ The authors of the following blog posts, wiki pages, and books deserve special thanks for making public their tips for porting Python 2 code to Python 3 (and -thus helping provide information for this document): +thus helping provide information for this document and its various revisions +over the years): +* http://wiki.python.org/moin/PortingPythonToPy3k * http://python3porting.com/ * http://docs.pythonsprints.com/python3_porting/py-porting.html * http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/ * http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html * http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ * http://lucumr.pocoo.org/2010/2/11/porting-to-python-3-a-guide/ -* http://wiki.python.org/moin/PortingPythonToPy3k * https://wiki.ubuntu.com/Python/3 If you feel there is something missing from this document that should be added, please email the python-porting_ mailing list. + + +.. _2to3: http://docs.python.org/2/library/2to3.html +.. _3to2: https://pypi.python.org/pypi/3to2 +.. _Cheeseshop: PyPI_ +.. _coverage: https://pypi.python.org/pypi/coverage +.. _future: http://python-future.org/ +.. _modernize: https://github.com/mitsuhiko/python-modernize +.. _Porting to Python 3: http://python3porting.com/ +.. _PyPI: http://pypi.python.org/ +.. _Python 2.2: http://www.python.org/2.2.x +.. _Python 2.5: http://www.python.org/2.5.x +.. _Python 2.6: http://www.python.org/2.6.x +.. _Python 2.7: http://www.python.org/2.7.x +.. _Python 2.5: http://www.python.org/2.5.x +.. _Python 3.3: http://www.python.org/3.3.x +.. _Python 3 Packages: https://pypi.python.org/pypi?:action=browse&c=533&show=all +.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html .. _python-porting: http://mail.python.org/mailman/listinfo/python-porting +.. _six: https://pypi.python.org/pypi/six +.. _tox: https://pypi.python.org/pypi/tox +.. _trove classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 7 18:29:36 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 7 Mar 2014 18:29:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_null_merge_for_issue_=2320812?= Message-ID: <3fgbRh5Hsdz7LkM@mail.python.org> http://hg.python.org/cpython/rev/c83ce2a1841c changeset: 89499:c83ce2a1841c parent: 89497:2a922153463e parent: 89498:a24085e1b1f5 user: Brett Cannon date: Fri Mar 07 12:29:24 2014 -0500 summary: null merge for issue #20812 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 02:26:11 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 02:26:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzEwMTk3OiBVcGRh?= =?utf-8?q?te_get=5Bstatus=5Doutput_versionchanged_with_actual_version=2E?= Message-ID: <3fgp1b65G7z7LjN@mail.python.org> http://hg.python.org/cpython/rev/34df43c9c74a changeset: 89500:34df43c9c74a branch: 3.3 parent: 89498:a24085e1b1f5 user: R David Murray date: Fri Mar 07 20:04:17 2014 -0500 summary: #10197: Update get[status]output versionchanged with actual version. This was perhaps on the border between a bug fix and a feature since the Python3 docs did not originally say it was unix only. However, the functions never worked anywhere but unix, and the docs were changed to say it was unix only well before the windows support was added. Unfortunately, windows support was added in 3.3.4 as well as 3.4. That leaves us in the uncomfortable position of needing the tag to say "version changed: 3.3.4" :( files: Doc/library/subprocess.rst | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1072,8 +1072,10 @@ >>> subprocess.getstatusoutput('/bin/junk') (256, 'sh: /bin/junk: not found') - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added .. function:: getoutput(cmd) @@ -1086,8 +1088,10 @@ >>> subprocess.getoutput('ls /bin/ls') '/bin/ls' - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added Notes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 02:26:13 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 02:26:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=2310197=3A_Update_get=5Bstatus=5Doutput_versionchanged_?= =?utf-8?q?with_actual_version=2E?= Message-ID: <3fgp1d1fGmz7LlL@mail.python.org> http://hg.python.org/cpython/rev/ee277b383d33 changeset: 89501:ee277b383d33 parent: 89499:c83ce2a1841c parent: 89500:34df43c9c74a user: R David Murray date: Fri Mar 07 20:04:41 2014 -0500 summary: #10197: Update get[status]output versionchanged with actual version. This was perhaps on the border between a bug fix and a feature since the Python3 docs did not originally say it was unix only. However, the functions never worked anywhere but unix, and the docs were changed to say it was unix only well before the windows support was added. Unfortunately, windows support was added in 3.3.4 as well as 3.4. That leaves us in the uncomfortable position of needing the tag to say "version changed: 3.3.4" :( files: Doc/library/subprocess.rst | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1091,8 +1091,10 @@ >>> subprocess.getstatusoutput('/bin/junk') (256, 'sh: /bin/junk: not found') - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added .. function:: getoutput(cmd) @@ -1105,8 +1107,10 @@ >>> subprocess.getoutput('ls /bin/ls') '/bin/ls' - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added Notes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 02:26:14 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 02:26:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_get=5Bstatus?= =?utf-8?q?=7Coutput=5D_works_on_windows=2E?= Message-ID: <3fgp1f4Wmjz7LlX@mail.python.org> http://hg.python.org/cpython/rev/512032de01ab changeset: 89502:512032de01ab user: R David Murray date: Fri Mar 07 20:06:24 2014 -0500 summary: whatsnew: get[status|output] works on windows. Since 3.3.4.... :( files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1246,6 +1246,10 @@ be used to provide the contents of ``stdin`` for the command that is run. (Contributed by Zack Weinberg in :issue:`16624`.) +:func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now +work on Windows. This change was actually made in 3.3.4. (Contributed +by Tim Golden in :issue:`10197`.) + sunau ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 03:02:06 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 03:02:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_cp273_codec_?= =?utf-8?b?KCMxMDkwNzc5Nyk=?= Message-ID: <3fgpq25C68z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/d843a1caba78 changeset: 89503:d843a1caba78 user: R David Murray date: Fri Mar 07 21:00:34 2014 -0500 summary: whatsnew: cp273 codec (#10907797) Also updated the docs and added the aliases mentioned by the references. files: Doc/library/codecs.rst | 4 ++++ Doc/whatsnew/3.4.rst | 3 +++ Lib/encodings/aliases.py | 5 +++++ 3 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -971,6 +971,10 @@ +-----------------+--------------------------------+--------------------------------+ | cp037 | IBM037, IBM039 | English | +-----------------+--------------------------------+--------------------------------+ +| cp273 | 273, IBM273, csIBM273 | German | +| | | | +| | | .. versionadded:: 3.4 | ++-----------------+--------------------------------+--------------------------------+ | cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | +-----------------+--------------------------------+--------------------------------+ | cp437 | 437, IBM437 | English | diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -365,6 +365,9 @@ Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. +* New EBCDIC :ref:`codec ` ``cp273``. (Contributed by + Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) + * :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary buffer objects as arguments. (Contributed by Antoine Pitrou in :issue:`15958`.) diff --git a/Lib/encodings/aliases.py b/Lib/encodings/aliases.py --- a/Lib/encodings/aliases.py +++ b/Lib/encodings/aliases.py @@ -109,6 +109,11 @@ '1258' : 'cp1258', 'windows_1258' : 'cp1258', + # cp273 codec + '273' : 'cp273', + 'ibm273' : 'cp273', + 'csibm273' : 'cp273', + # cp424 codec '424' : 'cp424', 'csibm424' : 'cp424', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 03:38:25 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 03:38:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_sunau_supports?= =?utf-8?q?_24_bit_samples=2E?= Message-ID: <3fgqcx3mZJz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/12fb14bcde41 changeset: 89504:12fb14bcde41 user: R David Murray date: Fri Mar 07 21:15:56 2014 -0500 summary: whatsnew: sunau supports 24 bit samples. files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1265,6 +1265,10 @@ called automatically at the end of the block. (Contributed by Serhiy Storchaka in :issue:`18878`.) +:meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding +support for writing 24 sample using the module. (Contributed by +Serhiy Storchaka in :issue:`19261`.) + sys --- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 03:38:26 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 03:38:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_inet=5Fpton/in?= =?utf-8?q?et=5Fntop_support_windows_=28=237171=29=2E?= Message-ID: <3fgqcy5cYjz7LlR@mail.python.org> http://hg.python.org/cpython/rev/f82145a516f0 changeset: 89505:f82145a516f0 user: R David Murray date: Fri Mar 07 21:22:39 2014 -0500 summary: whatsnew: inet_pton/inet_ntop support windows (#7171). Added versionchanged to docs. files: Doc/library/socket.rst | 6 ++++++ Doc/whatsnew/3.4.rst | 3 +++ 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -633,6 +633,9 @@ Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. function:: inet_ntop(address_family, packed_ip) @@ -649,6 +652,9 @@ Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1178,6 +1178,9 @@ The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. +:func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported +on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) + sqlite3 ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 03:38:28 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 03:38:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_audioop_accept?= =?utf-8?q?s_any_bytes-like=2C_rejects_strings=2E?= Message-ID: <3fgqd00Gcfz7Lm6@mail.python.org> http://hg.python.org/cpython/rev/2776aa0198a9 changeset: 89506:2776aa0198a9 user: R David Murray date: Fri Mar 07 21:35:31 2014 -0500 summary: whatsnew: audioop accepts any bytes-like, rejects strings. files: Doc/library/audioop.rst | 6 ++---- Doc/whatsnew/3.4.rst | 9 ++++++++- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -12,10 +12,8 @@ .. versionchanged:: 3.4 Support for 24-bit samples was added. - -.. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted by all functions in this - module. Strings no more supported. + All functions now accept any :term:`bytes-like object`. + String input now results in an immediate error. .. index:: single: Intel/DVI ADPCM diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -554,12 +554,16 @@ audioop ------- -The module now supports 24-bit samples. (Contributed by Serhiy Storchaka +:mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka in :issue:`12866`.) Added the :func:`~audioop.byteswap` function to convert big-endian samples to little-endian and vice versa (:issue:`19641`). +All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings +are not accepted: they didn't work before, now they raise an error right away. +(Contributed by Serhiy Storchaka in :issue:`16685`.) + base64 ------ @@ -2050,6 +2054,9 @@ matched the input type, so if your code was depending on the return value being, say, a ``bytearray``, you will need to change your code. +* :mod:`audioop` functions now raise an error immediately if passed string + input, instead of failing randomly later on (:issue:`16685`). + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 07:36:51 2014 From: python-checkins at python.org (nick.coghlan) Date: Sat, 8 Mar 2014 07:36:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Improve_descriptions_of_in?= =?utf-8?q?trospection_changes?= Message-ID: <3fgww34ySrz7Lnj@mail.python.org> http://hg.python.org/cpython/rev/0909f73cfa3a changeset: 89507:0909f73cfa3a user: Nick Coghlan date: Sat Mar 08 16:36:37 2014 +1000 summary: Improve descriptions of introspection changes Several of the introspection changes in Python 3.4 are indirect, where inspect module changes affected pydoc, and those in turn affected the help builtin. This update adds versionchanged notes in the key locations, as well as more coverage in the What's New document (in particular, a note in the porting section regarding the expanded domain for inspect.getfullargspec). files: Doc/library/functions.rst | 4 ++ Doc/library/inspect.rst | 5 ++ Doc/library/pydoc.rst | 5 ++ Doc/whatsnew/3.4.rst | 53 ++++++++++++++++++++++---- 4 files changed, 59 insertions(+), 8 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -610,6 +610,10 @@ This function is added to the built-in namespace by the :mod:`site` module. + .. versionchanged:: 3.4 + Changes to :mod:`pydoc` and :mod:`inspect` mean that the reported + signatures for callables are now more comprehensive and consistent. + .. function:: hex(x) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -729,6 +729,11 @@ Consider using the new :ref:`Signature Object ` interface, which provides a better way of introspecting functions. + .. versionchanged:: 3.4 + This function is now based on :func:`signature`, but still ignores + ``__wrapped__`` attributes and includes the already bound first + parameter in the signature output for bound methods. + .. function:: getargvalues(frame) diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -84,3 +84,8 @@ .. versionchanged:: 3.2 Added the ``-b`` option, deprecated the ``-g`` option. + +.. versionchanged:: 3.4 + :mod:`pydoc` now uses :func:`inspect.signature` rather than + :func:`inspect.getfullargspec` to extract signature information from + callables. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -133,6 +133,8 @@ a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME handling ` (:issue:`18891`). +* The :mod:`inspect` and :mod:`pydoc` modules are now capable of + correct introspection of a much wider variety of callable objects * The :mod:`ipaddress` module API has been declared stable @@ -384,6 +386,10 @@ and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan and Claudiu Popa in :issue:`18690` and :issue:`19078`.) +* Signatures reported by :func:`help` have been modified and improved in + several cases as a result of the introduction of Argument Clinic and other + changes to the :mod:`inspect` and :mod:`pydoc` modules. + New Modules =========== @@ -871,7 +877,7 @@ inspect ------- -The inspect module now offers a basic :ref:`command line interface +The :mod:`inspect` module now offers a basic :ref:`command line interface ` to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in :issue:`18626`) @@ -889,10 +895,14 @@ :func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` now use the :func:`~inspect.signature` API. This allows them to -support much broader range of functions, including some builtins and -callables that follow ``__signature__`` protocol. It is still -recommended to update your code to use :func:`~inspect.signature` -directly. (Contributed by Yury Selivanov in :issue:`17481`) +support a much broader range of callables, including those with +``__signature__`` attributes, those with metadata provided by argument +clinic, :func:`functools.partial` objects and more. Note that, unlike +:func:`~inspect.signature`, these functions still ignore ``__wrapped__`` +attributes, and report the already bound first argument for bound methods, +so it is still necessary to update your code to use +:func:`~inspect.signature` directly if those features are desired. +(Contributed by Yury Selivanov in :issue:`17481`) :func:`~inspect.signature` now supports duck types of CPython functions, which adds support for functions compiled with Cython. (Contributed @@ -1086,11 +1096,25 @@ pydoc ----- -While significant changes have not been made to :mod:`pydoc` directly, +The :mod:`pydoc` module is now based directly on the +:func:`inspect.signature` introspection API, allowing it to provide +signature information for a wider variety of callable objects. This change +also means that ``__wrapped__`` attributes are now taken into account when +display help information (Contributed by Larry Hastings in :issue:`19674`) + +The :mod:`pydoc` module no longer displays the ``self`` parameter for +already bound methods. Instead, it aims to always display the exact current +signature of the supplied callable (Contributed by Larry Hastings in +:issue:`20710`) + +In addition to the changes that have been made to :mod:`pydoc` directly, its handling of custom ``__dir__`` methods and various descriptor -behaviours has been improved substantially by the underlying changes in +behaviours has also been improved substantially by the underlying changes in the :mod:`inspect` module. +As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also +affect the behaviour of :func:`help`. + re -- @@ -1532,7 +1556,12 @@ implemented in C. Some standard library extension modules have been converted to use Argument -Clinic in Python 3.4, and :mod:`inspect` has been updated accordingly. +Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` has been updated +accordingly. + +It is expected that signature metadata for programmatic introspection will +be added to additional callables implemented in C as part of Python 3.4 +maintenance releases. .. note:: The Argument Clinic PEP is not fully up to date with the state of the @@ -1956,6 +1985,14 @@ :func:`inspect.unwrap` to access the first function in the chain that has no ``__wrapped__`` attribute. +* :func:`inspect.getfullargspec` has been reimplemented on top of + :func`inspect.signature` and hence handles a much wider variety of callable + objects than it did in the past. It is expected that additional builtin and + extension module callables will gain signature metadata over the course of + the Python 3.4 series. Code that assumes that + :func:`inspect.getfullargspec` will fail on non-Python callables may need + to be adjusted accordingly. + * :class:`importlib.machinery.PathFinder` now passes on the current working directory to objects in :data:`sys.path_hooks` for the empty string. This results in :data:`sys.path_importer_cache` never containing ``''``, thus -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Mar 8 09:34:54 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 08 Mar 2014 09:34:54 +0100 Subject: [Python-checkins] Daily reference leaks (d843a1caba78): sum=0 Message-ID: results for d843a1caba78 on branch "default" -------------------------------------------- test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogUJ5Tpu', '-x'] From python-checkins at python.org Sat Mar 8 18:54:19 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 18:54:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Tweak_NEWS_so_some_additio?= =?utf-8?q?nal_issue_numbers_become_links_in_html_version=2E?= Message-ID: <3fhCxl1hN5z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/58a47fc5cdff changeset: 89508:58a47fc5cdff user: R David Murray date: Sat Mar 08 10:06:04 2014 -0500 summary: Tweak NEWS so some additional issue numbers become links in html version. files: Misc/NEWS | 14 ++++++++------ 1 files changed, 8 insertions(+), 6 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1262,8 +1262,9 @@ - Issue #19523: Closed FileHandler leak which occurred when delay was set. -- Issue #19544 and #6516: Restore support for --user and --group parameters to - sdist command accidentally rolled back as part of the distutils2 rollback. +- Issue #19544 and Issue #6516: Restore support for --user and --group + parameters to sdist command accidentally rolled back as part of the + distutils2 rollback. - Issue #13674: Prevented time.strftime from crashing on Windows when given a year before 1900 and a format of %y. @@ -1609,8 +1610,9 @@ - Issue #19254: Provide an optimized Python implementation of pbkdf2_hmac. -- Issues #19201, #19222, #19223: Add "x" mode (exclusive creation) in opening - file to bz2, gzip and lzma modules. Patches by Tim Heaney and Vajrasky Kok. +- Issues #19201, Issue #19222, Issue #19223: Add "x" mode (exclusive creation) + in opening file to bz2, gzip and lzma modules. Patches by Tim Heaney and + Vajrasky Kok. - Fix a reference count leak in _sre. @@ -6328,7 +6330,7 @@ deallocator calls one of the methods on the type (e.g. when subclassing IOBase). Diagnosis and patch by Davide Rizzo. -- Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on Windows. +- Issue #9611, Issue #9015: FileIO.read() clamps the length to INT_MAX on Windows. - Issue #9642: Uniformize the tests on the availability of the mbcs codec, add a new HAVE_MBCS define. @@ -6796,7 +6798,7 @@ 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', and private attributes of 'smtpd.SMTPChannel'. -- Issue #5905, #13560: time.strftime() is now using the current locale +- Issue #5905, Issue #13560: time.strftime() is now using the current locale encoding, instead of UTF-8, if the wcsftime() function is not available. - Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 18:54:20 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 18:54:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_Wave=5Fwrite_h?= =?utf-8?q?andles_unseekable_files=2E_=28=235202=29?= Message-ID: <3fhCxm3PMvz7LnK@mail.python.org> http://hg.python.org/cpython/rev/b861c7717c79 changeset: 89509:b861c7717c79 user: R David Murray date: Sat Mar 08 11:14:29 2014 -0500 summary: whatsnew: Wave_write handles unseekable files. (#5202) Also clarify the documentation of this feature. files: Doc/library/wave.rst | 31 +++++++++++++++++++++++++------ Doc/whatsnew/3.4.rst | 4 ++++ 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -150,14 +150,30 @@ Wave_write Objects ------------------ +For seekable output streams, the ``wave`` header will automatically be updated +to reflect the number of frames actually written. For unseekable streams, the +*nframes* value must be accurate when the first frame data is written. An +accurate *nframes* value can be achieved either by calling +:meth:`~Wave_write.setnframes` or :meth:`~Wave_write.setparams` with the number +of frames that will be written before :meth:`~Wave_write.close` is called and +then using :meth:`~Wave_write.writeframesraw` to write the frame data, or by +calling :meth:`~Wave_write.writeframes` with all of the frame data to be +written. In the latter case :meth:`~Wave_write.writeframes` will calculate +the number of frames in the data and set *nframes* accordingly before writing +the frame data. + Wave_write objects, as returned by :func:`.open`, have the following methods: +.. versionchanged:: 3.4 + Added support for unseekable files. + .. method:: Wave_write.close() Make sure *nframes* is correct, and close the file if it was opened by - :mod:`wave`. This method is called upon object collection. Can raise an - exception if *nframes* is not correct and a file is not seekable. + :mod:`wave`. This method is called upon object collection. It will raise + an exception if the output stream is not seekable and *nframes* does not + match the number of frames actually written. .. method:: Wave_write.setnchannels(n) @@ -181,8 +197,9 @@ .. method:: Wave_write.setnframes(n) - Set the number of frames to *n*. This will be changed later if more frames are - written. + Set the number of frames to *n*. This will be changed later if the number + of frames actually written is different (this update attempt will + raise an error if the output stream is not seekable). .. method:: Wave_write.setcomptype(type, name) @@ -214,8 +231,10 @@ .. method:: Wave_write.writeframes(data) - Write audio frames and make sure *nframes* is correct. Can raise an - exception if a file is not seekable. + Write audio frames and make sure *nframes* is correct. It will raise an + error if the output stream is not seekable and the total number of frames + that have been written after *data* has been written does not match the + previously set value for *nframes*. .. versionchanged:: 3.4 Any :term:`bytes-like object`\ s are now accepted. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1439,6 +1439,10 @@ :meth:`wave.open` now supports the context manager protocol. (Contributed by Claudiu Popa in :issue:`17616`.) +:mod:`wave` can now :ref:`write output to unssekable files +`. (Contributed by David Jones, Guilherme Polo, and Serhiy +Storchaka in :issue:`5202`.) + weakref ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 18:54:21 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 18:54:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_sunau/aifc/wav?= =?utf-8?q?e_writeframes=5Braw=5D_accept_any_bytes-like_=28=238311=29?= Message-ID: <3fhCxn4l5Lz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/b72615222c98 changeset: 89510:b72615222c98 user: R David Murray date: Sat Mar 08 11:46:05 2014 -0500 summary: whatsnew: sunau/aifc/wave writeframes[raw] accept any bytes-like (#8311) files: Doc/library/aifc.rst | 4 ++-- Doc/library/sunau.rst | 4 ++-- Doc/library/wave.rst | 4 ++-- Doc/whatsnew/3.4.rst | 14 +++++++++++++- 4 files changed, 19 insertions(+), 7 deletions(-) diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -226,7 +226,7 @@ file parameters have been set. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.writeframesraw(data) @@ -235,7 +235,7 @@ updated. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.close() diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -251,7 +251,7 @@ Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.writeframes(data) @@ -259,7 +259,7 @@ Write audio frames and make sure *nframes* is correct. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.close() diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -226,7 +226,7 @@ Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: Wave_write.writeframes(data) @@ -237,7 +237,7 @@ previously set value for *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. Note that it is invalid to set any parameters after calling :meth:`writeframes` diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -540,7 +540,7 @@ aifc ---- -The :meth:`~aifc.getparams` method now returns a namedtuple rather than a +The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) :func:`aifc.open` now supports the context manager protocol: when used in a @@ -548,6 +548,10 @@ object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in :issue:`16486`.) +The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` +methods now accept any :term:`bytes-like object`. (Contributed by Serhiy +Storchaka in :issue:`8311`.) + argparse -------- @@ -1300,6 +1304,10 @@ support for writing 24 sample using the module. (Contributed by Serhiy Storchaka in :issue:`19261`.) +The :meth:`~sunau.AU_write.writeframesraw` and +:meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) + sys --- @@ -1443,6 +1451,10 @@ `. (Contributed by David Jones, Guilherme Polo, and Serhiy Storchaka in :issue:`5202`.) +The :meth:`~wave.Wave_write.writeframesraw` and +:meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) + weakref ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 18:54:22 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 18:54:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_dbm=2Eopen_is_?= =?utf-8?q?context_manager=2E_=28=2319282=29?= Message-ID: <3fhCxp6lS9z7LpF@mail.python.org> http://hg.python.org/cpython/rev/200207e50cbf changeset: 89511:200207e50cbf user: R David Murray date: Sat Mar 08 12:00:52 2014 -0500 summary: whatsnew: dbm.open is context manager. (#19282) files: Doc/whatsnew/3.4.rst | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -617,6 +617,15 @@ differences between single use, reusable and reentrant context managers. +dbm +--- + +:func:`dbm.open` objects now support the context management protocol. When +used in a :keyword:`with` statement, the ``close`` method of the database +object will be called automatically at the end of the block. (Contributed by +Claudiu Popa and Nick Coghlan in :issue:`19282`.) + + dis --- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 18:54:24 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 8 Mar 2014 18:54:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_base65_encodin?= =?utf-8?b?Z3MuICgjMTc2MTgp?= Message-ID: <3fhCxr29Zlz7LpR@mail.python.org> http://hg.python.org/cpython/rev/1853679c6f71 changeset: 89512:1853679c6f71 user: R David Murray date: Sat Mar 08 12:53:28 2014 -0500 summary: whatsnew: base65 encodings. (#17618) Also updated the base64 module title and introduction to adjust for the fact that these new encodings are included. Also adjusted the wording about the base64 alphabets (see issue #20837). files: Doc/library/base64.rst | 32 +++++++++++++++++------------ Doc/whatsnew/3.4.rst | 8 +++++++ 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -1,27 +1,33 @@ -:mod:`base64` --- RFC 3548: Base16, Base32, Base64 Data Encodings -================================================================= +:mod:`base64` --- Base16, Base32, Base64, Base85 Data Encodings +=============================================================== .. module:: base64 - :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings + :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings; + Base85 and Ascii85 .. index:: pair: base64; encoding single: MIME; base64 encoding -This module provides data encoding and decoding as specified in :rfc:`3548`. -This standard defines the Base16, Base32, and Base64 algorithms for encoding -and decoding arbitrary binary strings into ASCII-only byte strings that can be +This module provides functions for encoding binary data to printable +ASCII characters and decoding such encodings back to binary data. +It provides encoding and decoding functions for the encodings specified in +in :rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, +and for the de-facto standard Ascii85 and Base85 encodings. + +The :rfc:`3548` encodings are suitable for encoding binary data so that it can safely sent by email, used as parts of URLs, or included as part of an HTTP POST request. The encoding algorithm is not the same as the :program:`uuencode` program. -There are two interfaces provided by this module. The modern interface -supports encoding and decoding ASCII byte string objects using all three -alphabets. Additionally, the decoding functions of the modern interface also -accept Unicode strings containing only ASCII characters. The legacy interface -provides for encoding and decoding to and from file-like objects as well as -byte strings, but only using the Base64 standard alphabet. +There are two :rfc:`3548` interfaces provided by this module. The modern +interface supports encoding and decoding ASCII byte string objects using all +three :rfc:`3548` defined alphabets (normal, URL-safe, and filesystem-safe). +Additionally, the decoding functions of the modern interface also accept +Unicode strings containing only ASCII characters. The legacy interface provides +for encoding and decoding to and from file-like objects as well as byte +strings, but only using the Base64 standard alphabet. .. versionchanged:: 3.3 ASCII-only Unicode strings are now accepted by the decoding functions of @@ -29,7 +35,7 @@ .. versionchanged:: 3.4 Any :term:`bytes-like object`\ s are now accepted by all - encoding and decoding functions in this module. + encoding and decoding functions in this module. Ascii85/Base85 support added. The modern interface provides: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -583,6 +583,14 @@ :class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in :issue:`17839`.) +New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, +:func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to +encode and decode binary data from and to ``Ascii85`` and the git/mercurial +``Base85`` formats, respectively. The ``a85`` functions have options that can +be sued to make them compatible with the variants of the ``Ascii85`` encoding, +including the Adobe variant. (Contributed by Martin Morrison, the Mercurial +project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) + colorsys -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 8 18:56:15 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 8 Mar 2014 18:56:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Improve_the_de?= =?utf-8?q?fault_seeding_in_random_module_to_use_32_bytes_of_entropy_when?= Message-ID: <3fhCzz360kz7LjT@mail.python.org> http://hg.python.org/cpython/rev/1ba8deb4536d changeset: 89513:1ba8deb4536d branch: 2.7 parent: 89484:52256a5861fa user: Raymond Hettinger date: Sat Mar 08 09:56:08 2014 -0800 summary: Improve the default seeding in random module to use 32 bytes of entropy when available. files: Lib/random.py | 2 +- Misc/NEWS | 4 ++++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/random.py b/Lib/random.py --- a/Lib/random.py +++ b/Lib/random.py @@ -108,7 +108,7 @@ if a is None: try: - a = long(_hexlify(_urandom(16)), 16) + a = long(_hexlify(_urandom(32)), 16) except NotImplementedError: import time a = long(time.time() * 256) # use fractional seconds diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ as documented. The pattern and source keyword parameters are left as deprecated aliases. +- Improve the random module's default seeding to use 256 bits of entropy + from os.urandom(). This was already done for Python 3, mildly improving + security with a bigger seed space. + - Issue #15618: Make turtle.py compatible with 'from __future__ import unicode_literals'. Initial patch by Juancarlo A?ez. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 03:41:47 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 03:41:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_glob=2Eescape?= =?utf-8?q?=2E?= Message-ID: <3fhRfM5Nwjz7Lqr@mail.python.org> http://hg.python.org/cpython/rev/e617f0ac696d changeset: 89514:e617f0ac696d parent: 89512:1853679c6f71 user: R David Murray date: Sat Mar 08 21:23:21 2014 -0500 summary: whatsnew: glob.escape. files: Doc/whatsnew/3.4.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -793,6 +793,14 @@ (Contributed by Antoine Pitrou in :issue:`16351`.) +glob +---- + +A new function :func:`~glob.escape` provides a way to escape special characters +in a filename so that they do not become part of the globbing expansion but are +instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) + + hashlib ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 03:41:48 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 03:41:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_tidy_up_html_n?= =?utf-8?q?otes=2C_add_porting_note_for_expand=5Fcharrefs=2E?= Message-ID: <3fhRfN6j80z7Llx@mail.python.org> http://hg.python.org/cpython/rev/fd53f083768c changeset: 89515:fd53f083768c user: R David Murray date: Sat Mar 08 21:40:29 2014 -0500 summary: whatsnew: tidy up html notes, add porting note for expand_charrefs. files: Doc/whatsnew/3.4.rst | 26 ++++++++++++++++---------- 1 files changed, 16 insertions(+), 10 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -830,16 +830,16 @@ html ---- -Added a new :func:`html.unescape` function that converts HTML5 character -references to the corresponding Unicode characters. -(Contributed by Ezio Melotti in :issue:`2927`) - -Added a new *convert_charrefs* keyword argument to -:class:`~html.parser.HTMLParser` that, when ``True``, automatically converts -all character references. For backward-compatibility, its value defaults -to ``False``, but it will change to ``True`` in future versions, so you -are invited to set it explicitly and update your code to use this new feature. -(Contributed by Ezio Melotti in :issue:`13633`) +New function :func:`~html.unescape` function converts HTML5 character references to +the corresponding Unicode characters. (Contributed by Ezio Melotti in +:issue:`2927`) + +:class:`~html.parser.HTMLParser` accepts a new keyword argument +*convert_charrefs* that, when ``True``, automatically converts all character +references. For backward-compatibility, its value defaults to ``False``, but +it will change to ``True`` in a future version of Python, so you are invited to +set it explicitly and update your code to use this new feature. (Contributed +by Ezio Melotti in :issue:`13633`) The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. (Contributed by Ezio Melotti in :issue:`15114`) @@ -2135,6 +2135,12 @@ * :mod:`audioop` functions now raise an error immediately if passed string input, instead of failing randomly later on (:issue:`16685`). +* The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` + currently defaults to ``False`` for backward compatibility, but will + eventually be changed to default to ``True``. It is recommended that you add + this keyword, with the appropriate value, to any + :class:`~html.parser.HTMLParser` calls in your code. + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Mar 9 09:43:31 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 09 Mar 2014 09:43:31 +0100 Subject: [Python-checkins] Daily reference leaks (1853679c6f71): sum=-4 Message-ID: results for 1853679c6f71 on branch "default" -------------------------------------------- test_site leaked [-2, 0, 0] references, sum=-2 test_site leaked [-2, 0, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogpikoPI', '-x'] From python-checkins at python.org Sun Mar 9 09:47:16 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 09:47:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_=2Emd5_files_are_not_neede?= =?utf-8?q?d=2E?= Message-ID: <3fhbm436ZBz7Lqv@mail.python.org> http://hg.python.org/release/rev/adec99d760a3 changeset: 82:adec99d760a3 user: Georg Brandl date: Sun Mar 09 09:47:08 2014 +0100 summary: .md5 files are not needed. files: release.py | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/release.py b/release.py --- a/release.py +++ b/release.py @@ -232,10 +232,6 @@ checksum_tgz.hexdigest(), int(os.path.getsize(tgz)), tgz)) print(' %s %8s %s' % ( checksum_xz.hexdigest(), int(os.path.getsize(xz)), xz)) - with open(tgz + '.md5', 'w', encoding="ascii") as fp: - fp.write(checksum_tgz.hexdigest()) - with open(xz + '.md5', 'w', encoding="ascii") as fp: - fp.write(checksum_xz.hexdigest()) print('Signing tarballs') print('List of available private keys:') -- Repository URL: http://hg.python.org/release From python-checkins at python.org Sun Mar 9 10:01:42 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:01:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_101_a_bit=2E_Remov?= =?utf-8?q?e_outdated_Windows_instructions=2E?= Message-ID: <3fhc4k5Ryjz7Ljk@mail.python.org> http://hg.python.org/peps/rev/95aaea3e65b0 changeset: 5404:95aaea3e65b0 user: Georg Brandl date: Sun Mar 09 10:01:33 2014 +0100 summary: Update PEP 101 a bit. Remove outdated Windows instructions. files: pep-0101.txt | 95 ++++++++++++++------------------------- 1 files changed, 35 insertions(+), 60 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -59,7 +59,7 @@ done by the Release Manager (RM), the designated person performing the release. The roles and their current experts are: - * RM = Release Manager: Georg Brandl (Central Europe) + * RM = Release Manager: Larry Hastings (US) * WE = Windows: Martin von Loewis (Central Europe) * ME = Mac: Ned Deily (US) * DE = Docs: Georg Brandl (Central Europe) @@ -70,13 +70,10 @@ in different timezones, the RM must ensure that the release tag is created in enough time for the Experts to cut binary releases. - IT IS HIGHLY RECOMMENDED THAT YOU AT LEAST TAG THE TREE 24 HOURS - BEFORE A FINAL RELEASE. This will give the Experts enough time to - do their bits before the announcement goes out. - - In any case, the RM MUST wait for the "green light" from the - following experts before updating the web pages and sending the - announcement: WE, DE + You should not make the release public (by updating the website and + sending announcements) before all experts have updated their bits. + In rare cases where the expert for Windows or Mac is MIA, you may add + a message "(Platform) binaries will be provided shortly" and proceed. XXX: We should include a dependency graph to illustrate the steps that can be taken in parallel, or those that depend on other steps. @@ -139,14 +136,14 @@ Create a local clone of the cpython repository (called the "release clone" from now on). - + Also clone the repo at http://hg.python.org/cpython using the server-side clone feature. The name of the new clone should preferably have a "releasing/" prefix. The other experts will use the release clone for making the binaries, so it is important that they have access to it! - Optionally, set up your local release clone to push to the remote + It's best to set up your local release clone to push to the remote release clone by default (by editing .hg/hgrc to that effect). ___ Notify all committers by sending email to python-committers at python.org. @@ -208,10 +205,6 @@ ___ PC/python_nt.rc sets up the DLL version resource for Windows (displayed when you right-click on the DLL and select Properties). - ___ The license.ht file for the distribution on the website - contains what purports to be an HTML-ized copy of the LICENSE - file from the distribution. You'll need to bump the version number to - the one you're releasing. BROKEN ___ Check with the IE (if there is one ) to be sure that Lib/idlelib/NEWS.txt has been similarly updated. @@ -266,7 +259,7 @@ $ hg mv -f PC/python32gen.py PC/python33gen.py ___ Commit these changes to the default branch. - + ___ Now, go back to the previously noted revision and make the maintenance branch *from there*. @@ -285,43 +278,23 @@ order to create the release. There are things you can do while you wait though, so keep reading until you hit the next STOP. - ___ XXX The WE builds the Windows helpfile, using (in Doc/) either + ___ The WE builds the Windows helpfile, using (in Doc/) - $ make htmlhelp (on Unix) - - or - - > make.bat htmlhelp (on Windows) + > make.bat htmlhelp (on Windows) to create suitable input for HTML Help Workshop in build/htmlhelp. HTML - Help Workshop is then fired up on the created python26.hhp file, finally - resulting in an python26.chm file. He then copies the file into the Doc - directories of the build trees (once for each target architecture). + Help Workshop is then fired up on the created python33.hhp file, finally + resulting in an python33.chm file. - XXX The CHM file should also be scp'd to the docs download location. + ___ The WE then generates Windows installer files for each Windows + target architecture (for Python 3.3, this means x86 and AMD64). - ___ XXX The WE then generates Windows installer files for each Windows - target architecture (for Python 2.6, this means x86 and AMD64). He has - one checkout tree per target architecture, and builds the pcbuild.sln - project for the appropriate architecture. He then edits - Tools/msi/config.py to update full_current_version, sets snapshot to - False and runs msi.py with ActivePython 2.5 or Python 2.5 with pywin32. - For that to work, the following prerequisites must be met: + The WE checksums the files (*.msi, *.chm, *-pdb.zip), uploads them to + dinsdale together with gpg signature files, and emails you the location + and md5sums. - - PC\icons.mak must have been run with nmake. - - - The cmd.exe window in which this is run must have Cygwin/bin in its - path (atleast for x86). - - - The cmd.exe window must have MS compiler tools for the target - architecture in its path (VS 2003 for x86, the platform SDK for - AMD64). - - - The cmd.exe window must also have cabarc.exe from the CAB SDK in its - path. - - The WE checksums the files (*.msi and *.chm), uploads them to some place - in the net, and emails you the location and md5sums. + ___ The ME builds Mac installer packages and uploads them to dinsdale together + with gpg signature files. ___ Time to build the source tarball. Be sure to update your clone to the correct branch. E.g. @@ -333,13 +306,18 @@ You should not see any files. I.e. you better not have any uncommitted changes in your working directory. - ___ Use the release script to create the source gzip and bz2 tarballs, md5 - checksums, documentation tar and zip files, and gpg signature files. + ___ Make sure you have an up-to-date Sphinx toolchain installed. + + $ pip install -U Sphinx + + ___ Use the release script to create the source gzip and xz tarballs, + documentation tar and zip files, and gpg signature files. $ .../release/release.py --export X.Y.ZaN - This will leave all the relevant files in a subdirectory called - 'X.Y.ZaN/src', and the built docs in 'X.Y.ZaN/docs' (for final releases). + This can take a while for final releases, and it will leave all the + tarballs and signatures in a subdirectory called 'X.Y.ZaN/src', and the + built docs in 'X.Y.ZaN/docs' (for final releases). ___ scp or rsync all the files to your home directory on dinsdale.python.org. @@ -371,7 +349,7 @@ If you're feeling lucky and have some time to kill, or if you are making a release candidate or final release, run the full test suite: - $ make TESTOPTS='-u all' test + $ make testall If the tests pass, then you can feel good that the tarball is fine. If some of the tests fail, or anything else about the @@ -380,7 +358,7 @@ ___ Now you need to go to dinsdale.python.org and move all the files in place over there. Our policy is that every Python version gets its own - directory, but each directory may contain several releases. + directory, but each directory contains all releases of that version. ___ On dinsdale, cd /data/ftp.python.org/pub/python/X.Y.Z creating it if necessary. Make sure it is owned by group 'webmaster' @@ -409,16 +387,15 @@ ___ Let the DE check if the docs are built and work all right. - ___ If this is a major release: Tell the DE to adapt redirects for + ___ If this is a final major release: Tell the DE to adapt redirects for docs.python.org/X.Y in the Apache config for docs.python.org, update the script Doc/tools/dailybuild.py to point to the right stable/development branches, and to install it and make the initial checkout. The Doc's version_switcher.js script also needs to be updated. - ___ For the extra paranoid, do a completely clean test of the - release. This includes downloading the tarball from - www.python.org. + ___ For the extra paranoid, do a completely clean test of the release. + This includes downloading the tarball from www.python.org. Make sure the md5 checksums match. Then unpack the tarball, and do a clean make test. @@ -436,9 +413,7 @@ don't have that, ask someone on pydotorg at python.org for the proper permissions. It's insane for you not to have it. - I'm not going to go into the details of building the site or pushing it - live. All the directories below are named relative to the data subdirectory - unless otherwise noted. + XXX This is completely out of date for Django based python.org. This page will probably come in handy: @@ -550,7 +525,7 @@ ___ You can delete the remote release clone, or simply reuse it for the next release. - + ___ Send email to python-committers informing them that the release has been published. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 9 10:22:58 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:22:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogaGd0b3VjaDogcmVt?= =?utf-8?q?ove_meta-variable_from_option_spec?= Message-ID: <3fhcYG6qCQz7LjS@mail.python.org> http://hg.python.org/cpython/rev/6e9827ba455f changeset: 89516:6e9827ba455f branch: 3.3 parent: 89500:34df43c9c74a user: Georg Brandl date: Sun Mar 09 10:22:10 2014 +0100 summary: hgtouch: remove meta-variable from option spec This fifth tuple element is unsupported in an old hg version still used on one buildbot, and it is not really necessary (only used in touch --help). files: Tools/hg/hgtouch.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/hg/hgtouch.py b/Tools/hg/hgtouch.py --- a/Tools/hg/hgtouch.py +++ b/Tools/hg/hgtouch.py @@ -125,6 +125,6 @@ cmdtable = { "touch": (touch, - [('b', 'basedir', '', 'base dir of the tree to apply touching', 'BASEDIR')], + [('b', 'basedir', '', 'base dir of the tree to apply touching')], "hg touch [-b BASEDIR]") } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 10:23:00 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:23:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3fhcYJ1P4Pz7Lm3@mail.python.org> http://hg.python.org/cpython/rev/c88553c48d76 changeset: 89517:c88553c48d76 parent: 89515:fd53f083768c parent: 89516:6e9827ba455f user: Georg Brandl date: Sun Mar 09 10:22:46 2014 +0100 summary: merge with 3.3 files: Tools/hg/hgtouch.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/hg/hgtouch.py b/Tools/hg/hgtouch.py --- a/Tools/hg/hgtouch.py +++ b/Tools/hg/hgtouch.py @@ -125,6 +125,6 @@ cmdtable = { "touch": (touch, - [('b', 'basedir', '', 'base dir of the tree to apply touching', 'BASEDIR')], + [('b', 'basedir', '', 'base dir of the tree to apply touching')], "hg touch [-b BASEDIR]") } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 10:23:56 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:23:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMDcz?= =?utf-8?q?5=3A_remove_erroneous_deprecated_marker_from_stringprep_docs?= Message-ID: <3fhcZN2FRKz7LjS@mail.python.org> http://hg.python.org/cpython/rev/fadde95c134e changeset: 89518:fadde95c134e branch: 2.7 parent: 89352:0f0e9b7d4f1d user: Georg Brandl date: Mon Feb 24 09:26:53 2014 +0100 summary: Closes #20735: remove erroneous deprecated marker from stringprep docs files: Doc/library/stringprep.rst | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Doc/library/stringprep.rst b/Doc/library/stringprep.rst --- a/Doc/library/stringprep.rst +++ b/Doc/library/stringprep.rst @@ -4,7 +4,6 @@ .. module:: stringprep :synopsis: String preparation, as per RFC 3453 - :deprecated: .. moduleauthor:: Martin v. L?wis .. sectionauthor:: Martin v. L?wis -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 10:23:57 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:23:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge?= Message-ID: <3fhcZP3YYnz7LjS@mail.python.org> http://hg.python.org/cpython/rev/de632ccaffd6 changeset: 89519:de632ccaffd6 branch: 2.7 parent: 89513:1ba8deb4536d parent: 89518:fadde95c134e user: Georg Brandl date: Sun Mar 09 10:23:24 2014 +0100 summary: merge files: Doc/library/stringprep.rst | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Doc/library/stringprep.rst b/Doc/library/stringprep.rst --- a/Doc/library/stringprep.rst +++ b/Doc/library/stringprep.rst @@ -4,7 +4,6 @@ .. module:: stringprep :synopsis: String preparation, as per RFC 3453 - :deprecated: .. moduleauthor:: Martin v. L?wis .. sectionauthor:: Martin v. L?wis -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 10:32:07 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 10:32:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Fix_markup=2E?= Message-ID: <3fhclq5Cm0z7LrC@mail.python.org> http://hg.python.org/devguide/rev/57e7eb25a230 changeset: 671:57e7eb25a230 user: Georg Brandl date: Sun Mar 09 10:32:01 2014 +0100 summary: Fix markup. files: gdb.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/gdb.rst b/gdb.rst --- a/gdb.rst +++ b/gdb.rst @@ -277,7 +277,7 @@ #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=, _stdout_buffer=, _stderr_buffer=, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={: , : , : , : , http://hg.python.org/cpython/rev/1f2c16c7ecc2 changeset: 89520:1f2c16c7ecc2 branch: 3.3 parent: 89458:ca5635efe090 user: Georg Brandl date: Sun Mar 02 09:29:00 2014 +0100 summary: Added tag v3.3.5rc2 for changeset ca5635efe090 files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -121,3 +121,4 @@ fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 +ca5635efe090f78806188ac2758f9948596aa8b2 v3.3.5rc2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:18:28 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:18:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQnVtcCB0byAzLjMu?= =?utf-8?q?5_final=2E?= Message-ID: <3fhdnJ3qKwz7LlV@mail.python.org> http://hg.python.org/cpython/rev/62cf4e77f785 changeset: 89521:62cf4e77f785 branch: 3.3 tag: v3.3.5 user: Georg Brandl date: Sun Mar 09 09:37:14 2014 +0100 summary: Bump to 3.3.5 final. files: Include/patchlevel.h | 6 +++--- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 8 ++++++++ Misc/RPM/python-3.3.spec | 2 +- README | 4 ++-- 6 files changed, 16 insertions(+), 8 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 3 #define PY_MICRO_VERSION 5 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 2 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.3.5rc2" +#define PY_VERSION "3.3.5" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.5rc2" +__version__ = "3.3.5" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.5rc2" +IDLE_VERSION = "3.3.5" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,14 @@ Python News +++++++++++ +What's New in Python 3.3.5? +=========================== + +*Release date: 09-Mar-2014* + +No changes from release candidate 2. + + What's New in Python 3.3.5 release candidate 2? =============================================== diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.5rc2 +%define version 3.3.5 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.3.5 release candidate 2 -================================================ +This is Python version 3.3.5 +============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:18:29 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:18:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Added_tag_v3?= =?utf-8?q?=2E3=2E5_for_changeset_62cf4e77f785?= Message-ID: <3fhdnK58Tsz7Llh@mail.python.org> http://hg.python.org/cpython/rev/993955b807b3 changeset: 89522:993955b807b3 branch: 3.3 user: Georg Brandl date: Sun Mar 09 09:37:32 2014 +0100 summary: Added tag v3.3.5 for changeset 62cf4e77f785 files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -122,3 +122,4 @@ 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 ca5635efe090f78806188ac2758f9948596aa8b2 v3.3.5rc2 +62cf4e77f78564714e7ea3d4bf1479ca1fbd0758 v3.3.5 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:18:30 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:18:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Post-release_u?= =?utf-8?q?pdate=2E?= Message-ID: <3fhdnL6c5hz7Lqv@mail.python.org> http://hg.python.org/cpython/rev/732401b87be6 changeset: 89523:732401b87be6 branch: 3.3 user: Georg Brandl date: Sun Mar 09 11:01:14 2014 +0100 summary: Post-release update. files: Include/patchlevel.h | 2 +- Misc/NEWS | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.3.5" +#define PY_VERSION "3.3.5+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,18 @@ Python News +++++++++++ +What's New in Python 3.3.6 release candidate 1? +=============================================== + +*Release date: XX-XXX-XXXX* + +Core and Builtins +----------------- + +Library +------- + + What's New in Python 3.3.5? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:18:32 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:18:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge_with_3=2E3=2E5_release_repo?= Message-ID: <3fhdnN1R0Dz7Lrh@mail.python.org> http://hg.python.org/cpython/rev/95d8ca61cdbf changeset: 89524:95d8ca61cdbf branch: 3.3 parent: 89516:6e9827ba455f parent: 89523:732401b87be6 user: Georg Brandl date: Sun Mar 09 11:17:23 2014 +0100 summary: merge with 3.3.5 release repo files: .hgtags | 2 + Include/patchlevel.h | 6 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 28 +++++++++++++++++++++++--- Misc/RPM/python-3.3.spec | 2 +- README | 4 +- 7 files changed, 34 insertions(+), 12 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -121,3 +121,5 @@ fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 +ca5635efe090f78806188ac2758f9948596aa8b2 v3.3.5rc2 +62cf4e77f78564714e7ea3d4bf1479ca1fbd0758 v3.3.5 diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 3 #define PY_MICRO_VERSION 5 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 2 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.3.5rc2" +#define PY_VERSION "3.3.5+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.5rc2" +__version__ = "3.3.5" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.5rc2" +IDLE_VERSION = "3.3.5" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,30 @@ Python News +++++++++++ +What's New in Python 3.3.6 release candidate 1? +=============================================== + +*Release date: XX-XXX-XXXX* + +Core and Builtins +----------------- + +Library +------- + +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + + +What's New in Python 3.3.5? +=========================== + +*Release date: 09-Mar-2014* + +No changes from release candidate 2. + + What's New in Python 3.3.5 release candidate 2? =============================================== @@ -22,10 +46,6 @@ Library ------- -- Issue #20283: RE pattern methods now accept the string keyword parameters - as documented. The pattern and source keyword parameters are left as - deprecated aliases. - - Issue #20778: Fix modulefinder to work with bytecode-only modules. - Issue #20791: copy.copy() now doesn't make a copy when the input is diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.5rc2 +%define version 3.3.5 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.3.5 release candidate 2 -================================================ +This is Python version 3.3.5 +============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:18:33 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:18:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3fhdnP3H7qz7Lrq@mail.python.org> http://hg.python.org/cpython/rev/3ae2cd85a908 changeset: 89525:3ae2cd85a908 parent: 89517:c88553c48d76 parent: 89524:95d8ca61cdbf user: Georg Brandl date: Sun Mar 09 11:18:16 2014 +0100 summary: merge with 3.3 files: .hgtags | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -121,6 +121,8 @@ fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 +ca5635efe090f78806188ac2758f9948596aa8b2 v3.3.5rc2 +62cf4e77f78564714e7ea3d4bf1479ca1fbd0758 v3.3.5 46535f65e7f3bcdcf176f36d34bc1fed719ffd2b v3.4.0a1 9265a2168e2cb2a84785d8717792acc661e6b692 v3.4.0a2 dd9cdf90a5073510877e9dd5112f8e6cf20d5e89 v3.4.0a3 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 11:28:27 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 9 Mar 2014 11:28:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Small_fixes_to_the_add-to-?= =?utf-8?q?pydotorg_script=2E?= Message-ID: <3fhf0q2fsKz7LlK@mail.python.org> http://hg.python.org/release/rev/31cba510ad2d changeset: 83:31cba510ad2d user: Georg Brandl date: Sun Mar 09 11:28:19 2014 +0100 summary: Small fixes to the add-to-pydotorg script. files: add-to-pydotorg.py | 19 ++++++++++++------- 1 files changed, 12 insertions(+), 7 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -26,6 +26,7 @@ import re import sys import json +import time import hashlib from os import path @@ -71,7 +72,7 @@ ('-' + release[5:] if release[5:] else '') def sigfile_for(release, rfile): - return download_root + '%s/%s.asc' % (release[:5], rfile) + return download_root + '%s/%s.asc' % (release, rfile) def md5sum_for(release, rfile): return hashlib.md5(open(ftp_root + release[:5] + '/' + rfile, 'rb').read()).hexdigest() @@ -100,13 +101,13 @@ """Return a dictionary with all needed fields for a ReleaseFile object.""" return dict( name = file_desc, - slug = slug_for(release) + '-' + make_slug(file_desc), + slug = slug_for(release) + '-' + make_slug(file_desc)[:40], os = '/api/v1/downloads/os/%s/' % os_pk, release = '/api/v1/downloads/release/%s/' % rel_pk, description = add_desc, is_source = os_pk == 3, url = download_root + '%s/%s' % (release[:5], rfile), - gpg_signature_file = sigfile_for(release, rfile), + gpg_signature_file = sigfile_for(release[:5], rfile), md5_sum = md5sum_for(release, rfile), filesize = filesize_for(release, rfile), download_button = 'tar.xz' in rfile or @@ -165,10 +166,13 @@ data=json.dumps(datadict), headers=headers) if resp.status_code != 201: try: - print resp.json['traceback'] + info = json.loads(resp.text) + print info.get('error_message', 'No error message.') + print info.get('traceback', '') except: pass - raise RuntimeError('creating %s failed: %s' % (objtype, resp.status_code)) + print 'Creating %s failed: %s' % (objtype, resp.status_code) + return -1 newloc = resp.headers['Location'] pk = int(newloc.strip('/').split('/')[-1]) return pk @@ -188,8 +192,9 @@ print 'Creating ReleaseFile object for', rfile file_dict = build_file_dict(rel, rfile, rel_pk, file_desc, os_pk, add_desc) file_pk = post_object('release_file', file_dict) - print 'Created as id =', file_pk - n += 1 + if file_pk >= 0: + print 'Created as id =', file_pk + n += 1 print 'Done - %d files added' % n main() -- Repository URL: http://hg.python.org/release From python-checkins at python.org Sun Mar 9 20:01:26 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 9 Mar 2014 20:01:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_note_that_futu?= =?utf-8?q?re=5Fbuiltin=27s_map_is_not_quite_like_python_3=27s_=28closes_?= =?utf-8?q?=2319363=29?= Message-ID: <3fhsNk2fMKz7Ll2@mail.python.org> http://hg.python.org/cpython/rev/16c5d7c289c6 changeset: 89526:16c5d7c289c6 branch: 2.7 parent: 89519:de632ccaffd6 user: Benjamin Peterson date: Sun Mar 09 14:01:09 2014 -0500 summary: note that future_builtin's map is not quite like python 3's (closes #19363) Patch by Gareth Rees. files: Doc/library/future_builtins.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/library/future_builtins.rst b/Doc/library/future_builtins.rst --- a/Doc/library/future_builtins.rst +++ b/Doc/library/future_builtins.rst @@ -50,6 +50,11 @@ Works like :func:`itertools.imap`. + .. note:: + + In Python 3, :func:`map` does not accept ``None`` for the + function argument. + .. function:: oct(object) Works like the built-in :func:`oct`, but instead of :meth:`__oct__` it will -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:44 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_hmac_*digestmo?= =?utf-8?q?d*_accepts_strings=2C_and_default_is_deprecated=2E_=28=2317276?= =?utf-8?q?=29?= Message-ID: <3fhslX10tNz7Lmn@mail.python.org> http://hg.python.org/cpython/rev/c10ec51a2ce4 changeset: 89527:c10ec51a2ce4 parent: 89525:3ae2cd85a908 user: R David Murray date: Sat Mar 08 22:43:12 2014 -0500 summary: whatsnew: hmac *digestmod* accepts strings, and default is deprecated. (#17276) files: Doc/library/hmac.rst | 5 ++--- Doc/whatsnew/3.4.rst | 15 ++++++++++++++- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -23,9 +23,8 @@ defaults to the :data:`hashlib.md5` constructor. .. versionchanged:: 3.4 - Parameter *key* can be a bytes or bytearray object. Parameter *msg* can - be of any type supported by :mod:`hashlib`. - + Parameter *key* can be a bytes or bytearray object. + Parameter *msg* can be of any type supported by :mod:`hashlib`. Paramter *digestmod* can be the name of a hash algorithm. .. deprecated:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -826,6 +826,12 @@ accepts any type supported by the :mod:`hashlib` module. (Contributed by Jonas Borgstr?m in :issue:`18240`.) +The *digestmod* argument to the :func:`hmac.new` function may now be any hash +digest name recognized by :mod:`hashlib`. In addition, the current behavior in +which the value of *digestmod* defaults to ``MD5`` is deprecated: in a +future version of Python there will be no default value. (Contributed by +Christian Heimes in :issue:`17276`.) + html ---- @@ -1849,6 +1855,9 @@ * The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. +* The :func:`hmac.new` *digestmod* keyword having a default value (currently + ``MD5``) is deprecated. + Deprecated Functions and Types in the C API ------------------------------------------- @@ -2139,7 +2148,11 @@ currently defaults to ``False`` for backward compatibility, but will eventually be changed to default to ``True``. It is recommended that you add this keyword, with the appropriate value, to any - :class:`~html.parser.HTMLParser` calls in your code. + :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). + +* Since the *digestmod* argument to the :func:`hmac.new` function will in the + future have no default, all calls to :func:`hmac.new` should be changed to + explicitly specify a *digestmod*. (:issue:`17276`). Changes in the C API -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:45 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_hmac_conforms_?= =?utf-8?q?to_pep_247=2E_=28=2318775=29?= Message-ID: <3fhslY3bVrz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/dde3fb877b07 changeset: 89528:dde3fb877b07 user: R David Murray date: Sat Mar 08 23:18:09 2014 -0500 summary: whatsnew: hmac conforms to pep 247. (#18775) files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -832,6 +832,11 @@ future version of Python there will be no default value. (Contributed by Christian Heimes in :issue:`17276`.) +With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` +attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` +attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. +(Contributed by Christian Heimes in :issue:`18775`.) + html ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:46 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_SSLcontext=2El?= =?utf-8?q?oad=5Fverify=5Flocations_cadata_argument_=28=2318138=29?= Message-ID: <3fhslZ51njz7Ls4@mail.python.org> http://hg.python.org/cpython/rev/8e3b3b4a90fb changeset: 89529:8e3b3b4a90fb user: R David Murray date: Sat Mar 08 23:19:45 2014 -0500 summary: whatsnew: SSLcontext.load_verify_locations cadata argument (#18138) files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1269,6 +1269,11 @@ Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orr? and Antoine Pitrou in :issue:`16692`) +:class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` +accepts a new optional argument *cadata*, which can be used to provide PEM or +DER encoded certificates directly via strings or bytes, respectively. +(Contributed by Christian Heimes in :issue:`18138`.) + New function :func:`~ssl.get_default_verify_paths` returns a named tuple of the paths and environment variables that the :meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:48 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbjogd2hhdHNuZXc6IHN5cy5fX2lu?= =?utf-8?b?dGVyYWN0aXZlaG9va19fLiAoIzU4NDUp?= Message-ID: <3fhslc1GgNz7Lrp@mail.python.org> http://hg.python.org/cpython/rev/69c451851c71 changeset: 89530:69c451851c71 user: R David Murray date: Sun Mar 09 12:03:30 2014 -0400 summary: whatsnew: sys.__interactivehook__. (#5845) I decided to leave the (slightly expanded) Other Improvements entry as well, since someone looking for command line behavior changes isn't going to look in the 'sys' module section. files: Doc/library/site.rst | 10 +++++++--- Doc/library/sys.rst | 9 +++++---- Doc/whatsnew/3.4.rst | 17 ++++++++++++++++- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/Doc/library/site.rst b/Doc/library/site.rst --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -123,9 +123,13 @@ configure the :mod:`rlcompleter` module, if Python is started in :ref:`interactive mode ` and without the :option:`-S` option. The default behavior is enable tab-completion and to use -:file:`~/.python_history` as the history save file. To disable it, override -the :data:`sys.__interactivehook__` attribute in your :mod:`sitecustomize` -or :mod:`usercustomize` module or your :envvar:`PYTHONSTARTUP` file. +:file:`~/.python_history` as the history save file. To disable it, delete (or +override) the :data:`sys.__interactivehook__` attribute in your +:mod:`sitecustomize` or :mod:`usercustomize` module or your +:envvar:`PYTHONSTARTUP` file. + +.. versionchanged:: 3.4 + Activation of rlcompleter and history was made automatic. Module contents diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -692,10 +692,11 @@ .. data:: __interactivehook__ - When present, this function is automatically called (with no arguments) - when the interpreter is launched in :ref:`interactive mode `. - This is done after the :envvar:`PYTHONSTARTUP` file is read, so that you - can set this hook there. + When this attribute exists, its value is automatically called (with no + arguments) when the interpreter is launched in :ref:`interactive mode + `. This is done after the :envvar:`PYTHONSTARTUP` file is + read, so that you can set this hook there. The :mod:`site` module + :ref:`sets this `. .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1360,6 +1360,19 @@ especially if automated via a test suite. (Contributed by Antoine Pitrou in :issue:`13390`.) +When the Python interpreter starts in :ref:`interactive mode +`, it checks for an :data:`~sys.__interactivehook__` attribute +on the :mod:`sys` module. If the attribute exists, its value is called with no +arguments just before interactive mode is started. The check is made after the +:envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` +module :ref:`sets it ` to a function that enables tab +completion and history saving (in :file:`~/.python-history`) if the platform +supports :mod:`readline`. If you do not want this (new) behavior, you can +override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or +:mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it +to some other callable). (Contributed by ?ric Araujo and Antoine Pitrou in +:issue:`5845`.) + textwrap -------- @@ -1694,7 +1707,9 @@ most POSIX systems it can and should be used in the ``#!`` line of system scripts. (Contributed by Christian Heimes in :issue:`16499`.) -* Tab-completion is now enabled by default in the interactive interpreter. +* Tab-completion is now enabled by default in the interactive interpreter + on systems that support :mod:`readline`. History is also enabled by default, + and is written to (and read from) the file :file:`~/.python-history`. (Contributed by Antoine Pitrou and ?ric Araujo in :issue:`5845`.) * Invoking the Python interpreter with ``--version`` now outputs the version to -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:49 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_SSLSocket=2Ege?= =?utf-8?q?tpeercert_new_values_returned=2E_=28=2318379=29?= Message-ID: <3fhsld2w0Nz7Ll8@mail.python.org> http://hg.python.org/cpython/rev/7885876b6503 changeset: 89531:7885876b6503 user: R David Murray date: Sun Mar 09 12:55:58 2014 -0400 summary: whatsnew: SSLSocket.getpeercert new values returned. (#18379) files: Doc/library/ssl.rst | 4 +--- Doc/whatsnew/3.4.rst | 4 ++++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -841,10 +841,8 @@ .. versionchanged:: 3.4 :exc:`ValueError` is raised when the handshake isn't done. - - .. versionchanged:: 3.4 The returned dictionary includes additional X509v3 extension items - such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. + such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. .. method:: SSLSocket.cipher() diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1297,6 +1297,10 @@ :meth:`ssl.SSLContext.set_servername_callback` method. (Contributed by Daniel Black in :issue:`8109`.) +The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional +``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and +``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) + stat ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:17:50 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:17:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_plistlib_new_a?= =?utf-8?q?pi_and_deprecations_=28=2314455=29?= Message-ID: <3fhslf4b12z7Lll@mail.python.org> http://hg.python.org/cpython/rev/728f626ee337 changeset: 89532:728f626ee337 user: R David Murray date: Sun Mar 09 15:09:04 2014 -0400 summary: whatsnew: plistlib new api and deprecations (#14455) Also fixed the versionchanged/deprecation tags in the docs. files: Doc/library/plistlib.rst | 13 ++++++++----- Doc/whatsnew/3.4.rst | 17 ++++++++++++++--- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -32,6 +32,9 @@ (but only with string keys), :class:`Data`, :class:`bytes`, :class:`bytesarray` or :class:`datetime.datetime` objects. +.. versionchanged:: 3.4 + New API, old API deprecated. Support for binary format plists added. + .. seealso:: `PList manual page `_ @@ -107,7 +110,7 @@ An :exc:`OverflowError` will be raised for integer values that cannot be represented in (binary) plist files. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: dumps(value, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) @@ -116,7 +119,7 @@ the documentation for :func:`dump` for an explanation of the keyword arguments of this function. - .. versionadded: 3.4 + .. versionadded:: 3.4 The following functions are deprecated: @@ -135,7 +138,7 @@ to ``__getitem_``. This means that you can use attribute access to access items of these dictionaries. - .. deprecated: 3.4 Use :func:`load` instead. + .. deprecated:: 3.4 Use :func:`load` instead. .. function:: writePlist(rootObject, pathOrFile) @@ -143,7 +146,7 @@ Write *rootObject* to an XML plist file. *pathOrFile* may be either a file name or a (writable and binary) file object - .. deprecated: 3.4 Use :func:`dump` instead. + .. deprecated:: 3.4 Use :func:`dump` instead. .. function:: readPlistFromBytes(data) @@ -194,7 +197,7 @@ .. deprecated:: 3.4 Use a :class:`bytes` object instead -The following constants are avaiable: +The following constants are available: .. data:: FMT_XML diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1097,9 +1097,13 @@ plistlib -------- -:mod:`plistlib` now supports binary plist files, and offers the common -``load``/``loads``/``dump``/``dumps`` API pattern for serialization formats -(Contributed by Ronald Oussoren and others in :issue:`14455`). +:mod:`plistlib` now has an API that is similar to the standard pattern for +stdlib serialization protocols, with new :func:`~plistlib.load`, +:func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` +functions. (The older API is now deprecated.) In addition to the already +supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports +the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald +Oussoren and others in :issue:`14455`). poplib @@ -1887,6 +1891,13 @@ * The :func:`hmac.new` *digestmod* keyword having a default value (currently ``MD5``) is deprecated. +* The :mod:`plistlib` :func:`~plistlib.readPlist`, + :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and + :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the + corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, + :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` + is deprecated in favor of just using the :class:`bytes` constructor. + Deprecated Functions and Types in the C API ------------------------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 20:30:34 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 20:30:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2320871=3A_improve_email_?= =?utf-8?q?policy_test_coverage=2E?= Message-ID: <3fht2L65zPz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/8f7486263212 changeset: 89533:8f7486263212 user: R David Murray date: Sun Mar 09 15:29:24 2014 -0400 summary: #20871: improve email policy test coverage. More tests of the concrete policy methods would probably be a good idea, but this is a start, and it brings line coverage up to 100% for the policy module. Patch by Milan Oberkirch. files: Lib/test/test_email/test_policy.py | 9 +++++++++ Misc/ACKS | 1 + 2 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py --- a/Lib/test/test_email/test_policy.py +++ b/Lib/test/test_email/test_policy.py @@ -319,5 +319,14 @@ self.assertEqual(msg.as_string(), "Subject: testXTo: fooXX") +class TestConcretePolicies(unittest.TestCase): + + def test_header_store_parse_rejects_newlines(self): + instance = email.policy.EmailPolicy() + self.assertRaises(ValueError, + instance.header_store_parse, + 'From', 'spam\negg at foo.py') + + if __name__ == '__main__': unittest.main() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -951,6 +951,7 @@ James Oakley Elena Oat Jon Oberheide +Milan Oberkirch Pascal Oberndoerfer Jeffrey Ollie Adam Olsen -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 22:48:32 2014 From: python-checkins at python.org (ned.deily) Date: Sun, 9 Mar 2014 22:48:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwODc1?= =?utf-8?q?=3A_Prevent_possible_gzip_=22=27read=27_is_not_defined=22_NameE?= =?utf-8?q?rror=2E?= Message-ID: <3fhx5X2SRpz7LjP@mail.python.org> http://hg.python.org/cpython/rev/c3836de644e0 changeset: 89534:c3836de644e0 branch: 3.3 parent: 89524:95d8ca61cdbf user: Ned Deily date: Sun Mar 09 14:44:34 2014 -0700 summary: Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. files: Lib/gzip.py | 2 +- Lib/test/test_gzip.py | 7 +++++++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/gzip.py b/Lib/gzip.py --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -99,7 +99,7 @@ self._read -= len(prepend) return else: - self._buffer = self._buffer[read:] + prepend + self._buffer = self._buffer[self._read:] + prepend self._length = len(self._buffer) self._read = 0 diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -396,6 +396,13 @@ with gzip.GzipFile(fileobj=io.BytesIO(gzdata)) as f: self.assertEqual(f.read(), b'Test') + def test_prepend_error(self): + # See issue #20875 + with gzip.open(self.filename, "wb") as f: + f.write(data1) + with gzip.open(self.filename, "rb") as f: + f.fileobj.prepend() + class TestOpen(BaseTest): def test_binary_modes(self): uncompressed = data1 * 50 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ Library ------- +- Issue #20875: Prevent possible gzip "'read' is not defined" NameError. + Patch by Claudiu Popa. + - Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 22:48:33 2014 From: python-checkins at python.org (ned.deily) Date: Sun, 9 Mar 2014 22:48:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320875=3A_Merge_from_3=2E3?= Message-ID: <3fhx5Y4Gnvz7LjP@mail.python.org> http://hg.python.org/cpython/rev/42599de6a4de changeset: 89535:42599de6a4de parent: 89533:8f7486263212 parent: 89534:c3836de644e0 user: Ned Deily date: Sun Mar 09 14:47:58 2014 -0700 summary: Issue #20875: Merge from 3.3 files: Lib/gzip.py | 2 +- Lib/test/test_gzip.py | 7 +++++++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/gzip.py b/Lib/gzip.py --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -96,7 +96,7 @@ self._read -= len(prepend) return else: - self._buffer = self._buffer[read:] + prepend + self._buffer = self._buffer[self._read:] + prepend self._length = len(self._buffer) self._read = 0 diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -421,6 +421,13 @@ with gzip.GzipFile(fileobj=io.BytesIO(gzdata)) as f: self.assertEqual(f.read(), b'Test') + def test_prepend_error(self): + # See issue #20875 + with gzip.open(self.filename, "wb") as f: + f.write(data1) + with gzip.open(self.filename, "rb") as f: + f.fileobj.prepend() + class TestOpen(BaseTest): def test_binary_modes(self): uncompressed = data1 * 50 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #20875: Prevent possible gzip "'read' is not defined" NameError. + Patch by Claudiu Popa. + - Issue #11558: ``email.message.Message.attach`` now returns a more useful error message if ``attach`` is called on a message for which ``is_multipart`` is False. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:50:27 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:50:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_SSLContext=2Ev?= =?utf-8?q?erify=5Fflags_and_constants=2E_=28=238813=29?= Message-ID: <3fhySz0KKqz7Ll3@mail.python.org> http://hg.python.org/cpython/rev/1508c4c9e747 changeset: 89536:1508c4c9e747 user: R David Murray date: Sun Mar 09 17:01:34 2014 -0400 summary: whatsnew: SSLContext.verify_flags and constants. (#8813) files: Doc/whatsnew/3.4.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1305,6 +1305,14 @@ ``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and ``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) +If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new +attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the +certificate verification process by setting it to some combination of the new +constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, +:data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. +OpenSSL does not do any CRL verification by default. (Contributed by +Christien Heimes in :issue:`8813`.) + stat ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:50:28 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:50:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_deprecation_of?= =?utf-8?q?_sysconfig_SO_key_=28=2319555=29=2E?= Message-ID: <3fhyT025x6z7LmQ@mail.python.org> http://hg.python.org/cpython/rev/ac2ee9fc353a changeset: 89537:ac2ee9fc353a user: R David Murray date: Sun Mar 09 17:11:00 2014 -0400 summary: whatsnew: deprecation of sysconfig SO key (#19555). files: Doc/whatsnew/3.4.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1906,6 +1906,9 @@ :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` is deprecated in favor of just using the :class:`bytes` constructor. +* The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by + ``EXT_SUFFIX``. + Deprecated Functions and Types in the C API ------------------------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:50:29 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:50:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_more_accurate_?= =?utf-8?q?deprecation_section_titles=2E?= Message-ID: <3fhyT13mpwz7Lls@mail.python.org> http://hg.python.org/cpython/rev/4d9e0a6b0ddc changeset: 89538:4d9e0a6b0ddc user: R David Murray date: Sun Mar 09 17:13:31 2014 -0400 summary: whatsnew: more accurate deprecation section titles. files: Doc/whatsnew/3.4.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1845,8 +1845,8 @@ using ``-Wd``). -Deprecated Python Modules, Functions and Methods ------------------------------------------------- +Deprecations in the Python API +------------------------------ * :meth:`difflib.SequenceMatcher.isbjunk` and :meth:`difflib.SequenceMatcher.isbpopular` were removed: use ``x in sm.bjunk`` and @@ -1910,8 +1910,8 @@ ``EXT_SUFFIX``. -Deprecated Functions and Types in the C API -------------------------------------------- +Deprecations in the C API +------------------------- XXX: None so far -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:50:30 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:50:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_venv_with=5Fpi?= =?utf-8?q?p=2C_pyvenv_--without-pip_=28=2319552=29?= Message-ID: <3fhyT2594sz7Lm1@mail.python.org> http://hg.python.org/cpython/rev/a140caad76bc changeset: 89539:a140caad76bc user: R David Murray date: Sun Mar 09 18:09:51 2014 -0400 summary: whatsnew: venv with_pip, pyvenv --without-pip (#19552) files: Doc/whatsnew/3.4.rst | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1507,6 +1507,12 @@ :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` shells (Contributed by Andrew Svetlov in :issue:`15417`.) +:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function +take a new keyword argument *with_pip*, which defaults to ``False``, that +controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is +installed in the virtual environment. (Contributed by Nick Coghlan in +:issue:`19552` as part of the :pep:`453` implementation.) + wave ---- @@ -1772,6 +1778,11 @@ to use copies rather than symlinks even on systems where symlinks are the default. (Contributed by Vinay Sajip in :issue:`18807`.) +* The :ref:`pyvenv ` command also accepts a ``--without-pip`` + option to suppress the otherwise-automatic bootstrapping of pip into + the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` + as part of the :pep:`453` implementation.) + * The encoding name is now optional in the value set for the :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to set just the error handler, without changing the default encoding. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:52:09 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:52:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5OTUzOiBDbGFy?= =?utf-8?q?ify_the_wording_of_the_augmented_assignment_discussion=2E?= Message-ID: <3fhyVx2SWCz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/f9cb5a44879c changeset: 89540:f9cb5a44879c branch: 3.3 parent: 89534:c3836de644e0 user: R David Murray date: Sun Mar 09 18:51:16 2014 -0400 summary: #19953: Clarify the wording of the augmented assignment discussion. Patch by Priya Pappachan, based on suggestions from Terry Reedy and myself. files: Doc/faq/programming.rst | 1 + Doc/reference/datamodel.rst | 12 +++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1103,6 +1103,7 @@ result = [obj.method() for obj in mylist] +.. _faq-augmented-assignment-tuple-error: Why does a_tuple[i] += ['item'] raise an exception when the addition works? --------------------------------------------------------------------------- diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2023,11 +2023,13 @@ ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation in-place (modifying *self*) and return the result (which could be, but does not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, to execute the - statement ``x += y``, where *x* is an instance of a class that has an - :meth:`__iadd__` method, ``x.__iadd__(y)`` is called. If *x* is an instance - of a class that does not define a :meth:`__iadd__` method, ``x.__add__(y)`` - and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. + assignment falls back to the normal methods. For instance, if *x* is an + instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent + to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` + are considered, as with the evaluation of ``x + y``. In certain situations, + augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in + fact part of the data model. .. method:: object.__neg__(self) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 9 23:52:10 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 9 Mar 2014 23:52:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_=2319953=3A_Clarify_the_wording_of_the_augmented_a?= =?utf-8?q?ssignment_discussion=2E?= Message-ID: <3fhyVy4T16z7Lm1@mail.python.org> http://hg.python.org/cpython/rev/61ceb299a255 changeset: 89541:61ceb299a255 parent: 89539:a140caad76bc parent: 89540:f9cb5a44879c user: R David Murray date: Sun Mar 09 18:51:35 2014 -0400 summary: Merge #19953: Clarify the wording of the augmented assignment discussion. files: Doc/faq/programming.rst | 1 + Doc/reference/datamodel.rst | 12 +++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1103,6 +1103,7 @@ result = [obj.method() for obj in mylist] +.. _faq-augmented-assignment-tuple-error: Why does a_tuple[i] += ['item'] raise an exception when the addition works? --------------------------------------------------------------------------- diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2050,11 +2050,13 @@ ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation in-place (modifying *self*) and return the result (which could be, but does not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, to execute the - statement ``x += y``, where *x* is an instance of a class that has an - :meth:`__iadd__` method, ``x.__iadd__(y)`` is called. If *x* is an instance - of a class that does not define a :meth:`__iadd__` method, ``x.__add__(y)`` - and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. + assignment falls back to the normal methods. For instance, if *x* is an + instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent + to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` + are considered, as with the evaluation of ``x + y``. In certain situations, + augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in + fact part of the data model. .. method:: object.__neg__(self) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:06 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_pdb_works_for_?= =?utf-8?q?debugging_asyncio_programs_=28=2316596=29=2E?= Message-ID: <3fj26y1Y81z7LkY@mail.python.org> http://hg.python.org/cpython/rev/5c6c96c82afb changeset: 89542:5c6c96c82afb user: R David Murray date: Sun Mar 09 19:40:30 2014 -0400 summary: whatsnew: pdb works for debugging asyncio programs (#16596). files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1066,6 +1066,11 @@ pdb --- +:mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and +:keyword:`yield from` in a more useful fashion. This is especially helpful when +debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and +Xavier de Gaye in :issue:`16596`.) + The ``print`` command has been removed from :mod:`pdb`, restoring access to the Python :func:`print` function from the pdb command line. Python2's ``pdb`` did not have a ``print`` command; instead, entering ``print`` executed the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:07 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_SSLContext=2El?= =?utf-8?b?b2FkX2RlZmF1bHRfY2VydHMgKCMxOTI5Miku?= Message-ID: <3fj26z3QFQz7LkB@mail.python.org> http://hg.python.org/cpython/rev/35a5284d5388 changeset: 89543:35a5284d5388 user: R David Murray date: Sun Mar 09 19:41:25 2014 -0400 summary: whatsnew: SSLContext.load_default_certs (#19292). files: Doc/whatsnew/3.4.rst | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1318,6 +1318,15 @@ OpenSSL does not do any CRL verification by default. (Contributed by Christien Heimes in :issue:`8813`.) +New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` +loads a set of dfault "certificate authority" (CA) certificates from default +locations, which vary according to the platform. It can be used to load both +TLS web server authentication certificates +(``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a +server, and certificates for a server to use in verifying client certificates +(``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian +Heimes in :issue:`19292`.) + stat ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:08 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_allowZip64_def?= =?utf-8?q?aults_to_True_=28=2317201=29=2E?= Message-ID: <3fj2704vP8z7LkR@mail.python.org> http://hg.python.org/cpython/rev/86338438b70d changeset: 89544:86338438b70d user: R David Murray date: Sun Mar 09 19:57:19 2014 -0400 summary: whatsnew: allowZip64 defaults to True (#17201). files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1589,6 +1589,10 @@ example, this could be used to exclude test files from the archive. (Contributed by Christian Tismer in :issue:`19274`.) +The *allowZip64* parameter to :class:`~zipfile.ZipFile` and +:class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by +William Mallard in :issue:`17201`.) + CPython Implementation Changes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:09 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_ssl=2Ecreate?= =?utf-8?b?X2RlZmF1bHRfY29udGV4dCAoIzE5Njg5KS4=?= Message-ID: <3fj2716GPDz7LlY@mail.python.org> http://hg.python.org/cpython/rev/8b4b6609cd31 changeset: 89545:8b4b6609cd31 user: R David Murray date: Sun Mar 09 19:58:17 2014 -0400 summary: whatsnew: ssl.create_default_context (#19689). files: Doc/whatsnew/3.4.rst | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1327,6 +1327,13 @@ (``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian Heimes in :issue:`19292`.) +New function :func:`~ssl.create_default_context` provides a standard way to +obtain an :class:`~ssl.SSLContext` based on the settings Python uses by +default. These settings are more stringent than the defaults provided by the +:class:`~ssl.SSLContext` constructor, and may be adjusted (without notice) in +the future to maintain a balance between compatibility and security. +(Contributed by Christian Heimes in :issue:`19689`.) + stat ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:11 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_cp1125_codec_?= =?utf-8?b?KCMxOTY2OCku?= Message-ID: <3fj2730xsyz7LmK@mail.python.org> http://hg.python.org/cpython/rev/11bc9f669015 changeset: 89546:11bc9f669015 user: R David Murray date: Sun Mar 09 20:13:16 2014 -0400 summary: whatsnew: cp1125 codec (#19668). files: Doc/whatsnew/3.4.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -367,8 +367,11 @@ Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. -* New EBCDIC :ref:`codec ` ``cp273``. (Contributed by - Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) +* New German EBCDIC :ref:`codec ` ``cp273``. (Contributed + by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) + +* New Ukrainian :ref:`codec ` ``cp1125``. (Contributed by + Serhiy Storchaka in :issue:`19668`.) * :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary buffer objects as arguments. (Contributed by Antoine Pitrou in -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:12 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_=27U=27_mode_d?= =?utf-8?q?eprecation_=28=2315204=29=2E?= Message-ID: <3fj2742w8Tz7Lm2@mail.python.org> http://hg.python.org/cpython/rev/2d5544afb510 changeset: 89547:2d5544afb510 user: R David Murray date: Sun Mar 09 20:33:35 2014 -0400 summary: whatsnew: 'U' mode deprecation (#15204). files: Doc/whatsnew/3.4.rst | 14 +++++++++++++- 1 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1948,6 +1948,11 @@ * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by ``EXT_SUFFIX``. +* The ``U`` mode accepted by various ``open`` functions is deprecated. + In Python3 it does not do anything useful, and should be replaced by + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* + argument. + Deprecations in the C API ------------------------- @@ -2242,7 +2247,14 @@ * Since the *digestmod* argument to the :func:`hmac.new` function will in the future have no default, all calls to :func:`hmac.new` should be changed to - explicitly specify a *digestmod*. (:issue:`17276`). + explicitly specify a *digestmod* (:issue:`17276`). + +* Any calls to ``open`` functions that specify ``U`` should be modified. + ``U`` is ineffective in Python3 and will eventually raise an error if used. + Depending on the function, the equivalent of its old Python2 behavior can be + achieved using either a *newline* argument, or if necessary by wrapping the + stream in :mod:`~io.TextIOWrapper` to use its *newline* argument + (:issue:`15204`). Changes in the C API -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:13 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_porting_note_f?= =?utf-8?q?or_sysconfig_=27SO=27_key_deprecation=2E?= Message-ID: <3fj2754rjBz7LlM@mail.python.org> http://hg.python.org/cpython/rev/39b2679ec4e5 changeset: 89548:39b2679ec4e5 user: R David Murray date: Sun Mar 09 20:39:52 2014 -0400 summary: whatsnew: porting note for sysconfig 'SO' key deprecation. files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2249,6 +2249,11 @@ future have no default, all calls to :func:`hmac.new` should be changed to explicitly specify a *digestmod* (:issue:`17276`). +* Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking + ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` + is deprecated. This key should be replaced by ``EXT_SUFFIX`` or + ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). + * Any calls to ``open`` functions that specify ``U`` should be modified. ``U`` is ineffective in Python3 and will eventually raise an error if used. Depending on the function, the equivalent of its old Python2 behavior can be -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:14 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_auidioop=2Ebyt?= =?utf-8?b?ZXN3YXAgKCMxNjY4NSku?= Message-ID: <3fj2766bJRz7Lm2@mail.python.org> http://hg.python.org/cpython/rev/c375697f062e changeset: 89549:c375697f062e user: R David Murray date: Sun Mar 09 20:42:49 2014 -0400 summary: whatsnew: auidioop.byteswap (#16685). files: Doc/library/audioop.rst | 2 +- Doc/whatsnew/3.4.rst | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -80,7 +80,7 @@ "Byteswap" all samples in a fragment and returns the modified fragment. Converts big-endian samples to little-endian and vice versa. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: cross(fragment, width) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -570,8 +570,9 @@ :mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka in :issue:`12866`.) -Added the :func:`~audioop.byteswap` function to convert big-endian samples -to little-endian and vice versa (:issue:`19641`). +New :func:`~audioop.byteswap` function converts big-endian samples to +little-endian and vice versa (Contributed by Serhiy Storchaka in +:issue:`19641`). All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings are not accepted: they didn't work before, now they raise an error right away. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:16 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_give_Hugo_cred?= =?utf-8?q?it_for_his_preliminary_work_on_regex_patch=2E?= Message-ID: <3fj27810g6z7LkY@mail.python.org> http://hg.python.org/cpython/rev/8bef27cdcd0b changeset: 89550:8bef27cdcd0b user: R David Murray date: Sun Mar 09 20:46:32 2014 -0400 summary: whatsnew: give Hugo credit for his preliminary work on regex patch. files: Doc/whatsnew/3.4.rst | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1178,8 +1178,9 @@ The repr of :ref:`regex objects ` now includes the pattern and the flags; the repr of :ref:`match objects ` now -includes the start, end, and the part of the string that matched. -(Contributed by Serhiy Storchaka in :issue:`13592` and :issue:`17087`.) +includes the start, end, and the part of the string that matched. (Contributed +by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and +:issue:`17087`.) resource -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:17 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_expand_re=2Efu?= =?utf-8?q?llmatch_entry_with_the_motivation_for_the_feature=2E?= Message-ID: <3fj2792K4vz7LmS@mail.python.org> http://hg.python.org/cpython/rev/382691325133 changeset: 89551:382691325133 user: R David Murray date: Sun Mar 09 20:52:28 2014 -0400 summary: whatsnew: expand re.fullmatch entry with the motivation for the feature. files: Doc/whatsnew/3.4.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1172,9 +1172,12 @@ re -- -Added :func:`re.fullmatch` function and :meth:`regex.fullmatch` method, -which anchor the pattern at both ends of the string to match. -(Contributed by Matthew Barnett in :issue:`16203`.) +New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor +the pattern at both ends of the string to match. This provides a way to be +explicit about the goal of the match, which avoids a class of subtle bugs where +``$`` characters get lost during code changes or the addition of alternatives +to an existing regular expression. (Contributed by Matthew Barnett in +:issue:`16203`.) The repr of :ref:`regex objects ` now includes the pattern and the flags; the repr of :ref:`match objects ` now -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:18 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_oops=2C_create?= =?utf-8?q?=5Fdefault=5Fcontext_doesn=27t_return_what_the_stdlib_uses=2E?= Message-ID: <3fj27B3hCjz7LkB@mail.python.org> http://hg.python.org/cpython/rev/fcd1eb5c6026 changeset: 89552:fcd1eb5c6026 user: R David Murray date: Sun Mar 09 21:00:14 2014 -0400 summary: whatsnew: oops, create_default_context doesn't return what the stdlib uses. files: Doc/whatsnew/3.4.rst | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1336,11 +1336,12 @@ Heimes in :issue:`19292`.) New function :func:`~ssl.create_default_context` provides a standard way to -obtain an :class:`~ssl.SSLContext` based on the settings Python uses by -default. These settings are more stringent than the defaults provided by the -:class:`~ssl.SSLContext` constructor, and may be adjusted (without notice) in -the future to maintain a balance between compatibility and security. -(Contributed by Christian Heimes in :issue:`19689`.) +obtain an :class:`~ssl.SSLContext` whose settings are intended to be a +reasonable balance between compatibility and security. These settings are +more stringent than the defaults provided by the :class:`~ssl.SSLContext` +constructor, and may be adjusted in the future, without prior deprecation, if +best-practice security requirements change. (Contributed by Christian Heimes +in :issue:`19689`.) stat -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:19 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_dis=2Estack=5F?= =?utf-8?q?effect_=28=2319722=29=2E?= Message-ID: <3fj27C564qz7LlB@mail.python.org> http://hg.python.org/cpython/rev/4a801f8b7e2d changeset: 89553:4a801f8b7e2d user: R David Murray date: Sun Mar 09 21:07:37 2014 -0400 summary: whatsnew: dis.stack_effect (#19722). files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -696,6 +696,10 @@ (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` and Claudiu Popa in :issue:`17916`) +New function :func:`~dis.stack_effect` computes the effect on the Python stack +of a given opcode and argument, information that is not otherwise available. +(Contributed by Larry Hastings in :issue:`19722`.) + doctest ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:35:21 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 02:35:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_tarfile_cli_?= =?utf-8?b?KCMxMzQ3Nyku?= Message-ID: <3fj27F00g1z7Lls@mail.python.org> http://hg.python.org/cpython/rev/5b52db6fc7dc changeset: 89554:5b52db6fc7dc user: R David Murray date: Sun Mar 09 21:19:57 2014 -0400 summary: whatsnew: tarfile cli (#13477). files: Doc/whatsnew/3.4.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1424,6 +1424,14 @@ :issue:`5845`.) +tarfile +------- + +The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when +called as a script directly or via ``-m``. This can be used to create and +extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) + + textwrap -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 02:59:36 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 10 Mar 2014 02:59:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_use_assertEqual_instead_of?= =?utf-8?q?_the_deprecated_assertEquals_=28closes_=2320877=29?= Message-ID: <3fj2gD0ZM6zS0d@mail.python.org> http://hg.python.org/cpython/rev/2bda1065e931 changeset: 89555:2bda1065e931 user: Benjamin Peterson date: Sun Mar 09 20:59:24 2014 -0500 summary: use assertEqual instead of the deprecated assertEquals (closes #20877) files: Lib/test/test_grammar.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -319,8 +319,8 @@ def f(self, *, __kw:1): pass class Ham(Spam): pass - self.assertEquals(Spam.f.__annotations__, {'_Spam__kw': 1}) - self.assertEquals(Ham.f.__annotations__, {'_Spam__kw': 1}) + self.assertEqual(Spam.f.__annotations__, {'_Spam__kw': 1}) + self.assertEqual(Ham.f.__annotations__, {'_Spam__kw': 1}) # Check for SF Bug #1697248 - mixing decorators and a return annotation def null(x): return x @null -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 08:04:25 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 08:04:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Updated_Windows_instructions_?= =?utf-8?q?as_per_MvL=2E?= Message-ID: <3fj9Qx0LbKz7LjQ@mail.python.org> http://hg.python.org/peps/rev/027b1e215e08 changeset: 5405:027b1e215e08 user: Georg Brandl date: Mon Mar 10 08:04:10 2014 +0100 summary: Updated Windows instructions as per MvL. files: pep-0101.txt | 24 ++++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -289,6 +289,30 @@ ___ The WE then generates Windows installer files for each Windows target architecture (for Python 3.3, this means x86 and AMD64). + - He has one checkout tree per target architecture, and builds the + pcbuild.sln project for the appropriate architecture. + + - PC\icons.mak must have been run with nmake. + + - The cmd.exe window in which this is run must have Cygwin/bin in its + path (at least for x86). + + - The cmd.exe window must have MS compiler tools for the target + architecture in its path (VS 2010 for Python 3.3). + + - The WE then edits Tools/msi/config.py (a file only present locally) + to update full_current_version and sets snapshot to false. Currently + for a release config.py looks like + + snapshot=0 + full_current_version="3.3.5rc2" + certname="Python Software Foundation + PCBUILD='PCbuild\\amd64' + + The last line is only present for the amd64 checkout. + + - Now he runs msi.py with ActivePython or Python with pywin32. + The WE checksums the files (*.msi, *.chm, *-pdb.zip), uploads them to dinsdale together with gpg signature files, and emails you the location and md5sums. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 10 09:06:31 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 09:06:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Add_a_shebang=2C_for_good_?= =?utf-8?q?measure=2E?= Message-ID: <3fjBpb4y25z7LjV@mail.python.org> http://hg.python.org/release/rev/a3391892eb9e changeset: 84:a3391892eb9e user: Georg Brandl date: Mon Mar 10 09:06:18 2014 +0100 summary: Add a shebang, for good measure. files: add-to-pydotorg.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python """ Script to add ReleaseFile objects for Python releases on the new pydotorg. To use (RELEASE is something like 3.3.5rc2): -- Repository URL: http://hg.python.org/release From python-checkins at python.org Mon Mar 10 09:19:48 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 10 Mar 2014 09:19:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_Now_that_it_has_a_shiny_ne?= =?utf-8?q?w_shebang_line=2C_made_add-to-pydotorg=2Epy_executable=2E?= Message-ID: <3fjC5w2LX6z7LjV@mail.python.org> http://hg.python.org/release/rev/39a3e4c740ed changeset: 85:39a3e4c740ed user: Larry Hastings date: Mon Mar 10 01:19:53 2014 -0700 summary: Now that it has a shiny new shebang line, made add-to-pydotorg.py executable. files: add-to-pydotorg.py | 0 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py old mode 100644 new mode 100755 -- Repository URL: http://hg.python.org/release From solipsis at pitrou.net Mon Mar 10 10:04:50 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 10 Mar 2014 10:04:50 +0100 Subject: [Python-checkins] Daily reference leaks (2bda1065e931): sum=0 Message-ID: results for 2bda1065e931 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogLDShFw', '-x'] From python-checkins at python.org Mon Mar 10 11:06:52 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 10 Mar 2014 11:06:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_tracemalloc=3A_filter=5Ftr?= =?utf-8?q?aces=28=29_raises_a_TypeError_if_filters_is_not_an_iterable?= Message-ID: <3fjFTS1y2pz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/d083b6ead156 changeset: 89556:d083b6ead156 user: Victor Stinner date: Mon Mar 10 11:05:07 2014 +0100 summary: tracemalloc: filter_traces() raises a TypeError if filters is not an iterable files: Lib/test/test_tracemalloc.py | 2 ++ Lib/tracemalloc.py | 5 ++++- 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -346,6 +346,8 @@ self.assertIsNot(snapshot5.traces, snapshot.traces) self.assertEqual(snapshot5.traces, snapshot.traces) + self.assertRaises(TypeError, snapshot.filter_traces, filter1) + def test_snapshot_group_by_line(self): snapshot, snapshot2 = create_snapshots() tb_0 = traceback_lineno('', 0) diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py --- a/Lib/tracemalloc.py +++ b/Lib/tracemalloc.py @@ -1,4 +1,4 @@ -from collections import Sequence +from collections import Sequence, Iterable from functools import total_ordering import fnmatch import linecache @@ -382,6 +382,9 @@ is a list of Filter instances. If filters is an empty list, return a new Snapshot instance with a copy of the traces. """ + if not isinstance(filters, Iterable): + raise TypeError("filters must be a list of filters, not %s" + % type(filters).__name__) if filters: include_filters = [] exclude_filters = [] -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Mon Mar 10 14:04:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:04:19 +1000 Subject: [Python-checkins] cpython: whatsnew: venv with_pip, pyvenv --without-pip (#19552) In-Reply-To: <3fhyT2594sz7Lm1@mail.python.org> References: <3fhyT2594sz7Lm1@mail.python.org> Message-ID: On 10 Mar 2014 08:51, "r.david.murray" wrote: > > http://hg.python.org/cpython/rev/a140caad76bc > changeset: 89539:a140caad76bc > user: R David Murray > date: Sun Mar 09 18:09:51 2014 -0400 > summary: > whatsnew: venv with_pip, pyvenv --without-pip (#19552) > > files: > Doc/whatsnew/3.4.rst | 11 +++++++++++ > 1 files changed, 11 insertions(+), 0 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -1507,6 +1507,12 @@ > :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` > shells (Contributed by Andrew Svetlov in :issue:`15417`.) > > +:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function > +take a new keyword argument *with_pip*, which defaults to ``False``, that > +controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is > +installed in the virtual environment. (Contributed by Nick Coghlan in > +:issue:`19552` as part of the :pep:`453` implementation.) > + > > wave > ---- > @@ -1772,6 +1778,11 @@ > to use copies rather than symlinks even on systems where symlinks are the > default. (Contributed by Vinay Sajip in :issue:`18807`.) > > +* The :ref:`pyvenv ` command also accepts a ``--without-pip`` > + option to suppress the otherwise-automatic bootstrapping of pip into > + the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` > + as part of the :pep:`453` implementation.) It just occurred to me that we should probably have a porting note about that CLI change. We decided that changing the default CLI behaviour, while preserving the default module API was the best available compromise, but anyone that is invoking the CLI programmatically and wants to skip bootstrapping pip will still need to change their command. Cheers, Nick. > + > * The encoding name is now optional in the value set for the > :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to > set just the error handler, without changing the default encoding. > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 10 14:09:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:09:53 +1000 Subject: [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: <3fj2742w8Tz7Lm2@mail.python.org> References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: On 10 Mar 2014 11:36, "r.david.murray" wrote: > > http://hg.python.org/cpython/rev/2d5544afb510 > changeset: 89547:2d5544afb510 > user: R David Murray > date: Sun Mar 09 20:33:35 2014 -0400 > summary: > whatsnew: 'U' mode deprecation (#15204). > > files: > Doc/whatsnew/3.4.rst | 14 +++++++++++++- > 1 files changed, 13 insertions(+), 1 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -1948,6 +1948,11 @@ > * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by > ``EXT_SUFFIX``. > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > + In Python3 it does not do anything useful, and should be replaced by > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* > + argument. > + Huh, I missed that change. Don't we still need U when writing 2/3 compatible code at this point? If so, perhaps we may want to reconsider this particular deprecation in 3.4.1. Cheers, Nick. > > Deprecations in the C API > ------------------------- > @@ -2242,7 +2247,14 @@ > > * Since the *digestmod* argument to the :func:`hmac.new` function will in the > future have no default, all calls to :func:`hmac.new` should be changed to > - explicitly specify a *digestmod*. (:issue:`17276`). > + explicitly specify a *digestmod* (:issue:`17276`). > + > +* Any calls to ``open`` functions that specify ``U`` should be modified. > + ``U`` is ineffective in Python3 and will eventually raise an error if used. > + Depending on the function, the equivalent of its old Python2 behavior can be > + achieved using either a *newline* argument, or if necessary by wrapping the > + stream in :mod:`~io.TextIOWrapper` to use its *newline* argument > + (:issue:`15204`). > > > Changes in the C API > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Mar 10 14:18:03 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 09:18:03 -0400 Subject: [Python-checkins] [Python-Dev] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: <20140310131803.C68BC250CBE@webabinitio.net> On Mon, 10 Mar 2014 23:09:53 +1000, Nick Coghlan wrote: > On 10 Mar 2014 11:36, "r.david.murray" wrote: > > > > http://hg.python.org/cpython/rev/2d5544afb510 > > changeset: 89547:2d5544afb510 > > user: R David Murray > > date: Sun Mar 09 20:33:35 2014 -0400 > > summary: > > whatsnew: 'U' mode deprecation (#15204). > > > > files: > > Doc/whatsnew/3.4.rst | 14 +++++++++++++- > > 1 files changed, 13 insertions(+), 1 deletions(-) > > > > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > > --- a/Doc/whatsnew/3.4.rst > > +++ b/Doc/whatsnew/3.4.rst > > @@ -1948,6 +1948,11 @@ > > * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by > > ``EXT_SUFFIX``. > > > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > > + In Python3 it does not do anything useful, and should be replaced by > > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its > *newline* > > + argument. > > + > > Huh, I missed that change. Don't we still need U when writing 2/3 > compatible code at this point? If so, perhaps we may want to reconsider > this particular deprecation in 3.4.1. It's a deprecation, not a removal. I believe the idea is to not remove it before 4.0. --David From python-checkins at python.org Mon Mar 10 14:21:04 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 14:21:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_add_porting_no?= =?utf-8?q?te_for_pyvenv_--without-pip=2E?= Message-ID: <3fjKnX6N7yz7Llh@mail.python.org> http://hg.python.org/cpython/rev/74961f17cc43 changeset: 89557:74961f17cc43 user: R David Murray date: Mon Mar 10 09:20:37 2014 -0400 summary: whatsnew: add porting note for pyvenv --without-pip. files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2279,6 +2279,10 @@ stream in :mod:`~io.TextIOWrapper` to use its *newline* argument (:issue:`15204`). +* If you use :ref:`pyvenv ` in a script and desire that pip + *not* be installed, you must add ``--wihtout-pip`` to your command + invocation. + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Mar 10 14:25:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 10 Mar 2014 14:25:21 +0100 Subject: [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: <20140310142521.69c3f957@fsol> On Mon, 10 Mar 2014 23:09:53 +1000 Nick Coghlan wrote: > On 10 Mar 2014 11:36, "r.david.murray" wrote: > > > > http://hg.python.org/cpython/rev/2d5544afb510 > > changeset: 89547:2d5544afb510 > > user: R David Murray > > date: Sun Mar 09 20:33:35 2014 -0400 > > summary: > > whatsnew: 'U' mode deprecation (#15204). > > > > files: > > Doc/whatsnew/3.4.rst | 14 +++++++++++++- > > 1 files changed, 13 insertions(+), 1 deletions(-) > > > > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > > --- a/Doc/whatsnew/3.4.rst > > +++ b/Doc/whatsnew/3.4.rst > > @@ -1948,6 +1948,11 @@ > > * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by > > ``EXT_SUFFIX``. > > > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > > + In Python3 it does not do anything useful, and should be replaced by > > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its > *newline* > > + argument. > > + > > Huh, I missed that change. Don't we still need U when writing 2/3 > compatible code at this point? Define "need" :-) I rarely if ever see "U" used in 2.x code, so I don't think the need is very pressing in general. Regards Antoine. From ncoghlan at gmail.com Mon Mar 10 14:36:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:36:19 +1000 Subject: [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: <20140310142521.69c3f957@fsol> References: <3fj2742w8Tz7Lm2@mail.python.org> <20140310142521.69c3f957@fsol> Message-ID: On 10 Mar 2014 23:26, "Antoine Pitrou" wrote: > > On Mon, 10 Mar 2014 23:09:53 +1000 > Nick Coghlan wrote: > > On 10 Mar 2014 11:36, "r.david.murray" wrote: > > > > > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > > > + In Python3 it does not do anything useful, and should be replaced by > > > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its > > *newline* > > > + argument. > > > + > > > > Huh, I missed that change. Don't we still need U when writing 2/3 > > compatible code at this point? > > Define "need" :-) I rarely if ever see "U" used in 2.x code, so I don't > think the need is very pressing in general. True, normal text mode is much more common. I'm OK with waiting to see if anyone actually complains about the deprecation :) Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Mon Mar 10 19:19:50 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 19:19:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Closes_=2315605=3A_update?= =?utf-8?q?_and_refurbish_doc_building_section_of_the_devguide=2E?= Message-ID: <3fjSQG33v7z7LlF@mail.python.org> http://hg.python.org/devguide/rev/f32569ddc2db changeset: 672:f32569ddc2db user: Georg Brandl date: Mon Mar 10 19:19:07 2014 +0100 summary: Closes #15605: update and refurbish doc building section of the devguide. Adapts suggestions by Daniel and Terry, and handles the new situation in 3.4+. files: documenting.rst | 72 ++++++++++++++---------------------- 1 files changed, 29 insertions(+), 43 deletions(-) diff --git a/documenting.rst b/documenting.rst --- a/documenting.rst +++ b/documenting.rst @@ -1442,28 +1442,37 @@ Building the documentation ========================== -You need to have Python 2.4 or higher installed. The toolset used to build -the docs is written in Python and is called Sphinx_. Sphinx is maintained -separately and is not included in this tree. Also needed are docutils_, -supplying the base markup that Sphinx uses; Jinja_, a templating engine; and -optionally Pygments_, a code highlighter. +The toolset used to build the docs is written in Python and is called Sphinx_. +Sphinx is maintained separately and is not included in this tree. Also needed +are docutils_, supplying the base markup that Sphinx uses; Jinja_, a templating +engine; and optionally Pygments_, a code highlighter. To build the documentation, follow the instructions from one of the sections below. You can view the documentation after building the HTML by pointing a browser at the file :file:`Doc/build/html/index.html`. +In the Python 2.7 and 3.3 branches, the Sphinx toolchain will be checked out +using Subversion from ``svn.python.org`` by the Makefile. This toolchain will +need an installed Python 2 to run. -Using make ----------- +In the Python 3.4 and later branches, you are expected to have installed a +recent version of Sphinx on your system, so that the Makefile can find the +``sphinx-build`` command. -On Unix, if you have Subversion installed, run the following from the root of -your :ref:`repository clone `:: + +Using make / make.bat +--------------------- + +On Unix, run the following from the root of your :ref:`repository clone +`:: cd Doc make html -or alternatively ``make -C Doc html``. This checks out the needed toolset -in the :file:`Doc/tools/` directory and builds the output as HTML. +or alternatively ``make -C Doc html``. This builds the output as HTML. + +For Windows users there is a :file:`make.bat` batchfile that tries to work like +``make`` does. Available :command:`make` targets are: @@ -1494,47 +1503,24 @@ * "pydoc-topics", which builds a Python module containing a dictionary with plain text documentation for the labels defined in - :file:`tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and - keyword help. + :file:`Doc/tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic + and keyword help. -A "make update" updates the Subversion checkouts in :file:`tools/`. (These -Subversion checkouts are ignored by the main Mercurial repository.) + * "suspicious", which checks the parsed markup for text that looks like + malformed and thus unconverted reST. Without make ------------ -You'll need to install the Sphinx package, either by checking it out via :: +Install the Sphinx package and its dependencies from PyPI. - svn co http://svn.python.org/projects/external/Sphinx-1.0.7/sphinx tools/sphinx +Then, from the ``Docs`` directory, run :: -or by installing it from PyPI. + sphinx-build -b . build/ -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b . build/ - -where `` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). +where ```` is one of html, text, latex, or htmlhelp (for explanations +see the make targets above). .. _docutils: http://docutils.sourceforge.net/ .. _Jinja: http://jinja.pocoo.org/ -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon Mar 10 19:29:50 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 19:29:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMDA5?= =?utf-8?q?0=3A_update_Doc/README_and_the_error_message_in_sphinx-build=2E?= =?utf-8?q?py_to?= Message-ID: <3fjSdp5QK9z7Lkh@mail.python.org> http://hg.python.org/cpython/rev/0ae254c11921 changeset: 89558:0ae254c11921 branch: 2.7 parent: 89526:16c5d7c289c6 user: Georg Brandl date: Mon Mar 10 19:26:57 2014 +0100 summary: Closes #20090: update Doc/README and the error message in sphinx-build.py to make it clear that any Python 3.x is not usable with the checked out Sphinx. files: Doc/README.txt | 44 +++++++------------------- Doc/tools/sphinx-build.py | 8 ++-- 2 files changed, 17 insertions(+), 35 deletions(-) diff --git a/Doc/README.txt b/Doc/README.txt --- a/Doc/README.txt +++ b/Doc/README.txt @@ -7,14 +7,13 @@ Documentation on the authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the -documentation. There's also a chapter intended to point out differences to -those familiar with the previous docs written in LaTeX. +documentation. Building the docs ================= -You need to have Python 2.4 or higher installed; the toolset used to build the +You need to have Python 2 installed; the toolset used to build the docs is written in Python. It is called *Sphinx*, it is not included in this tree, but maintained separately. Also needed are the docutils, supplying the base markup that Sphinx uses, Jinja, a templating engine, and optionally @@ -33,6 +32,9 @@ HTML output files. To view the generated HTML, point your favorite browser at the top-level index `build/html/index.html` after running "make". +On Windows, we try to emulate the Makefile as closely as possible with a +``make.bat`` file. + Available make targets are: * "html", which builds standalone HTML files for offline viewing. @@ -65,43 +67,23 @@ `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and keyword help. + * "suspicious", which checks the parsed markup for text that looks like + malformed and thus unconverted reST. + A "make update" updates the Subversion checkouts in `tools/`. Without make ------------ -You'll need to install the Sphinx package, either by checking it out via :: +Install the Sphinx package and its dependencies from PyPI. - svn co http://svn.python.org/projects/external/Sphinx-0.6.7/sphinx tools/sphinx +Then, from the ``Docs`` directory, run :: -or by installing it from PyPI. + sphinx-build -b . build/ -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b . build/ - -where `` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). +where ```` is one of html, text, latex, or htmlhelp (for explanations +see the make targets above). Contributing diff --git a/Doc/tools/sphinx-build.py b/Doc/tools/sphinx-build.py --- a/Doc/tools/sphinx-build.py +++ b/Doc/tools/sphinx-build.py @@ -15,13 +15,13 @@ if __name__ == '__main__': - if sys.version_info[:3] < (2, 4, 0): - print >>sys.stderr, """\ -Error: Sphinx needs to be executed with Python 2.4 or newer + if sys.version_info[:3] < (2, 4, 0) or sys.version_info[:3] > (3, 0, 0): + sys.stderr.write("""\ +Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.x though). (If you run this from the Makefile, you can set the PYTHON variable to the path of an alternative interpreter executable, e.g., ``make html PYTHON=python2.5``). -""" +""") sys.exit(1) from sphinx import main -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 19:32:09 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 19:32:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_unused_sphinx-build?= =?utf-8?q?=2Epy_entry_script=2E?= Message-ID: <3fjShT0hBzz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/9765cb577bcb changeset: 89559:9765cb577bcb parent: 89557:74961f17cc43 user: Georg Brandl date: Mon Mar 10 19:10:29 2014 +0100 summary: Remove unused sphinx-build.py entry script. files: Doc/tools/sphinx-build.py | 28 --------------------------- 1 files changed, 0 insertions(+), 28 deletions(-) diff --git a/Doc/tools/sphinx-build.py b/Doc/tools/sphinx-build.py deleted file mode 100644 --- a/Doc/tools/sphinx-build.py +++ /dev/null @@ -1,28 +0,0 @@ -# -*- coding: utf-8 -*- -""" - Sphinx - Python documentation toolchain - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - :copyright: 2007-2010 by Georg Brandl. - :license: Python license. -""" - -import sys -import warnings - -# Get rid of UserWarnings reported by pkg_resources. -warnings.filterwarnings('ignore', category=UserWarning, module='jinja2') - -if __name__ == '__main__': - - if sys.version_info[:3] < (2, 4, 0) or sys.version_info[:3] > (3, 0, 0): - sys.stderr.write("""\ -Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.0 though). -(If you run this from the Makefile, you can set the PYTHON variable -to the path of an alternative interpreter executable, e.g., -``make html PYTHON=python2.5``). -""") - sys.exit(1) - - from sphinx import main - sys.exit(main(sys.argv)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 19:32:10 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 19:32:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQ2xvc2VzICMyMDA5?= =?utf-8?q?0=3A_update_Doc/README_and_the_error_message_in_sphinx-build=2E?= =?utf-8?q?py_to?= Message-ID: <3fjShV52ycz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/d975f50de5aa changeset: 89560:d975f50de5aa branch: 3.3 parent: 89540:f9cb5a44879c user: Georg Brandl date: Mon Mar 10 19:26:57 2014 +0100 summary: Closes #20090: update Doc/README and the error message in sphinx-build.py to make it clear that any Python 3.x is not usable with the checked out Sphinx. files: Doc/README.txt | 44 +++++++------------------- Doc/tools/sphinx-build.py | 2 +- 2 files changed, 14 insertions(+), 32 deletions(-) diff --git a/Doc/README.txt b/Doc/README.txt --- a/Doc/README.txt +++ b/Doc/README.txt @@ -7,14 +7,13 @@ Documentation on the authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the -documentation. There's also a chapter intended to point out differences to -those familiar with the previous docs written in LaTeX. +documentation. Building the docs ================= -You need to have Python 2.4 or higher installed; the toolset used to build the +You need to have Python 2 installed; the toolset used to build the docs is written in Python. It is called *Sphinx*, it is not included in this tree, but maintained separately. Also needed are the docutils, supplying the base markup that Sphinx uses, Jinja, a templating engine, and optionally @@ -33,6 +32,9 @@ HTML output files. To view the generated HTML, point your favorite browser at the top-level index `build/html/index.html` after running "make". +On Windows, we try to emulate the Makefile as closely as possible with a +``make.bat`` file. + To use a Python interpreter that's not called ``python``, use the standard way to set Makefile variables, using e.g. :: @@ -73,43 +75,23 @@ `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and keyword help. + * "suspicious", which checks the parsed markup for text that looks like + malformed and thus unconverted reST. + A "make update" updates the Subversion checkouts in `tools/`. Without make ------------ -You'll need to install the Sphinx package, either by checking it out via :: +Install the Sphinx package and its dependencies from PyPI. - svn co http://svn.python.org/projects/external/Sphinx-1.0.7/sphinx tools/sphinx +Then, from the ``Docs`` directory, run :: -or by installing it from PyPI. + sphinx-build -b . build/ -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b . build/ - -where `` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). +where ```` is one of html, text, latex, or htmlhelp (for explanations +see the make targets above). Contributing diff --git a/Doc/tools/sphinx-build.py b/Doc/tools/sphinx-build.py --- a/Doc/tools/sphinx-build.py +++ b/Doc/tools/sphinx-build.py @@ -17,7 +17,7 @@ if sys.version_info[:3] < (2, 4, 0) or sys.version_info[:3] > (3, 0, 0): sys.stderr.write("""\ -Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.0 though). +Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.x though). (If you run this from the Makefile, you can set the PYTHON variable to the path of an alternative interpreter executable, e.g., ``make html PYTHON=python2.5``). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 19:32:13 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 10 Mar 2014 19:32:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2320090=3A_update_Doc/README=2Etxt_for_the_chang?= =?utf-8?q?es_in_the_3=2E4_branch=2E?= Message-ID: <3fjShY1TNbz7LlM@mail.python.org> http://hg.python.org/cpython/rev/43820a4fcaba changeset: 89561:43820a4fcaba parent: 89559:9765cb577bcb parent: 89560:d975f50de5aa user: Georg Brandl date: Mon Mar 10 19:31:52 2014 +0100 summary: Closes #20090: update Doc/README.txt for the changes in the 3.4 branch. files: Doc/README.txt | 62 +++++++++++-------------------------- 1 files changed, 19 insertions(+), 43 deletions(-) diff --git a/Doc/README.txt b/Doc/README.txt --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,36 +3,34 @@ This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at http://docs.python.org/download/. +available at . Documentation on the authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the -developers guide (http://docs.python.org/devguide/documenting.html). -There's also a chapter intended to point out differences to -those familiar with the previous docs written in LaTeX. +developers guide . Building the docs ================= -You need to have Python 2.4 or higher installed; the toolset used to build the -docs is written in Python. It is called *Sphinx*, it is not included in this -tree, but maintained separately. Also needed are the docutils, supplying the -base markup that Sphinx uses, Jinja, a templating engine, and optionally -Pygments, a code highlighter. +You need to have Sphinx installed; it is the toolset +used to build the docs. It is not included in this tree, but maintained +separately and available from PyPI . Using make ---------- -Luckily, a Makefile has been prepared so that on Unix, provided you have -installed Python and Subversion, you can just run :: +A Makefile has been prepared so that on Unix, provided you have installed +Sphinx, you can just run :: make html -to check out the necessary toolset in the `tools/` subdirectory and build the -HTML output files. To view the generated HTML, point your favorite browser at -the top-level index `build/html/index.html` after running "make". +to build the HTML output files. To view the generated HTML, point your favorite +browser at the top-level index `build/html/index.html` after running "make". + +On Windows, we try to emulate the Makefile as closely as possible with a +``make.bat`` file. To use a Python interpreter that's not called ``python``, use the standard way to set Makefile variables, using e.g. :: @@ -74,43 +72,21 @@ `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and keyword help. -A "make update" updates the Subversion checkouts in `tools/`. + * "suspicious", which checks the parsed markup for text that looks like + malformed and thus unconverted reST. Without make ------------ -You'll need to install the Sphinx package, either by checking it out via :: +Install the Sphinx package and its dependencies from PyPI. - svn co http://svn.python.org/projects/external/Sphinx-1.0.7/sphinx tools/sphinx +Then, from the ``Docs`` directory, run :: -or by installing it from PyPI. + sphinx-build -b . build/ -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b . build/ - -where `` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). +where ```` is one of html, text, latex, or htmlhelp (for explanations +see the make targets above). Contributing -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:28 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_ChainMap=2Enew?= =?utf-8?q?=5Fchild_*m*_argument=2E?= Message-ID: <3fjYYX60qLz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/b11738a62669 changeset: 89562:b11738a62669 user: R David Murray date: Mon Mar 10 09:49:19 2014 -0400 summary: whatsnew: ChainMap.new_child *m* argument. And NEWS entry. files: Doc/whatsnew/3.4.rst | 9 +++++++++ Misc/NEWS | 3 +++ 2 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -596,6 +596,15 @@ project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) +collections +----------- + +The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying +the child map to add to the chain. This allows an existing mapping and/or a +custom mapping type to be used for the child. (Contributed by Vinay Sajip in +:issue:`16613`.) + + colorsys -------- diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2376,6 +2376,9 @@ - Issue #14850: Now a charmap decoder treats U+FFFE as "undefined mapping" in any mapping, not only in a string. +- Issue #16613: Add *m* argument to ``collections.Chainmap.new_child`` to + allow the new child map to be specified explicitly. + - Issue #16730: importlib.machinery.FileFinder now no longers raises an exception when trying to populate its cache and it finds out the directory is unreadable or has turned into a file. Reported and diagnosed by -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:30 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_Message=2Eset?= =?utf-8?q?=5Fparam_*replace*=2C_Policy=2Econtent=5Fmanager=2E?= Message-ID: <3fjYYZ1c5zz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/4a3f74b3758c changeset: 89563:4a3f74b3758c user: R David Murray date: Mon Mar 10 10:08:05 2014 -0400 summary: whatsnew: Message.set_param *replace*, Policy.content_manager. Also cleaned up the email section and fixed some markup bugs in the email docs. files: Doc/library/email-examples.rst | 6 +- Doc/library/email.contentmanager.rst | 19 ++++-- Doc/library/email.message.rst | 4 +- Doc/library/email.policy.rst | 2 +- Doc/whatsnew/3.4.rst | 43 ++++++++------- 5 files changed, 42 insertions(+), 32 deletions(-) diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -40,8 +40,10 @@ .. literalinclude:: ../includes/email-alternative.py -Examples using the Provision API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _email-contentmanager-api-examples: + +Examples using the Provisional API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is a reworking of the last example using the provisional API. To make things a bit more interesting, we include a related image in the html part, and we save a copy of what we are going to send to disk, as well as sending it. diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -54,6 +54,7 @@ documented in this module because of the provisional nature of the code, the implementation lives in the :mod:`email.message` module. +.. currentmodule:: email.message .. class:: EmailMessage(policy=default) @@ -235,6 +236,16 @@ all other headers intact and in their original order. +.. class:: MIMEPart(policy=default) + + This class represents a subpart of a MIME message. It is identical to + :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are + added when :meth:`~EmailMessage.set_content` is called, since sub-parts do + not need their own :mailheader:`MIME-Version` headers. + + +.. currentmodule:: email.contentmanager + .. class:: ContentManager() Base class for content managers. Provides the standard registry mechanisms @@ -305,14 +316,6 @@ values of *typekey*, see :meth:`set_content`. -.. class:: MIMEPart(policy=default) - - This class represents a subpart of a MIME message. It is identical to - :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are - added when :meth:`~EmailMessage.set_content` is called, since sub-parts do - not need their own :mailheader:`MIME-Version` headers. - - Content Manager Instances ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -466,7 +466,7 @@ to ``False``. - .. method:: set_param(param, value, header='Content-Type', requote=True, + .. method:: set_param(param, value, header='Content-Type', requote=True, \ charset=None, language='', replace=False) Set a parameter in the :mailheader:`Content-Type` header. If the @@ -488,7 +488,7 @@ end of the list of headers. If *replace* is ``True``, the header will be updated in place. - .. versionchanged: 3.4 ``replace`` keyword was added. + .. versionchanged:: 3.4 ``replace`` keyword was added. .. method:: del_param(param, header='content-type', requote=True) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -419,7 +419,7 @@ additional arguments. By default ``content_manager`` is set to :data:`~email.contentmanager.raw_data_manager`. - .. versionadded 3.4 + .. versionadded:: 3.4 The class provides the following concrete implementations of the abstract diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -731,7 +731,8 @@ override the default policy of the message when generating a string representation of it. This means that ``as_string`` can now be used in more circumstances, instead of having to create and use a :mod:`~email.generator` in -order to pass formatting parameters to its ``flatten`` method. +order to pass formatting parameters to its ``flatten`` method. (Contributed by +R. David Murray in :issue:`18600`.) New method :meth:`~email.message.Message.as_bytes` added to produce a bytes representation of the message in a fashion similar to how ``as_string`` @@ -739,26 +740,30 @@ argument, but does accept the *unixfrom* and *policy* arguments. The :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive -result: a bytes object containing the fully formatted message. - -(Contributed by R. David Murray in :issue:`18600`.) +result: a bytes object containing the fully formatted message. (Contributed +by R. David Murray in :issue:`18600`.) + +The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. +When specified, the associated header will be updated without changing +its location in the list of headers. For backward compatibility, the default +is ``False``. (Contributed by R. David Murray in :issue:`18891`.) + .. _whatsnew_email_contentmanager: -A pair of new subclasses of :class:`~email.message.Message` have been added, -along with a new sub-module, :mod:`~email.contentmanager`. All documentation -is currently in the new module, which is being added as part of the new -:term:`provisional ` email API. These classes provide a -number of new methods that make extracting content from and inserting content -into email messages much easier. See the :mod:`~email.contentmanager` -documentation for details. - -These API additions complete the bulk of the work that was planned as part of -the email6 project. The currently provisional API is scheduled to become final -in Python 3.5 (possibly with a few minor additions in the area of error -handling). - -(Contributed by R. David Murray in :issue:`18891`.) +A pair of new subclasses of :class:`~email.message.Message` have been added +(:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, +:mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute +:attr:`~email.policy.EmailPolicy.content_manager`. All documentation is +currently in the new module, which is being added as part of email's new +:term:`provisional API`. These classes provide a number of new methods that +make extracting content from and inserting content into email messages much +easier. For details, see the :mod:`~email.contentmanager` documentation and +the :ref:`email-contentmanager-api-examples`. These API additions complete the +bulk of the work that was planned as part of the email6 project. The currently +provisional API is scheduled to become final in Python 3.5 (possibly with a few +minor additions in the area of error handling). (Contributed by R. David +Murray in :issue:`18891`.) filecmp @@ -1084,7 +1089,7 @@ --- :mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and -:keyword:`yield from` in a more useful fashion. This is especially helpful when +``yield from`` in a more useful fashion. This is especially helpful when debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and Xavier de Gaye in :issue:`16596`.) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:31 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_filecmp=2EDEFA?= =?utf-8?q?ULT=5FIGNORES_=28=2315442=29?= Message-ID: <3fjYYb3FkLz7LkY@mail.python.org> http://hg.python.org/cpython/rev/2b1d5caf63ca changeset: 89564:2b1d5caf63ca user: R David Murray date: Mon Mar 10 10:47:56 2014 -0400 summary: whatsnew: filecmp.DEFAULT_IGNORES (#15442) files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -776,6 +776,11 @@ than the resolution of a particular filesystem's file modification time field. (Contributed by Mark Levitt in :issue:`18149`.) +New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of +directories that are used as the default value for the *ignore* parameter of +the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in +:issue:`15442`.) + functools --------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:32 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_total=5Forderi?= =?utf-8?q?ng_supports_NotImplemented_=28=2310042=29?= Message-ID: <3fjYYc6Wfmz7LkK@mail.python.org> http://hg.python.org/cpython/rev/1cc413874631 changeset: 89565:1cc413874631 user: R David Murray date: Mon Mar 10 11:02:45 2014 -0400 summary: whatsnew: total_ordering supports NotImplemented (#10042) files: Doc/whatsnew/3.4.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -811,6 +811,10 @@ available for other implementations to use. (Contributed by Brian Thorne in :issue:`12428`.) +:func:`~functools.total_ordering` now supports a return value of +:const:`NotImplemented` from the underlying comparison function. (Contributed +by Katie Miller in :issue:`10042`.) + gc -- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:34 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_logging=2Efile?= =?utf-8?q?Config_accepts_ConfigParser_instances=2E_=28=2316110=29?= Message-ID: <3fjYYf1JrWz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/113341605247 changeset: 89566:113341605247 user: R David Murray date: Mon Mar 10 11:51:19 2014 -0400 summary: whatsnew: logging.fileConfig accepts ConfigParser instances. (#16110) And missing NEWS entry. files: Doc/whatsnew/3.4.rst | 8 ++++++++ Misc/NEWS | 3 +++ 2 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1006,6 +1006,14 @@ setting *port* to ``None``). (Contributed by Vinay Sajip in commit ce46195b56a9.) +:func:`~logging.config.fileConfig` now accepts a +:class:`configparser.RawConfigParser` subclass instance for the *fname* +parameter. This facilitates using a configuration file when logging +configuration is just a part of the overall application configuration, or where +the application modifies the configuration before passing it to +:func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in +:issue:`16110`.) + .. _whatsnew-marshal-3: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3617,6 +3617,9 @@ - Issue #16169: Fix ctypes.WinError()'s confusion between errno and winerror. +- Issue #16110: logging.fileConfig now accepts a pre-initialised ConfigParser + instance. + - Issue #1492704: shutil.copyfile() raises a distinct SameFileError now if source and destination are the same file. Patch by Atsuo Ishimoto. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:35 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_logging=2Econf?= =?utf-8?q?ig=2Elisten_*verify*_=28=2315452=29=2E?= Message-ID: <3fjYYg3FGSz7LkD@mail.python.org> http://hg.python.org/cpython/rev/fe1804387687 changeset: 89567:fe1804387687 user: R David Murray date: Mon Mar 10 12:03:01 2014 -0400 summary: whatsnew: logging.config.listen *verify* (#15452). files: Doc/whatsnew/3.4.rst | 5 +++++ Misc/NEWS | 4 ++++ 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1014,6 +1014,11 @@ :func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in :issue:`16110`.) +Logging configuration data received from a socket via the +:func:`logging.config.listen` function can now be validated before being +processed by supplying a verification function as the argument to the new +*verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) + .. _whatsnew-marshal-3: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3638,6 +3638,10 @@ - Issue #9650: List commonly used format codes in time.strftime and time.strptime docsttings. Original patch by Mike Hoy. +- Issue #15452: logging configuration socket listener now has a verify option + that allows an application to apply a verification function to the + received configuration data before it is acted upon. + - Issue #16034: Fix performance regressions in the new `bz2.BZ2File` implementation. Initial patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:36 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_remove_redunda?= =?utf-8?q?nt_entry=2E?= Message-ID: <3fjYYh5Q7Zz7LkM@mail.python.org> http://hg.python.org/cpython/rev/69e589e5eb1a changeset: 89568:69e589e5eb1a user: R David Murray date: Mon Mar 10 13:21:09 2014 -0400 summary: whatsnew: remove redundant entry. files: Doc/whatsnew/3.4.rst | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1781,12 +1781,6 @@ object allocator have been silenced. (Contributed by Dhiru Kholia in :issue:`18596`.) -* New function :c:func:`Py_SetStandardStreamEncoding` allows an application - that is embedding Python to do the equivalent of setting - :envvar:`PYTHONIOENCODING`. Its arguments override the equivalent - values from :envvar:`PYTHONIOENCODING` if it exists. (Contributed - by Bastien Montagne and Nick Coghlan in :issue:`16129`.) - * The Windows build now uses `Address Space Layout Randomization `_ and `Data Execution Prevention `_. (Contributed by -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:38 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_multiprocessin?= =?utf-8?q?g_start_methods_and_context_=28=238713_and_=2318999=29?= Message-ID: <3fjYYk44f4z7LkB@mail.python.org> http://hg.python.org/cpython/rev/b941a320601a changeset: 89569:b941a320601a user: R David Murray date: Mon Mar 10 14:27:21 2014 -0400 summary: whatsnew: multiprocessing start methods and context (#8713 and #18999) Also tweaked the docs a bit to use our standard style for versionadded/changed. (I'm guessing there are other places in the multiprocessing docs where similar tweaks should be made.) files: Doc/library/multiprocessing.rst | 35 ++++++++++++-------- Doc/whatsnew/3.4.rst | 24 +++++++++++--- Misc/NEWS | 7 ++++ 3 files changed, 47 insertions(+), 19 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -136,9 +136,11 @@ Available on Unix platforms which support passing file descriptors over Unix pipes. -Before Python 3.4 *fork* was the only option available on Unix. Also, -prior to Python 3.4, child processes would inherit all the parents -inheritable handles on Windows. +.. versionchanged:: 3.4 + *span* added on all unix platforms, and *forkserver* added for + some unix platforms. + Child processes no longer inherit all of the parents inheritable + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named @@ -1853,25 +1855,30 @@ callbacks and has a parallel map implementation. *processes* is the number of worker processes to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. If - *initializer* is not ``None`` then each worker process will call + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + *context* can be used to specify the context used for starting + the worker processes. Usually a pool is created using the + function :func:`multiprocessing.Pool` or the :meth:`Pool` method + of a context object. In both cases *context* is set + appropriately. + Note that the methods of the pool object should only be called by the process which created the pool. .. versionadded:: 3.2 - *maxtasksperchild* is the number of tasks a worker process can complete - before it will exit and be replaced with a fresh worker process, to enable - unused resources to be freed. The default *maxtasksperchild* is None, which - means worker processes will live as long as the pool. + *maxtasksperchild* .. versionadded:: 3.4 - *context* can be used to specify the context used for starting - the worker processes. Usually a pool is created using the - function :func:`multiprocessing.Pool` or the :meth:`Pool` method - of a context object. In both cases *context* is set - appropriately. + *context* .. note:: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1063,11 +1063,25 @@ (``spawn`` and ``forkserver``) have been added for starting processes using :mod:`multiprocessing`. These make the mixing of processes with threads more robust, and the ``spawn`` method matches the semantics that multiprocessing has -always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). - -Also, except when using the old *fork* start method, child processes -will no longer inherit unneeded handles/file descriptors from their parents -(part of :issue:`8713`). +always used on Windows. New function +:func:`~multiprocessing.get_all_start_methods` reports all start methods +available on the platform, :func:`~multiprocessing.get_start_method` reports +the current start method, and :func:`~multiprocessing.set_start_method` sets +the start method. (Contributed by Richard Oudkerk in :issue:`8713`). + +:mod:`multiprocessing` also now has the concept of a ``context``, which +determines how child processes are created. New function +:func:`~multiprocessing.get_context` returns a context that uses a specified +start method. It has the same API as the :mod:`multiprocessing` module itself, +so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other +objects that will operate within that context. This allows a framework and an +application or different parts of the same application to use multiprocessing +without interfering with each other. (Contributed by Richard Oudkerk in +:issue:`18999`.) + +Except when using the old *fork* start method, child processes no longer +inherit unneeded handles/file descriptors from their parents (part of +:issue:`8713`). :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1636,6 +1636,9 @@ - Issue #18281: Unused stat constants removed from `tarfile`. +- Issue #18999: Multiprocessing now supports 'contexts' with the same API + as the module, but bound to specified start methods. + - Issue #18468: The re.split, re.findall, and re.sub functions and the group() and groups() methods of match object now always return a string or a bytes object. @@ -2051,6 +2054,10 @@ - Issue #18532: Change the builtin hash algorithms' names to lower case names as promised by hashlib's documentation. +- Issue #8713: add new spwan and forkserver start methods, and new functions + get_all_start_methods, get_start_method, and set_start_method, to + multiprocessing. + - Issue #18405: Improve the entropy of crypt.mksalt(). - Issue #12015: The tempfile module now uses a suffix of 8 random characters -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:39 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_PEP_424_implem?= =?utf-8?q?entation=2E?= Message-ID: <3fjYYl65mhz7LkK@mail.python.org> http://hg.python.org/cpython/rev/fada398b5d83 changeset: 89570:fada398b5d83 user: R David Murray date: Mon Mar 10 15:00:33 2014 -0400 summary: whatsnew: PEP 424 implementation. files: Doc/c-api/object.rst | 6 +++--- Doc/library/operator.rst | 4 ++-- Doc/whatsnew/3.4.rst | 13 +++++++++++++ Misc/NEWS | 3 +++ 4 files changed, 21 insertions(+), 5 deletions(-) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -357,9 +357,9 @@ .. c:function:: Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t default) - Return an estimated length for the object *o*. First trying to return its - actual length, then an estimate using ``__length_hint__``, and finally - returning the default value. On error ``-1`` is returned. This is the + Return an estimated length for the object *o*. First try to return its + actual length, then an estimate using :meth:`~object.__length_hint__`, and + finally return the default value. On error return ``-1``. This is the equivalent to the Python expression ``operator.length_hint(o, default)``. .. versionadded:: 3.4 diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -240,9 +240,9 @@ .. function:: length_hint(obj, default=0) - Return an estimated length for the object *o*. First trying to return its + Return an estimated length for the object *o*. First try to return its actual length, then an estimate using :meth:`object.__length_hint__`, and - finally returning the default value. + finally return the default value. .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -393,6 +393,10 @@ several cases as a result of the introduction of Argument Clinic and other changes to the :mod:`inspect` and :mod:`pydoc` modules. +* :meth:`~object.__length_hint__` is now part of the formal language + specification (see :pep:`424`). (Contributed by Armin Ronacher in + :issue:`16148`.) + New Modules =========== @@ -1098,6 +1102,11 @@ reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in :issue:`16694`.) +New function :func:`~operator.length_hint` provides an implementation of the +specification for how the :meth:`~object.__length_hint__` special method should +be used, as part of the :pep:`424` formal specification of this language +feature. (Contributed by Armin Ronacher in :issue:`16148`.) + os -- @@ -1800,6 +1809,10 @@ `_. (Contributed by Christian Heimes in :issue:`16632`.) +* New function :c:func:`PyObject_LengthHint` is the C API equivalent + of :func:`operator.length_hint`. (Contributed by Armin Ronacher in + :issue:`16148`.) + .. _other-improvements-3.4: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2538,6 +2538,9 @@ - Issue #16160: Subclass support now works for types.SimpleNamespace. +- Issue #16148: Implement PEP 424, adding operator.length_hint and + PyObject_LengthHint. + - Upgrade Unicode data (UCD) to version 6.2. - Issue #15379: Fix passing of non-BMP characters as integers for the charmap -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:41 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_json_dump-with?= =?utf-8?q?-indent_whitespace_change_=28=2316333=29=2E?= Message-ID: <3fjYYn0r83z7Ljv@mail.python.org> http://hg.python.org/cpython/rev/bb43e8e05a7c changeset: 89571:bb43e8e05a7c user: R David Murray date: Mon Mar 10 15:41:28 2014 -0400 summary: whatsnew: json dump-with-indent whitespace change (#16333). files: Doc/whatsnew/3.4.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2340,6 +2340,12 @@ *not* be installed, you must add ``--wihtout-pip`` to your command invocation. +* The default behavior of :func:`json.dump` and :func:`json.dumps` when + an indent is specified has changed: it no longer produces trailing + spaces after the item separating commas at the ends of lines. This + will matter only if you have tests that are doing white-space-sensitive + comparisons of such output (:issue:`16333`). + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:42 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_os=2Eopen_O=5F?= =?utf-8?b?VE1QRklMRSAoIzE4NjczKS4=?= Message-ID: <3fjYYp2jjQz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/0525f7268067 changeset: 89572:0525f7268067 user: R David Murray date: Mon Mar 10 15:41:59 2014 -0400 summary: whatsnew: os.open O_TMPFILE (#18673). files: Doc/whatsnew/3.4.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1129,6 +1129,11 @@ :func:`os.path.ismount` now recognizes volumes mounted below a drive root on Windows. (Contributed by Tim Golden in :issue:`9035`.) +:func:`os.open` supports a new flag, :data:`~os.O_TMPFILE` (unnamed temporary +file) on platforms that provide it (as of 3.4.0 release, that means Linux +systems with a kernel version of 3.11 or newer that have uapi headers). +(Contributed by Christian Heimes in :issue:`18673`.) + pdb --- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:43 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_os=2Eopen_O=5F?= =?utf-8?q?PATH_and_O=5FTMPFILE=2E?= Message-ID: <3fjYYq4rRBz7LkG@mail.python.org> http://hg.python.org/cpython/rev/6c1af13faa11 changeset: 89573:6c1af13faa11 user: R David Murray date: Mon Mar 10 15:48:22 2014 -0400 summary: whatsnew: os.open O_PATH and O_TMPFILE. files: Doc/library/os.rst | 5 +++-- Doc/whatsnew/3.4.rst | 9 +++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -940,8 +940,9 @@ the C library. .. versionchanged:: 3.4 - Add :data:`O_TMPFILE` constant. It's only available on Linux Kernel 3.11 - or newer. + Add :data:`O_PATH` on systems that support it. + Add :data:`O_TMPFILE`, only available on Linux Kernel 3.11 + or newer. .. function:: openpty() diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1129,10 +1129,11 @@ :func:`os.path.ismount` now recognizes volumes mounted below a drive root on Windows. (Contributed by Tim Golden in :issue:`9035`.) -:func:`os.open` supports a new flag, :data:`~os.O_TMPFILE` (unnamed temporary -file) on platforms that provide it (as of 3.4.0 release, that means Linux -systems with a kernel version of 3.11 or newer that have uapi headers). -(Contributed by Christian Heimes in :issue:`18673`.) +:func:`os.open` supports two new flags on platforms that provide them, +:data:`~os.O_TMPFILE` (unnamed temporary file, as of 3.4.0 release available +only on Linux systems with a kernel version of 3.11 or newer that have uapi +headers), and :data:`~os.O_PATH` (un-opened file descriptor). (Contributed by +Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) pdb -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:44 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_shutil_copyfil?= =?utf-8?q?e_SameFileError_=28=231492704=29?= Message-ID: <3fjYYr6Vt3z7Ljl@mail.python.org> http://hg.python.org/cpython/rev/276227a93f6f changeset: 89574:276227a93f6f user: R David Murray date: Mon Mar 10 16:12:10 2014 -0400 summary: whatsnew: shutil copyfile SameFileError (#1492704) files: Doc/whatsnew/3.4.rst | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1297,6 +1297,16 @@ (Contributed by Filip Gruszczy?ski in :issue:`13896`.) +shutil +------ + +:func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, +:exc:`~shutil.SameFileError`, when the source and destination are the same +file, which allows an application to take appropriate action on this specific +error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in +:issue:`1492704`.) + + smtpd ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:46 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_urllib_Request?= =?utf-8?q?_objects_are_now_reusable=2E?= Message-ID: <3fjYYt1jqxz7LkS@mail.python.org> http://hg.python.org/cpython/rev/e6d862886e5c changeset: 89575:e6d862886e5c user: R David Murray date: Mon Mar 10 17:06:00 2014 -0400 summary: whatsnew: urllib Request objects are now reusable. #16464, #17485, #17272. files: Doc/whatsnew/3.4.rst | 13 +++++++++++++ Misc/NEWS | 5 ++++- 2 files changed, 17 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1562,6 +1562,19 @@ class attribute on the subclass. (Contributed by Jason R Coombs in :issue:`18978`.) +:class:`~urllib.request.Request` objects are now reusable: if the +:attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` +attributes are modified, all relevant internal properties are updated. This +means, for example, that it is now possible to use the same +:class:`~urllib.request.Request` object in more than one +:meth:`.OpenerDirector.open` call with different *data* arguments, or to +modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it +from scratch. There is also a new +:meth:`~urllib.request.Request.remove_header` method that can be used to remove +headers from a :class:`~urllib.request.Request`. (Contributed by Alexey +Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht +and Senthil Kumaran in :issue:`17272`.) + unittest -------- diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3053,7 +3053,7 @@ to procedurally generate, in an easy way, small test instances. - Issue #17485: Also delete the Request Content-Length header if the data - attribute is deleted. (Follow on to issue 16464). + attribute is deleted. (Follow on to issue Issue #16464). - Issue #15927: CVS now correctly parses escaped newlines and carriage when parsing with quoting turned off. @@ -3477,6 +3477,9 @@ list() calls aren't added to filter(), map(), and zip() which are directly passed enumerate(). +- Issue #16464: Reset the Content-Length header when a urllib Request is reused + with new data. + - Issue #12848: The pure Python pickle implementation now treats object lengths as unsigned 32-bit integers, like the C implementation does. Patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:11:47 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:11:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_deprecation_of?= =?utf-8?q?_ElementTree_XMLParser_*html*_and_iterparse_*parser*=2E?= Message-ID: <3fjYYv4VWVz7Lkg@mail.python.org> http://hg.python.org/cpython/rev/31e6adf5bfba changeset: 89576:31e6adf5bfba user: R David Murray date: Mon Mar 10 18:09:54 2014 -0400 summary: whatsnew: deprecation of ElementTree XMLParser *html* and iterparse *parser*. #17741 files: Doc/library/xml.etree.elementtree.rst | 3 ++- Doc/whatsnew/3.4.rst | 5 +++++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -952,7 +952,8 @@ specified in the XML file. .. deprecated:: 3.4 - The *html* argument. + The *html* argument. The remaining arguments should be passed via + keywword to prepare for the removal of the *html* argument. .. method:: close() diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2057,6 +2057,11 @@ appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* argument. +* The *parser* argument of :func:`~xml.etree.ElementTree.iterparse` has + been deprecated, as has the *html* argument of + :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the + latter, all arguments to ``XMLParser`` should be passed by keyword. + Deprecations in the C API ------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 10 23:15:47 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 10 Mar 2014 23:15:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_take_credit=2E?= Message-ID: <3fjYfW6tv3z7LjY@mail.python.org> http://hg.python.org/cpython/rev/02fc4fc32eec changeset: 89577:02fc4fc32eec user: R David Murray date: Mon Mar 10 18:15:09 2014 -0400 summary: whatsnew: take credit. They aren't all my words, but most of them are :). files: Doc/whatsnew/3.4.rst | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2,8 +2,7 @@ What's New In Python 3.4 **************************** -.. :Author: Someone - (uncomment if there is a principal author) +:Author: R. David Murray .. Rules for maintenance: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 02:39:10 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 02:39:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_remove_XXXs=2E?= Message-ID: <3fjf9B5GY2z7LjS@mail.python.org> http://hg.python.org/cpython/rev/dfedeb76a20f changeset: 89578:dfedeb76a20f user: R David Murray date: Mon Mar 10 21:38:37 2014 -0400 summary: whatsnew: remove XXXs. Can't figure out what the other issue was for the importlib changes. Did find a small typo in the reload doc, though. files: Doc/library/importlib.rst | 2 +- Doc/whatsnew/3.4.rst | 8 +------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -134,7 +134,7 @@ When :func:`reload` is executed: - * Python modules' code is recompiled and the module-level code re-executed, + * Python module's code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module's dictionary by reusing the :term:`loader` which originally loaded the module. The ``init`` function of extension modules is not called a second diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2062,12 +2062,6 @@ latter, all arguments to ``XMLParser`` should be passed by keyword. -Deprecations in the C API -------------------------- - -XXX: None so far - - Deprecated Features ------------------- @@ -2205,7 +2199,7 @@ * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. Note that this restores a pre-3.3 behavior in that it means a module is - re-found when re-loaded (:issue:`19413` and XXX). + re-found when re-loaded (:issue:`19413`). * Frozen packages no longer set ``__path__`` to a list containing the package name, they now set it to an empty list. The previous behavior could cause -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 08:13:44 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 11 Mar 2014 08:13:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320888=3A_improve_?= =?utf-8?q?=22Pretty_Top=22_example_of_tracemalloc=2C_use_linecache?= Message-ID: <3fjnbD4gj5z7LjS@mail.python.org> http://hg.python.org/cpython/rev/14c1ff6a8086 changeset: 89579:14c1ff6a8086 user: Victor Stinner date: Tue Mar 11 08:12:48 2014 +0100 summary: Issue #20888: improve "Pretty Top" example of tracemalloc, use linecache files: Doc/library/tracemalloc.rst | 39 ++++++++++++++++-------- 1 files changed, 26 insertions(+), 13 deletions(-) diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -184,6 +184,7 @@ Code to display the 10 lines allocating the most memory with a pretty output, ignoring ```` and ```` files:: + import linecache import os import tracemalloc @@ -201,6 +202,9 @@ filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)) + line = linecache.getline(frame.filename, frame.lineno).strip() + if line: + print(' %s' % line) other = top_stats[limit:] if other: @@ -218,19 +222,28 @@ Example of output of the Python test suite:: - 2013-11-08 14:16:58.149320: Top 10 lines - #1: collections/__init__.py:368: 291.9 KiB - #2: Lib/doctest.py:1291: 200.2 KiB - #3: unittest/case.py:571: 160.3 KiB - #4: Lib/abc.py:133: 99.8 KiB - #5: urllib/parse.py:476: 71.8 KiB - #6: :5: 62.7 KiB - #7: Lib/base64.py:140: 59.8 KiB - #8: Lib/_weakrefset.py:37: 51.8 KiB - #9: collections/__init__.py:362: 50.6 KiB - #10: test/test_site.py:56: 48.0 KiB - 7496 other: 4161.9 KiB - Total allocated size: 5258.8 KiB + Top 10 lines + #1: Lib/base64.py:414: 419.8 KiB + _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] + #2: Lib/base64.py:306: 419.8 KiB + _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] + #3: collections/__init__.py:368: 293.6 KiB + exec(class_definition, namespace) + #4: Lib/abc.py:133: 115.2 KiB + cls = super().__new__(mcls, name, bases, namespace) + #5: unittest/case.py:574: 103.1 KiB + testMethod() + #6: Lib/linecache.py:127: 95.4 KiB + lines = fp.readlines() + #7: urllib/parse.py:476: 71.8 KiB + for a in _hexdig for b in _hexdig} + #8: :5: 62.0 KiB + #9: Lib/_weakrefset.py:37: 60.0 KiB + self.data = set() + #10: Lib/base64.py:142: 59.8 KiB + _b32tab2 = [a + b for a in _b32tab for b in _b32tab] + 6220 other: 3602.8 KiB + Total allocated size: 5303.1 KiB See :meth:`Snapshot.statistics` for more options. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Mar 11 10:13:23 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 11 Mar 2014 10:13:23 +0100 Subject: [Python-checkins] Daily reference leaks (dfedeb76a20f): sum=0 Message-ID: results for dfedeb76a20f on branch "default" -------------------------------------------- test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogTlLoH8', '-x'] From python-checkins at python.org Tue Mar 11 10:29:13 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Mar 2014 10:29:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzEzNTMwOiBwb3J0?= =?utf-8?q?_to_2=2E7_branch_=28document_what_os=2Elseek_returns=29=2E?= Message-ID: <3fjrbY57fVz7LkW@mail.python.org> http://hg.python.org/cpython/rev/168e40af4a20 changeset: 89580:168e40af4a20 branch: 2.7 parent: 89558:0ae254c11921 user: Georg Brandl date: Tue Mar 11 10:28:56 2014 +0100 summary: #13530: port to 2.7 branch (document what os.lseek returns). files: Doc/library/os.rst | 2 +- Modules/posixmodule.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -744,7 +744,7 @@ by *how*: :const:`SEEK_SET` or ``0`` to set the position relative to the beginning of the file; :const:`SEEK_CUR` or ``1`` to set it relative to the current position; :const:`SEEK_END` or ``2`` to set it relative to the end of - the file. + the file. Return the new cursor position in bytes, starting from the beginning. Availability: Unix, Windows. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -6667,7 +6667,8 @@ PyDoc_STRVAR(posix_lseek__doc__, "lseek(fd, pos, how) -> newpos\n\n\ -Set the current position of a file descriptor."); +Set the current position of a file descriptor.\n\ +Return the new cursor position in bytes, starting from the beginning."); static PyObject * posix_lseek(PyObject *self, PyObject *args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 13:17:40 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 11 Mar 2014 13:17:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_What=27s_New_in_Python_3?= =?utf-8?q?=2E4=3A_Security_improvements?= Message-ID: <3fjwKw20ZHz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/16384988a526 changeset: 89581:16384988a526 parent: 89579:14c1ff6a8086 user: Victor Stinner date: Tue Mar 11 13:17:30 2014 +0100 summary: What's New in Python 3.4: Security improvements files: Doc/library/multiprocessing.rst | 2 +- Doc/whatsnew/3.4.rst | 30 ++++++++++++++++++-- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -137,7 +137,7 @@ over Unix pipes. .. versionchanged:: 3.4 - *span* added on all unix platforms, and *forkserver* added for + *spawn* added on all unix platforms, and *forkserver* added for some unix platforms. Child processes no longer inherit all of the parents inheritable handles on Windows. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -124,8 +124,6 @@ * :ref:`Single-dispatch generic functions ` in :mod:`functools` (:pep:`443`). * New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`). -* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl` - (:issue:`16692`). * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and @@ -136,6 +134,26 @@ correct introspection of a much wider variety of callable objects * The :mod:`ipaddress` module API has been declared stable +Security improvements: + +* :ref:`Secure and interchangeable hash algorithm ` + (:pep:`456`). +* :ref:`Make newly created file descriptors non-inheritable ` + (:pep:`446`) to avoid leaking file descriptors to child processes. +* A new :func:`hashlib.pbkdf2_hmac` function provides + the `PKCS#5 password-based key derivation function 2 + `_. +* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. +* :ref:`Retrieving certificates from the Windows system cert store support + ` for :mod:`ssl`. +* :ref:`Server-side SNI (Server Name Indication) support + ` for :mod:`ssl`. +* The :class:`ssl.SSLContext` class got a :ref:`lot of improvements + `. +* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork + on Unix `: *spawn* and *forkserver* avoid + sharing data with child processes; child processes no longer inherit all of + the parents inheritable handles on Windows. CPython implementation improvements: @@ -1364,6 +1382,8 @@ Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orr? and Antoine Pitrou in :issue:`16692`) +.. _whatsnew34-sslcontext: + :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` accepts a new optional argument *cadata*, which can be used to provide PEM or DER encoded certificates directly via strings or bytes, respectively. @@ -1383,12 +1403,16 @@ list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in and :issue:`18147`.) +.. _whatsnew34-win-cert-store: + Two new windows-only functions, :func:`~ssl.enum_certificates` and :func:`~ssl.enum_crls` provide the ability to retrieve certificates, certificate information, and CRLs from the Windows cert store. (Contributed by Christian Heimes in :issue:`17134`.) -Support for server-side SNI using the new +.. _whatsnew34-sni: + +Support for server-side SNI (Server Name Indication) using the new :meth:`ssl.SSLContext.set_servername_callback` method. (Contributed by Daniel Black in :issue:`8109`.) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 17:57:21 2014 From: python-checkins at python.org (peter.moody) Date: Tue, 11 Mar 2014 17:57:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319157=3A_Include_?= =?utf-8?q?the_broadcast_address_in_the_usuable_hosts_for_IPv6?= Message-ID: <3fk2Xd1p5xz7LjR@mail.python.org> http://hg.python.org/cpython/rev/b6271cbcc762 changeset: 89582:b6271cbcc762 user: Peter Moody date: Tue Mar 11 09:55:46 2014 -0700 summary: Issue #19157: Include the broadcast address in the usuable hosts for IPv6 in ipaddress. files: Lib/ipaddress.py | 12 ++++++++++++ Misc/NEWS | 3 +++ 2 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -2155,6 +2155,18 @@ if self._prefixlen == (self._max_prefixlen - 1): self.hosts = self.__iter__ + def hosts(self): + """Generate Iterator over usable hosts in a network. + + This is like __iter__ except it doesn't return the + Subnet-Router anycast address. + + """ + network = int(self.network_address) + broadcast = int(self.broadcast_address) + for x in range(network + 1, broadcast + 1): + yield self._address_class(x) + @property def is_site_local(self): """Test if the address is reserved for site-local. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #19157: Include the broadcast address in the usuable hosts for IPv6 + in ipaddress. + - Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 19:05:22 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 19:05:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_summary_sectio?= =?utf-8?q?n_tweaks=2E?= Message-ID: <3fk4365L4Jz7LjS@mail.python.org> http://hg.python.org/cpython/rev/9ae1707d427a changeset: 89583:9ae1707d427a user: R David Murray date: Tue Mar 11 14:03:24 2014 -0400 summary: whatsnew: summary section tweaks. Removed an entry made redundant by the new security section, Moved 'marshal' to the section on features other platforms will need to support, and mention that 'help' is improved by the inspect/pydoc changes (#19561 msg213153). files: Doc/whatsnew/3.4.rst | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -100,6 +100,8 @@ that are not text encodings (multiple issues). * :ref:`A ModuleSpec Type ` for the Import System (:pep:`451`). (Affects importer authors.) +* The :mod:`marshal` format has been made :ref:`more compact and efficient + ` (:issue:`16475`). New library modules: @@ -131,7 +133,8 @@ (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME handling ` (:issue:`18891`). * The :mod:`inspect` and :mod:`pydoc` modules are now capable of - correct introspection of a much wider variety of callable objects + correct introspection of a much wider variety of callable objects, + which improves the output of the Python :func:`help` system. * The :mod:`ipaddress` module API has been declared stable Security improvements: @@ -161,11 +164,7 @@ * Leveraging :pep:`442`, :ref:`module globals are no longer set to None during finalization `, in most cases (:issue:`18214`). * :ref:`Configurable memory allocators ` (:pep:`445`). -* :ref:`Secure and interchangeable hash algorithm ` - (:pep:`456`). * :ref:`Argument Clinic ` (:pep:`436`). -* The :mod:`marshal` format has been made :ref:`more compact and efficient - ` (:issue:`16475`). Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 19:58:00 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 19:58:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_mention_that_u?= =?utf-8?q?pgrading_PIP_means_it_doesn=27t_get_unistalled=2E?= Message-ID: <3fk5Cr3JRhz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/9ec36351f684 changeset: 89584:9ec36351f684 user: R David Murray date: Tue Mar 11 14:57:30 2014 -0400 summary: whatsnew: mention that upgrading PIP means it doesn't get unistalled. #19861 msg213156. files: Doc/whatsnew/3.4.rst | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -450,9 +450,11 @@ :mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ``ensurepip`` does not -access the internet. (If the installation has Internet access, it is of course -possible to upgrade ``pip`` to a release more recent than the bundled ``pip`` -by using the bundled ``pip`` command itself once it is installed.) +access the internet. If the installation has Internet access, after +``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a +more recent release than the bundled one. (Note that such an upgraded version +of ``pip`` is considered to be a separately installed package and will not be +removed if Python is uninstalled.) The module is named *ensure*\ pip because if called when ``pip`` is already installed, it does nothing. It also has an ``--upgrade`` option that will -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:00:06 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 11 Mar 2014 20:00:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogZml4IHR5cG8gKGNs?= =?utf-8?q?oses_=2320892=29?= Message-ID: <3fk5GG4txxz7LkM@mail.python.org> http://hg.python.org/cpython/rev/2f149207d3cf changeset: 89585:2f149207d3cf branch: 3.3 parent: 89560:d975f50de5aa user: Benjamin Peterson date: Tue Mar 11 13:59:37 2014 -0500 summary: fix typo (closes #20892) files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -111,7 +111,7 @@ Projects to Consider -------------------- -The lowest level library for suppoting Python 2 & 3 simultaneously is six_. +The lowest level library for supporting Python 2 & 3 simultaneously is six_. Reading through its documentation will give you an idea of where exactly the Python language changed between versions 2 & 3 and thus what you will want the library to help you continue to support. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:00:08 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 11 Mar 2014 20:00:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_3=2E3_=28closes_=2320892=29?= Message-ID: <3fk5GJ19Sfz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/e47646fca428 changeset: 89586:e47646fca428 parent: 89584:9ec36351f684 parent: 89585:2f149207d3cf user: Benjamin Peterson date: Tue Mar 11 13:59:48 2014 -0500 summary: merge 3.3 (closes #20892) files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -111,7 +111,7 @@ Projects to Consider -------------------- -The lowest level library for suppoting Python 2 & 3 simultaneously is six_. +The lowest level library for supporting Python 2 & 3 simultaneously is six_. Reading through its documentation will give you an idea of where exactly the Python language changed between versions 2 & 3 and thus what you will want the library to help you continue to support. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:13:35 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 20:13:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_doctest_finds_?= =?utf-8?q?tests_in_extension_modules_=28=233158=29?= Message-ID: <3fk5Yq1PmVz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/8520e0ff8e36 changeset: 89587:8520e0ff8e36 user: R David Murray date: Tue Mar 11 15:10:53 2014 -0400 summary: whatsnew: doctest finds tests in extension modules (#3158) #19861 msg213159 files: Doc/whatsnew/3.4.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -745,6 +745,9 @@ shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the :mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) +:mod:`doctest` will now find doctests in extension module ``__doc__`` strings. +(Contributed by Zachary Ware in :issue:`3158`.) + email ----- @@ -2398,6 +2401,11 @@ will matter only if you have tests that are doing white-space-sensitive comparisons of such output (:issue:`16333`). +* :mod:`doctest` now looks for doctests in extension module ``__doc__`` + strings, so if your doctest test discovery includes extension modules that + have things that look like doctests in them you may see test failures you've + never seen before when running your tests (:issue:`3158`). + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:26:32 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 11 Mar 2014 20:26:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_improve_algori?= =?utf-8?q?thms=5F*_documentation?= Message-ID: <3fk5rm1CN2z7LjY@mail.python.org> http://hg.python.org/cpython/rev/abb3d29d1bff changeset: 89588:abb3d29d1bff branch: 3.3 parent: 89585:2f149207d3cf user: Benjamin Peterson date: Tue Mar 11 14:26:12 2014 -0500 summary: improve algorithms_* documentation files: Doc/library/hashlib.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -95,18 +95,18 @@ .. data:: algorithms_guaranteed - Contains the names of the hash algorithms guaranteed to be supported + A set containing the names of the hash algorithms guaranteed to be supported by this module on all platforms. .. versionadded:: 3.2 .. data:: algorithms_available - Contains the names of the hash algorithms that are available - in the running Python interpreter. These names will be recognized - when passed to :func:`new`. :attr:`algorithms_guaranteed` - will always be a subset. Duplicate algorithms with different - name formats may appear in this set (thanks to OpenSSL). + A set containing the names of the hash algorithms that are available in the + running Python interpreter. These names will be recognized when passed to + :func:`new`. :attr:`algorithms_guaranteed` will always be a subset. The + same algorithm may appear multiple times in this set under different names + (thanks to OpenSSL). .. versionadded:: 3.2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:26:33 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 11 Mar 2014 20:26:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fk5rn4KfZz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/0d230ad6b51a changeset: 89589:0d230ad6b51a parent: 89587:8520e0ff8e36 parent: 89588:abb3d29d1bff user: Benjamin Peterson date: Tue Mar 11 14:26:21 2014 -0500 summary: merge 3.3 files: Doc/library/hashlib.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -101,18 +101,18 @@ .. data:: algorithms_guaranteed - Contains the names of the hash algorithms guaranteed to be supported + A set containing the names of the hash algorithms guaranteed to be supported by this module on all platforms. .. versionadded:: 3.2 .. data:: algorithms_available - Contains the names of the hash algorithms that are available - in the running Python interpreter. These names will be recognized - when passed to :func:`new`. :attr:`algorithms_guaranteed` - will always be a subset. Duplicate algorithms with different - name formats may appear in this set (thanks to OpenSSL). + A set containing the names of the hash algorithms that are available in the + running Python interpreter. These names will be recognized when passed to + :func:`new`. :attr:`algorithms_guaranteed` will always be a subset. The + same algorithm may appear multiple times in this set under different names + (thanks to OpenSSL). .. versionadded:: 3.2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 20:28:49 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 11 Mar 2014 20:28:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_sync_pyporting?= =?utf-8?q?_howto_from_python_3_docs?= Message-ID: <3fk5vP3s9Tz7LjM@mail.python.org> http://hg.python.org/cpython/rev/3409fb4103e6 changeset: 89590:3409fb4103e6 branch: 2.7 parent: 89580:168e40af4a20 user: Benjamin Peterson date: Tue Mar 11 14:28:37 2014 -0500 summary: sync pyporting howto from python 3 docs files: Doc/howto/pyporting.rst | 614 ++++++++++++--------------- 1 files changed, 268 insertions(+), 346 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -10,238 +10,211 @@ With Python 3 being the future of Python while Python 2 is still in active use, it is good to have your project available for both major releases of - Python. This guide is meant to help you choose which strategy works best - for your project to support both Python 2 & 3 along with how to execute - that strategy. + Python. This guide is meant to help you figure out how best to support both + Python 2 & 3 simultaneously. If you are looking to port an extension module instead of pure Python code, please see :ref:`cporting-howto`. + If you would like to read one core Python developer's take on why Python 3 + came into existence, you can read Nick Coghlan's `Python 3 Q & A`_. -Choosing a Strategy -=================== + If you prefer to read a (free) book on porting a project to Python 3, + consider reading `Porting to Python 3`_ by Lennart Regebro which should cover + much of what is discussed in this HOWTO. -When a project makes the decision that it's time to support both Python 2 & 3, -a decision needs to be made as to how to go about accomplishing that goal. -The chosen strategy will depend on how large the project's existing -codebase is and how much divergence you want from your Python 2 codebase from -your Python 3 one (e.g., starting a new version with Python 3). + For help with porting, you can email the python-porting_ mailing list with + questions. -If your project is brand-new or does not have a large codebase, then you may -want to consider writing/porting :ref:`all of your code for Python 3 -and use 3to2 ` to port your code for Python 2. +The Short Version +================= -If you would prefer to maintain a codebase which is semantically **and** -syntactically compatible with Python 2 & 3 simultaneously, you can write -:ref:`use_same_source`. While this tends to lead to somewhat non-idiomatic -code, it does mean you keep a rapid development process for you, the developer. +* Decide what's the oldest version of Python 2 you want to support (if at all) +* Make sure you have a thorough test suite and use continuous integration + testing to make sure you stay compatible with the versions of Python you care + about +* If you have dependencies, check their Python 3 status using caniusepython3 + (`command-line tool `__, + `web app `__) -Finally, you do have the option of :ref:`using 2to3 ` to translate -Python 2 code into Python 3 code (with some manual help). This can take the -form of branching your code and using 2to3 to start a Python 3 branch. You can -also have users perform the translation at installation time automatically so -that you only have to maintain a Python 2 codebase. +With that done, your options are: -Regardless of which approach you choose, porting is not as hard or -time-consuming as you might initially think. You can also tackle the problem -piece-meal as a good portion of porting is simply updating your code to follow -current best practices in a Python 2/3 compatible way. +* If you are dropping Python 2 support, use 2to3_ to port to Python 3 +* If you are keeping Python 2 support, then start writing Python 2/3-compatible + code starting **TODAY** + + If you have dependencies that have not been ported, reach out to them to port + their project while working to make your code compatible with Python 3 so + you're ready when your dependencies are all ported + + If all your dependencies have been ported (or you have none), go ahead and + port to Python 3 -Universal Bits of Advice ------------------------- +* If you are creating a new project that wants to have 2/3 compatibility, + code in Python 3 and then backport to Python 2 -Regardless of what strategy you pick, there are a few things you should -consider. -One is make sure you have a robust test suite. You need to make sure everything -continues to work, just like when you support a new minor version of Python. -This means making sure your test suite is thorough and is ported properly -between Python 2 & 3. You will also most likely want to use something like tox_ -to automate testing between both a Python 2 and Python 3 VM. +Before You Begin +================ -Two, once your project has Python 3 support, make sure to add the proper -classifier on the Cheeseshop_ (PyPI_). To have your project listed as Python 3 -compatible it must have the -`Python 3 classifier `_ -(from -http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/):: +If your project is on the Cheeseshop_/PyPI_, make sure it has the proper +`trove classifiers`_ to signify what versions of Python it **currently** +supports. At minimum you should specify the major version(s), e.g. +``Programming Language :: Python :: 2`` if your project currently only supports +Python 2. It is preferrable that you be as specific as possible by listing every +major/minor version of Python that you support, e.g. if your project supports +Python 2.6 and 2.7, then you want the classifiers of:: - setup( - name='Your Library', - version='1.0', - classifiers=[ - # make sure to use :: Python *and* :: Python :: 3 so - # that pypi can list the package on the python 3 page - 'Programming Language :: Python', - 'Programming Language :: Python :: 3' - ], - packages=['yourlibrary'], - # make sure to add custom_fixers to the MANIFEST.in - include_package_data=True, - # ... - ) + Programming Language :: Python :: 2 + Programming Language :: Python :: 2.6 + Programming Language :: Python :: 2.7 +Once your project supports Python 3 you will want to go back and add the +appropriate classifiers for Python 3 as well. This is important as setting the +``Programming Language :: Python :: 3`` classifier will lead to your project +being listed under the `Python 3 Packages`_ section of PyPI. -Doing so will cause your project to show up in the -`Python 3 packages list -`_. You will know -you set the classifier properly as visiting your project page on the Cheeseshop -will show a Python 3 logo in the upper-left corner of the page. +Make sure you have a robust test suite. You need to +make sure everything continues to work, just like when you support a new +minor/feature release of Python. This means making sure your test suite is +thorough and is ported properly between Python 2 & 3 (consider using coverage_ +to measure that you have effective test coverage). You will also most likely +want to use something like tox_ to automate testing between all of your +supported versions of Python. You will also want to **port your tests first** so +that you can make sure that you detect breakage during the transition. Tests also +tend to be simpler than the code they are testing so it gives you an idea of how +easy it can be to port code. -Three, the six_ project provides a library which helps iron out differences -between Python 2 & 3. If you find there is a sticky point that is a continual -point of contention in your translation or maintenance of code, consider using -a source-compatible solution relying on six. If you have to create your own -Python 2/3 compatible solution, you can use ``sys.version_info[0] >= 3`` as a -guard. - -Four, read all the approaches. Just because some bit of advice applies to one -approach more than another doesn't mean that some advice doesn't apply to other -strategies. - -Five, drop support for older Python versions if possible. `Python 2.5`_ +Drop support for older Python versions if possible. `Python 2.5`_ introduced a lot of useful syntax and libraries which have become idiomatic in Python 3. `Python 2.6`_ introduced future statements which makes compatibility much easier if you are going from Python 2 to 3. -`Python 2.7`_ continues the trend in the stdlib. So choose the newest version +`Python 2.7`_ continues the trend in the stdlib. Choose the newest version of Python which you believe can be your minimum support version and work from there. +Target the newest version of Python 3 that you can. Beyond just the usual +bugfixes, compatibility has continued to improve between Python 2 and 3 as time +has passed. E.g. Python 3.3 added back the ``u`` prefix for +strings, making source-compatible Python code easier to write. -.. _tox: http://codespeak.net/tox/ -.. _Cheeseshop: -.. _PyPI: http://pypi.python.org/ -.. _six: http://packages.python.org/six -.. _Python 2.7: http://www.python.org/2.7.x -.. _Python 2.6: http://www.python.org/2.6.x -.. _Python 2.5: http://www.python.org/2.5.x -.. _Python 2.4: http://www.python.org/2.4.x -.. _Python 2.3: http://www.python.org/2.3.x -.. _Python 2.2: http://www.python.org/2.2.x +Writing Source-Compatible Python 2/3 Code +========================================= -.. _use_3to2: +Over the years the Python community has discovered that the easiest way to +support both Python 2 and 3 in parallel is to write Python code that works in +either version. While this might sound counter-intuitive at first, it actually +is not difficult and typically only requires following some select +(non-idiomatic) practices and using some key projects to help make bridging +between Python 2 and 3 easier. -Python 3 and 3to2 -================= +Projects to Consider +-------------------- -If you are starting a new project or your codebase is small enough, you may -want to consider writing your code for Python 3 and backporting to Python 2 -using 3to2_. Thanks to Python 3 being more strict about things than Python 2 -(e.g., bytes vs. strings), the source translation can be easier and more -straightforward than from Python 2 to 3. Plus it gives you more direct -experience developing in Python 3 which, since it is the future of Python, is a -good thing long-term. +The lowest level library for supporting Python 2 & 3 simultaneously is six_. +Reading through its documentation will give you an idea of where exactly the +Python language changed between versions 2 & 3 and thus what you will want the +library to help you continue to support. -A drawback of this approach is that 3to2 is a third-party project. This means -that the Python core developers (and thus this guide) can make no promises -about how well 3to2 works at any time. There is nothing to suggest, though, -that 3to2 is not a high-quality project. +To help automate porting your code over to using six, you can use +modernize_. This project will attempt to rewrite your code to be as modern as +possible while using six to smooth out any differences between Python 2 & 3. +If you want to write your compatible code to feel more like Python 3 there is +the future_ project. It tries to provide backports of objects from Python 3 so +that you can use them from Python 2-compatible code, e.g. replacing the +``bytes`` type from Python 2 with the one from Python 3. +It also provides a translation script like modernize (its translation code is +actually partially based on it) to help start working with a pre-existing code +base. It is also unique in that its translation script will also port Python 3 +code backwards as well as Python 2 code forwards. -.. _3to2: https://bitbucket.org/amentajo/lib3to2/overview +Tips & Tricks +------------- -.. _use_2to3: - -Python 2 and 2to3 -================= - -Included with Python since 2.6, the 2to3_ tool (and :mod:`lib2to3` module) -helps with porting Python 2 to Python 3 by performing various source -translations. This is a perfect solution for projects which wish to branch -their Python 3 code from their Python 2 codebase and maintain them as -independent codebases. You can even begin preparing to use this approach -today by writing future-compatible Python code which works cleanly in -Python 2 in conjunction with 2to3; all steps outlined below will work -with Python 2 code up to the point when the actual use of 2to3 occurs. - -Use of 2to3 as an on-demand translation step at install time is also possible, -preventing the need to maintain a separate Python 3 codebase, but this approach -does come with some drawbacks. While users will only have to pay the -translation cost once at installation, you as a developer will need to pay the -cost regularly during development. If your codebase is sufficiently large -enough then the translation step ends up acting like a compilation step, -robbing you of the rapid development process you are used to with Python. -Obviously the time required to translate a project will vary, so do an -experimental translation just to see how long it takes to evaluate whether you -prefer this approach compared to using :ref:`use_same_source` or simply keeping -a separate Python 3 codebase. - -Below are the typical steps taken by a project which uses a 2to3-based approach -to supporting Python 2 & 3. - +To help with writing source-compatible code using one of the projects mentioned +in `Projects to Consider`_, consider following the below suggestions. Some of +them are handled by the suggested projects, so if you do use one of them then +read their documentation first to see which suggestions below will taken care of +for you. Support Python 2.7 ------------------- +////////////////// As a first step, make sure that your project is compatible with `Python 2.7`_. This is just good to do as Python 2.7 is the last release of Python 2 and thus will be used for a rather long time. It also allows for use of the ``-3`` flag -to Python to help discover places in your code which 2to3 cannot handle but are -known to cause issues. +to Python to help discover places in your code where compatibility might be an +issue (the ``-3`` flag is in Python 2.6 but Python 2.7 adds more warnings). Try to Support `Python 2.6`_ and Newer Only -------------------------------------------- +/////////////////////////////////////////// While not possible for all projects, if you can support `Python 2.6`_ and newer **only**, your life will be much easier. Various future statements, stdlib additions, etc. exist only in Python 2.6 and later which greatly assist in -porting to Python 3. But if you project must keep support for `Python 2.5`_ (or -even `Python 2.4`_) then it is still possible to port to Python 3. +supporting Python 3. But if you project must keep support for `Python 2.5`_ then +it is still possible to simultaneously support Python 3. Below are the benefits you gain if you only have to support Python 2.6 and newer. Some of these options are personal choice while others are **strongly** recommended (the ones that are more for personal choice are labeled as such). If you continue to support older versions of Python then you -at least need to watch out for situations that these solutions fix. +at least need to watch out for situations that these solutions fix and handle +them appropriately (which is where library help from e.g. six_ comes in handy). ``from __future__ import print_function`` ''''''''''''''''''''''''''''''''''''''''' -This is a personal choice. 2to3 handles the translation from the print -statement to the print function rather well so this is an optional step. This -future statement does help, though, with getting used to typing -``print('Hello, World')`` instead of ``print 'Hello, World'``. +It will not only get you used to typing ``print()`` as a function instead of a +statement, but it will also give you the various benefits the function has over +the Python 2 statement (six_ provides a function if you support Python 2.5 or +older). ``from __future__ import unicode_literals`` ''''''''''''''''''''''''''''''''''''''''''' -Another personal choice. You can always mark what you want to be a (unicode) -string with a ``u`` prefix to get the same effect. But regardless of whether -you use this future statement or not, you **must** make sure you know exactly -which Python 2 strings you want to be bytes, and which are to be strings. This -means you should, **at minimum** mark all strings that are meant to be text -strings with a ``u`` prefix if you do not use this future statement. +If you choose to use this future statement then all string literals in +Python 2 will be assumed to be Unicode (as is already the case in Python 3). +If you choose not to use this future statement then you should mark all of your +text strings with a ``u`` prefix and only support Python 3.3 or newer. But you +are **strongly** advised to do one or the other (six_ provides a function in +case you don't want to use the future statement **and** you want to support +Python 3.2 or older). -Bytes literals -'''''''''''''' +Bytes/string literals +''''''''''''''''''''' -This is a **very** important one. The ability to prefix Python 2 strings that -are meant to contain bytes with a ``b`` prefix help to very clearly delineate -what is and is not a Python 3 string. When you run 2to3 on code, all Python 2 -strings become Python 3 strings **unless** they are prefixed with ``b``. +This is a **very** important one. Prefix Python 2 strings that +are meant to contain bytes with a ``b`` prefix to very clearly delineate +what is and is not a Python 3 text string (six_ provides a function to use for +Python 2.5 compatibility). + +This point cannot be stressed enough: make sure you know what all of your string +literals in Python 2 are meant to be in Python 3. Any string literal that +should be treated as bytes should have the ``b`` prefix. Any string literal +that should be Unicode/text in Python 2 should either have the ``u`` literal +(supported, but ignored, in Python 3.3 and later) or you should have +``from __future__ import unicode_literals`` at the top of the file. But the key +point is you should know how Python 3 will treat every one one of your string +literals and you should mark them as appropriate. There are some differences between byte literals in Python 2 and those in Python 3 thanks to the bytes type just being an alias to ``str`` in Python 2. -Probably the biggest "gotcha" is that indexing results in different values. In -Python 2, the value of ``b'py'[1]`` is ``'y'``, while in Python 3 it's ``121``. -You can avoid this disparity by always slicing at the size of a single element: -``b'py'[1:2]`` is ``'y'`` in Python 2 and ``b'y'`` in Python 3 (i.e., close -enough). +See the `Handle Common "Gotchas"`_ section for what to watch out for. -You cannot concatenate bytes and strings in Python 3. But since Python -2 has bytes aliased to ``str``, it will succeed: ``b'a' + u'b'`` works in -Python 2, but ``b'a' + 'b'`` in Python 3 is a :exc:`TypeError`. A similar issue -also comes about when doing comparisons between bytes and strings. +``from __future__ import absolute_import`` +'''''''''''''''''''''''''''''''''''''''''' +Discussed in more detail below, but you should use this future statement to +prevent yourself from accidentally using implicit relative imports. Supporting `Python 2.5`_ and Newer Only ---------------------------------------- +/////////////////////////////////////// If you are supporting `Python 2.5`_ and newer there are still some features of Python that you can utilize. @@ -251,7 +224,7 @@ '''''''''''''''''''''''''''''''''''''''''' Implicit relative imports (e.g., importing ``spam.bacon`` from within -``spam.eggs`` with the statement ``import bacon``) does not work in Python 3. +``spam.eggs`` with the statement ``import bacon``) do not work in Python 3. This future statement moves away from that and allows the use of explicit relative imports (e.g., ``from . import bacon``). @@ -261,16 +234,74 @@ the statement, but you still want the __future__ statement to prevent implicit relative imports. In `Python 2.7`_ the __future__ statement is not needed. In other words, unless you are only supporting Python 2.7 or a version earlier -than Python 2.5, use the __future__ statement. +than Python 2.5, use this __future__ statement. +Mark all Unicode strings with a ``u`` prefix +''''''''''''''''''''''''''''''''''''''''''''' + +While Python 2.6 has a ``__future__`` statement to automatically cause Python 2 +to treat all string literals as Unicode, Python 2.5 does not have that shortcut. +This means you should go through and mark all string literals with a ``u`` +prefix to turn them explicitly into text strings where appropriate and only +support Python 3.3 or newer. Otherwise use a project like six_ which provides a +function to pass all text string literals through. + + +Capturing the Currently Raised Exception +'''''''''''''''''''''''''''''''''''''''' + +In Python 2.5 and earlier the syntax to access the current exception is:: + + try: + raise Exception() + except Exception, exc: + # Current exception is 'exc'. + pass + +This syntax changed in Python 3 (and backported to `Python 2.6`_ and later) +to:: + + try: + raise Exception() + except Exception as exc: + # Current exception is 'exc'. + # In Python 3, 'exc' is restricted to the block; in Python 2.6/2.7 it will "leak". + pass + +Because of this syntax change you must change how you capture the current +exception in Python 2.5 and earlier to:: + + try: + raise Exception() + except Exception: + import sys + exc = sys.exc_info()[1] + # Current exception is 'exc'. + pass + +You can get more information about the raised exception from +:func:`sys.exc_info` than simply the current exception instance, but you most +likely don't need it. + +.. note:: + In Python 3, the traceback is attached to the exception instance + through the ``__traceback__`` attribute. If the instance is saved in + a local variable that persists outside of the ``except`` block, the + traceback will create a reference cycle with the current frame and its + dictionary of local variables. This will delay reclaiming dead + resources until the next cyclic :term:`garbage collection` pass. + + In Python 2, this problem only occurs if you save the traceback itself + (e.g. the third element of the tuple returned by :func:`sys.exc_info`) + in a variable. + Handle Common "Gotchas" ------------------------ +/////////////////////// -There are a few things that just consistently come up as sticking points for -people which 2to3 cannot handle automatically or can easily be done in Python 2 -to help modernize your code. +These are things to watch out for no matter what version of Python 2 you are +supporting which are not syntactic considerations. ``from __future__ import division`` @@ -327,9 +358,9 @@ the bytes/string dichotomy. Because Python 2 allowed the ``str`` type to hold textual data, people have over the years been rather loose in their delineation of what ``str`` instances held text compared to bytes. In Python 3 you cannot -be so care-free anymore and need to properly handle the difference. The key +be so care-free anymore and need to properly handle the difference. The key to handling this issue is to make sure that **every** string literal in your -Python 2 code is either syntactically of functionally marked as either bytes or +Python 2 code is either syntactically or functionally marked as either bytes or text data. After this is done you then need to make sure your APIs are designed to either handle a specific type or made to be properly polymorphic. @@ -436,14 +467,7 @@ happen to use the ``unicode(self).encode('utf8')`` idiom as the body of your ``__str__()`` method). -There are two ways to solve this issue. One is to use a custom 2to3 fixer. The -blog post at http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ -specifies how to do this. That will allow 2to3 to change all instances of ``def -__unicode(self): ...`` to ``def __str__(self): ...``. This does require that you -define your ``__str__()`` method in Python 2 before your ``__unicode__()`` -method. - -The other option is to use a mixin class. This allows you to only define a +You can use a mixin class to work around this. This allows you to only define a ``__unicode__()`` method for your class and let the mixin derive ``__str__()`` for you (code from http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/):: @@ -486,6 +510,7 @@ Even better is to use the documented attributes the exception provides. + Don't use ``__getslice__`` & Friends '''''''''''''''''''''''''''''''''''' @@ -497,189 +522,62 @@ Updating doctests ''''''''''''''''' -2to3_ will attempt to generate fixes for doctests that it comes across. It's -not perfect, though. If you wrote a monolithic set of doctests (e.g., a single -docstring containing all of your doctests), you should at least consider -breaking the doctests up into smaller pieces to make it more manageable to fix. -Otherwise it might very well be worth your time and effort to port your tests -to :mod:`unittest`. +Don't forget to make them Python 2/3 compatible as well. If you wrote a +monolithic set of doctests (e.g., a single docstring containing all of your +doctests), you should at least consider breaking the doctests up into smaller +pieces to make it more manageable to fix. Otherwise it might very well be worth +your time and effort to port your tests to :mod:`unittest`. +Update ``map`` for imbalanced input sequences +''''''''''''''''''''''''''''''''''''''''''''' + +With Python 2, when ``map`` was given more than one input sequence it would pad +the shorter sequences with `None` values, returning a sequence as long as the +longest input sequence. + +With Python 3, if the input sequences to ``map`` are of unequal length, ``map`` +will stop at the termination of the shortest of the sequences. For full +compatibility with ``map`` from Python 2.x, wrap the sequence arguments in +:func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes +``list(map(func, itertools.zip_longest(*sequences)))``. + Eliminate ``-3`` Warnings ------------------------- When you run your application's test suite, run it using the ``-3`` flag passed to Python. This will cause various warnings to be raised during execution about -things that 2to3 cannot handle automatically (e.g., modules that have been -removed). Try to eliminate those warnings to make your code even more portable -to Python 3. +things that are semantic changes between Python 2 and 3. Try to eliminate those +warnings to make your code even more portable to Python 3. -Run 2to3 --------- +Alternative Approaches +====================== -Once you have made your Python 2 code future-compatible with Python 3, it's -time to use 2to3_ to actually port your code. +While supporting Python 2 & 3 simultaneously is typically the preferred choice +by people so that they can continue to improve code and have it work for the +most number of users, your life may be easier if you only have to support one +major version of Python going forward. +Supporting Only Python 3 Going Forward From Python 2 Code +--------------------------------------------------------- -Manually -'''''''' +If you have Python 2 code but going forward only want to improve it as Python 3 +code, then you can use 2to3_ to translate your Python 2 code to Python 3 code. +This is only recommended, though, if your current version of your project is +going into maintenance mode and you want all new features to be exclusive to +Python 3. -To manually convert source code using 2to3_, you use the ``2to3`` script that -is installed with Python 2.6 and later.:: - 2to3 +Backporting Python 3 code to Python 2 +------------------------------------- -This will cause 2to3 to write out a diff with all of the fixers applied for the -converted source code. If you would like 2to3 to go ahead and apply the changes -you can pass it the ``-w`` flag:: - - 2to3 -w - -There are other flags available to control exactly which fixers are applied, -etc. - - -During Installation -''''''''''''''''''' - -When a user installs your project for Python 3, you can have either -:mod:`distutils` or Distribute_ run 2to3_ on your behalf. -For distutils, use the following idiom:: - - try: # Python 3 - from distutils.command.build_py import build_py_2to3 as build_py - except ImportError: # Python 2 - from distutils.command.build_py import build_py - - setup(cmdclass = {'build_py': build_py}, - # ... - ) - -For Distribute:: - - setup(use_2to3=True, - # ... - ) - -This will allow you to not have to distribute a separate Python 3 version of -your project. It does require, though, that when you perform development that -you at least build your project and use the built Python 3 source for testing. - - -Verify & Test -------------- - -At this point you should (hopefully) have your project converted in such a way -that it works in Python 3. Verify it by running your unit tests and making sure -nothing has gone awry. If you miss something then figure out how to fix it in -Python 3, backport to your Python 2 code, and run your code through 2to3 again -to verify the fix transforms properly. - - -.. _2to3: http://docs.python.org/py3k/library/2to3.html -.. _Distribute: http://packages.python.org/distribute/ - - -.. _use_same_source: - -Python 2/3 Compatible Source -============================ - -While it may seem counter-intuitive, you can write Python code which is -source-compatible between Python 2 & 3. It does lead to code that is not -entirely idiomatic Python (e.g., having to extract the currently raised -exception from ``sys.exc_info()[1]``), but it can be run under Python 2 -**and** Python 3 without using 2to3_ as a translation step (although the tool -should be used to help find potential portability problems). This allows you to -continue to have a rapid development process regardless of whether you are -developing under Python 2 or Python 3. Whether this approach or using -:ref:`use_2to3` works best for you will be a per-project decision. - -To get a complete idea of what issues you will need to deal with, see the -`What's New in Python 3.0`_. Others have reorganized the data in other formats -such as http://docs.pythonsprints.com/python3_porting/py-porting.html\ . - -The following are some steps to take to try to support both Python 2 & 3 from -the same source code. - - -.. _What's New in Python 3.0: http://docs.python.org/release/3.0/whatsnew/3.0.html - - -Follow The Steps for Using 2to3_ --------------------------------- - -All of the steps outlined in how to -:ref:`port Python 2 code with 2to3 ` apply -to creating a Python 2/3 codebase. This includes trying only support Python 2.6 -or newer (the :mod:`__future__` statements work in Python 3 without issue), -eliminating warnings that are triggered by ``-3``, etc. - -You should even consider running 2to3_ over your code (without committing the -changes). This will let you know where potential pain points are within your -code so that you can fix them properly before they become an issue. - - -Use six_ --------- - -The six_ project contains many things to help you write portable Python code. -You should make sure to read its documentation from beginning to end and use -any and all features it provides. That way you will minimize any mistakes you -might make in writing cross-version code. - - -Capturing the Currently Raised Exception ----------------------------------------- - -One change between Python 2 and 3 that will require changing how you code (if -you support `Python 2.5`_ and earlier) is -accessing the currently raised exception. In Python 2.5 and earlier the syntax -to access the current exception is:: - - try: - raise Exception() - except Exception, exc: - # Current exception is 'exc' - pass - -This syntax changed in Python 3 (and backported to `Python 2.6`_ and later) -to:: - - try: - raise Exception() - except Exception as exc: - # Current exception is 'exc' - # In Python 3, 'exc' is restricted to the block; Python 2.6 will "leak" - pass - -Because of this syntax change you must change to capturing the current -exception to:: - - try: - raise Exception() - except Exception: - import sys - exc = sys.exc_info()[1] - # Current exception is 'exc' - pass - -You can get more information about the raised exception from -:func:`sys.exc_info` than simply the current exception instance, but you most -likely don't need it. - -.. note:: - In Python 3, the traceback is attached to the exception instance - through the ``__traceback__`` attribute. If the instance is saved in - a local variable that persists outside of the ``except`` block, the - traceback will create a reference cycle with the current frame and its - dictionary of local variables. This will delay reclaiming dead - resources until the next cyclic :term:`garbage collection` pass. - - In Python 2, this problem only occurs if you save the traceback itself - (e.g. the third element of the tuple returned by :func:`sys.exc_info`) - in a variable. +If you have Python 3 code and have little interest in supporting Python 2 you +can use 3to2_ to translate from Python 3 code to Python 2 code. This is only +recommended if you don't plan to heavily support Python 2 users. Otherwise +write your code for Python 3 and then backport as far back as you want. This +is typically easier than going from Python 2 to 3 as you will have worked out +any difficulties with e.g. bytes/strings, etc. Other Resources @@ -687,17 +585,41 @@ The authors of the following blog posts, wiki pages, and books deserve special thanks for making public their tips for porting Python 2 code to Python 3 (and -thus helping provide information for this document): +thus helping provide information for this document and its various revisions +over the years): +* http://wiki.python.org/moin/PortingPythonToPy3k * http://python3porting.com/ * http://docs.pythonsprints.com/python3_porting/py-porting.html * http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/ * http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html * http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ * http://lucumr.pocoo.org/2010/2/11/porting-to-python-3-a-guide/ -* http://wiki.python.org/moin/PortingPythonToPy3k +* https://wiki.ubuntu.com/Python/3 If you feel there is something missing from this document that should be added, please email the python-porting_ mailing list. + + +.. _2to3: http://docs.python.org/2/library/2to3.html +.. _3to2: https://pypi.python.org/pypi/3to2 +.. _Cheeseshop: PyPI_ +.. _coverage: https://pypi.python.org/pypi/coverage +.. _future: http://python-future.org/ +.. _modernize: https://github.com/mitsuhiko/python-modernize +.. _Porting to Python 3: http://python3porting.com/ +.. _PyPI: http://pypi.python.org/ +.. _Python 2.2: http://www.python.org/2.2.x +.. _Python 2.5: http://www.python.org/2.5.x +.. _Python 2.6: http://www.python.org/2.6.x +.. _Python 2.7: http://www.python.org/2.7.x +.. _Python 2.5: http://www.python.org/2.5.x +.. _Python 3.3: http://www.python.org/3.3.x +.. _Python 3 Packages: https://pypi.python.org/pypi?:action=browse&c=533&show=all +.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html .. _python-porting: http://mail.python.org/mailman/listinfo/python-porting +.. _six: https://pypi.python.org/pypi/six +.. _tox: https://pypi.python.org/pypi/tox +.. _trove classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 23:47:12 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 23:47:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzIwMDMwOiBkb2Mg?= =?utf-8?q?that_TestLoader=2Ediscover_returns_a_TestSuite=2E?= Message-ID: <3fkBJJ4szGz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/cc74393062f7 changeset: 89591:cc74393062f7 branch: 2.7 user: R David Murray date: Tue Mar 11 18:12:25 2014 -0400 summary: #20030: doc that TestLoader.discover returns a TestSuite. Patch by Lita Cho. files: Doc/library/unittest.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1510,11 +1510,11 @@ .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) - Find and return all test modules from the specified start directory, - recursing into subdirectories to find them. Only test files that match - *pattern* will be loaded. (Using shell style pattern matching.) Only - module names that are importable (i.e. are valid Python identifiers) will - be loaded. + Find all the test modules by recursing into subdirectories from the + specified start directory, and return a TestSuite object containing them. + Only test files that match *pattern* will be loaded. (Using shell style + pattern matching.) Only module names that are importable (i.e. are valid + Python identifiers) will be loaded. All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 23:47:14 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 23:47:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzIwMDMwOiBkb2Mg?= =?utf-8?q?that_TestLoader=2Ediscover_returns_a_TestSuite=2E?= Message-ID: <3fkBJL1TyCz7LkM@mail.python.org> http://hg.python.org/cpython/rev/71df53af61ec changeset: 89592:71df53af61ec branch: 3.3 parent: 89588:abb3d29d1bff user: R David Murray date: Tue Mar 11 18:46:00 2014 -0400 summary: #20030: doc that TestLoader.discover returns a TestSuite. Patch by Lita Cho. files: Doc/library/unittest.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1489,11 +1489,11 @@ .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) - Find and return all test modules from the specified start directory, - recursing into subdirectories to find them. Only test files that match - *pattern* will be loaded. (Using shell style pattern matching.) Only - module names that are importable (i.e. are valid Python identifiers) will - be loaded. + Find all the test modules by recursing into subdirectories from the + specified start directory, and return a TestSuite object containing them. + Only test files that match *pattern* will be loaded. (Using shell style + pattern matching.) Only module names that are importable (i.e. are valid + Python identifiers) will be loaded. All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 11 23:47:15 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 11 Mar 2014 23:47:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_=2320030=3A_doc_that_TestLoader=2Ediscover_returns?= =?utf-8?q?_a_TestSuite=2E?= Message-ID: <3fkBJM5Cc9z7Lk2@mail.python.org> http://hg.python.org/cpython/rev/ecc26a050384 changeset: 89593:ecc26a050384 parent: 89589:0d230ad6b51a parent: 89592:71df53af61ec user: R David Murray date: Tue Mar 11 18:46:37 2014 -0400 summary: Merge #20030: doc that TestLoader.discover returns a TestSuite. files: Doc/library/unittest.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1624,11 +1624,11 @@ .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) - Find and return all test modules from the specified start directory, - recursing into subdirectories to find them. Only test files that match - *pattern* will be loaded. (Using shell style pattern matching.) Only - module names that are importable (i.e. are valid Python identifiers) will - be loaded. + Find all the test modules by recursing into subdirectories from the + specified start directory, and return a TestSuite object containing them. + Only test files that match *pattern* will be loaded. (Using shell style + pattern matching.) Only module names that are importable (i.e. are valid + Python identifiers) will be loaded. All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 02:59:15 2014 From: python-checkins at python.org (r.david.murray) Date: Wed, 12 Mar 2014 02:59:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_spelling_error?= =?utf-8?q?s=2E?= Message-ID: <3fkGYv0nRkzSqJ@mail.python.org> http://hg.python.org/cpython/rev/996652f3c136 changeset: 89594:996652f3c136 user: R David Murray date: Tue Mar 11 21:58:54 2014 -0400 summary: whatsnew: spelling errors. Corrections from Jim Jewett in #19861. files: Doc/whatsnew/3.4.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1680,7 +1680,7 @@ :meth:`wave.open` now supports the context manager protocol. (Contributed by Claudiu Popa in :issue:`17616`.) -:mod:`wave` can now :ref:`write output to unssekable files +:mod:`wave` can now :ref:`write output to unseekable files `. (Contributed by David Jones, Guilherme Polo, and Serhiy Storchaka in :issue:`5202`.) @@ -2392,7 +2392,7 @@ (:issue:`15204`). * If you use :ref:`pyvenv ` in a script and desire that pip - *not* be installed, you must add ``--wihtout-pip`` to your command + *not* be installed, you must add ``--without-pip`` to your command invocation. * The default behavior of :func:`json.dump` and :func:`json.dumps` when -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 05:31:42 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 05:31:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Link_to_OpenHatch_setup_g?= =?utf-8?b?dWlkZSAoIzIwODgyKS4=?= Message-ID: <3fkKxp3spsz7LjX@mail.python.org> http://hg.python.org/devguide/rev/3ffbbbac0a4d changeset: 673:3ffbbbac0a4d user: ?ric Araujo date: Wed Mar 12 00:31:33 2014 -0400 summary: Link to OpenHatch setup guide (#20882). This page contains information useful for total newcomers to Python or free software development (e.g. what is an IRC client and how to install one). files: setup.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -7,6 +7,12 @@ available from http://www.python.org/). It also gives an overview of the directory structure of the CPython source code. +OpenHatch also has a great `setup guide`_ for Python for people who are +completely new to contributing to open source. + +.. _setup guide: https://openhatch.org/wiki/Contributing_to_Python + + .. _setup: Getting Set Up -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Wed Mar 12 07:16:53 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 07:16:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Define_what_?= =?utf-8?b?4oCcdXBkYXRpbmfigJ0gbWVhbnMgaW4gZG9jcyBmb3Igb3BlbiAoIzE5NjI3?= =?utf-8?b?KS4=?= Message-ID: <3fkNH9070kz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/1f823c1559bd changeset: 89595:1f823c1559bd branch: 2.7 parent: 89591:cc74393062f7 user: ?ric Araujo date: Wed Mar 12 02:16:37 2014 -0400 summary: Define what ?updating? means in docs for open (#19627). Original patch by Lina Clark. files: Doc/library/functions.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -880,8 +880,8 @@ to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used. [#]_ - Modes ``'r+'``, ``'w+'`` and ``'a+'`` open the file for updating (note that - ``'w+'`` truncates the file). Append ``'b'`` to the mode to open the file in + Modes ``'r+'``, ``'w+'`` and ``'a+'`` open the file for updating (reading and writing); + note that ``'w+'`` truncates the file. Append ``'b'`` to the mode to open the file in binary mode, on systems that differentiate between binary and text files; on systems that don't have this distinction, adding the ``'b'`` has no effect. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 08:15:05 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 08:15:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQXZvaWQg4oCcZXJy?= =?utf-8?q?or=3A_None=E2=80=9D_messages_from_distutils_=28=234931=29=2E?= Message-ID: <3fkPZK2sS0z7Ljl@mail.python.org> http://hg.python.org/cpython/rev/a865f6fb82b4 changeset: 89596:a865f6fb82b4 branch: 2.7 user: ?ric Araujo date: Wed Mar 12 03:14:48 2014 -0400 summary: Avoid ?error: None? messages from distutils (#4931). Thanks to Amaury Forgeot d?Arc and Philip J. Eby. files: Doc/distutils/apiref.rst | 9 ------- Lib/distutils/core.py | 7 +---- Lib/distutils/dir_util.py | 4 +-- Lib/distutils/tests/test_util.py | 12 ++++++++- Lib/distutils/util.py | 23 +++---------------- Misc/NEWS | 3 ++ 6 files changed, 20 insertions(+), 38 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1167,15 +1167,6 @@ underscore. No { } or ( ) style quoting is available. -.. function:: grok_environment_error(exc[, prefix='error: ']) - - Generate a useful error message from an :exc:`EnvironmentError` (:exc:`IOError` - or :exc:`OSError`) exception object. Handles Python 1.5.1 and later styles, - and does what it can to deal with exception objects that don't have a filename - (which happens when the error is due to a two-file operation, such as - :func:`~os.rename` or :func:`~os.link`). Returns the error message as a - string prefixed with *prefix*. - .. function:: split_quoted(s) diff --git a/Lib/distutils/core.py b/Lib/distutils/core.py --- a/Lib/distutils/core.py +++ b/Lib/distutils/core.py @@ -14,7 +14,6 @@ from distutils.debug import DEBUG from distutils.errors import (DistutilsSetupError, DistutilsArgError, DistutilsError, CCompilerError) -from distutils.util import grok_environment_error # Mainly import these so setup scripts can "from distutils.core import" them. from distutils.dist import Distribution @@ -153,13 +152,11 @@ except KeyboardInterrupt: raise SystemExit, "interrupted" except (IOError, os.error), exc: - error = grok_environment_error(exc) - if DEBUG: - sys.stderr.write(error + "\n") + sys.stderr.write("error: %s\n" % (exc,)) raise else: - raise SystemExit, error + raise SystemExit, exc except (DistutilsError, CCompilerError), msg: diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -185,7 +185,6 @@ Any errors are ignored (apart from being reported to stdout if 'verbose' is true). """ - from distutils.util import grok_environment_error global _path_created if verbose >= 1: @@ -202,8 +201,7 @@ if abspath in _path_created: del _path_created[abspath] except (IOError, OSError), exc: - log.warn(grok_environment_error( - exc, "error removing %s: " % directory)) + log.warn("error removing %s: %s", directory, exc) def ensure_relative(path): """Take the full path 'path', and make it a relative path. diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -3,8 +3,9 @@ import unittest from test.test_support import run_unittest -from distutils.errors import DistutilsPlatformError, DistutilsByteCompileError -from distutils.util import byte_compile +from distutils.errors import DistutilsByteCompileError +from distutils.util import byte_compile, grok_environment_error + class UtilTestCase(unittest.TestCase): @@ -18,6 +19,13 @@ finally: sys.dont_write_bytecode = old_dont_write_bytecode + def test_grok_environment_error(self): + # test obsolete function to ensure backward compat (#4931) + exc = IOError("Unable to find batch file") + msg = grok_environment_error(exc) + self.assertEqual(msg, "error: Unable to find batch file") + + def test_suite(): return unittest.makeSuite(UtilTestCase) diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -213,25 +213,10 @@ def grok_environment_error (exc, prefix="error: "): - """Generate a useful error message from an EnvironmentError (IOError or - OSError) exception object. Handles Python 1.5.1 and 1.5.2 styles, and - does what it can to deal with exception objects that don't have a - filename (which happens when the error is due to a two-file operation, - such as 'rename()' or 'link()'. Returns the error message as a string - prefixed with 'prefix'. - """ - # check for Python 1.5.2-style {IO,OS}Error exception objects - if hasattr(exc, 'filename') and hasattr(exc, 'strerror'): - if exc.filename: - error = prefix + "%s: %s" % (exc.filename, exc.strerror) - else: - # two-argument functions in posix module don't - # include the filename in the exception object! - error = prefix + "%s" % exc.strerror - else: - error = prefix + str(exc[-1]) - - return error + # Function kept for backward compatibility. + # Used to try clever things with EnvironmentErrors, + # but nowadays str(exception) produces good messages. + return prefix + str(exc) # Needed by 'split_quoted()' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ as documented. The pattern and source keyword parameters are left as deprecated aliases. +- Issue #4931: distutils should not produce unhelpful "error: None" messages + anymore. distutils.util.grok_environment_error is kept but doc-deprecated. + - Improve the random module's default seeding to use 256 bits of entropy from os.urandom(). This was already done for Python 3, mildly improving security with a bigger seed space. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 08:19:46 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 08:19:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Restore_missin?= =?utf-8?q?g_part_of_error_message_=28=234931=29?= Message-ID: <3fkPgk0W9xz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/0f1237b61f58 changeset: 89597:0f1237b61f58 branch: 2.7 user: ?ric Araujo date: Wed Mar 12 03:19:33 2014 -0400 summary: Restore missing part of error message (#4931) files: Lib/distutils/core.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/core.py b/Lib/distutils/core.py --- a/Lib/distutils/core.py +++ b/Lib/distutils/core.py @@ -156,7 +156,7 @@ sys.stderr.write("error: %s\n" % (exc,)) raise else: - raise SystemExit, exc + raise SystemExit, "error: %s" % (exc,) except (DistutilsError, CCompilerError), msg: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 09:11:10 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 09:11:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQXZvaWQg4oCcZXJy?= =?utf-8?q?or=3A_None=E2=80=9D_messages_from_distutils_=28=234931=29=2E?= Message-ID: <3fkQq264Kvz7LjR@mail.python.org> http://hg.python.org/cpython/rev/504eb00998f2 changeset: 89598:504eb00998f2 branch: 3.3 parent: 89592:71df53af61ec user: ?ric Araujo date: Wed Mar 12 03:34:02 2014 -0400 summary: Avoid ?error: None? messages from distutils (#4931). Thanks to Amaury Forgeot d?Arc and Philip J. Eby. files: Doc/distutils/apiref.rst | 9 ------- Lib/distutils/core.py | 7 +---- Lib/distutils/dir_util.py | 6 +--- Lib/distutils/tests/test_util.py | 10 +++++++- Lib/distutils/util.py | 23 +++---------------- Misc/NEWS | 3 ++ 6 files changed, 20 insertions(+), 38 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1171,15 +1171,6 @@ underscore. No { } or ( ) style quoting is available. -.. function:: grok_environment_error(exc[, prefix='error: ']) - - Generate a useful error message from an :exc:`OSError` exception object. - Handles Python 1.5.1 and later styles, and does what it can to deal with - exception objects that don't have a filename (which happens when the error - is due to a two-file operation, such as :func:`~os.rename` or :func:`~os.link`). - Returns the error message as a string prefixed with *prefix*. - - .. function:: split_quoted(s) Split a string up according to Unix shell-like rules for quotes and backslashes. diff --git a/Lib/distutils/core.py b/Lib/distutils/core.py --- a/Lib/distutils/core.py +++ b/Lib/distutils/core.py @@ -11,7 +11,6 @@ from distutils.debug import DEBUG from distutils.errors import * -from distutils.util import grok_environment_error # Mainly import these so setup scripts can "from distutils.core import" them. from distutils.dist import Distribution @@ -149,13 +148,11 @@ except KeyboardInterrupt: raise SystemExit("interrupted") except (IOError, os.error) as exc: - error = grok_environment_error(exc) - if DEBUG: - sys.stderr.write(error + "\n") + sys.stderr.write("error: %s\n" % (exc,)) raise else: - raise SystemExit(error) + raise SystemExit("error: %s" % (exc,)) except (DistutilsError, CCompilerError) as msg: diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -2,7 +2,7 @@ Utility functions for manipulating directories and directory trees.""" -import os, sys +import os import errno from distutils.errors import DistutilsFileError, DistutilsInternalError from distutils import log @@ -182,7 +182,6 @@ Any errors are ignored (apart from being reported to stdout if 'verbose' is true). """ - from distutils.util import grok_environment_error global _path_created if verbose >= 1: @@ -199,8 +198,7 @@ if abspath in _path_created: del _path_created[abspath] except (IOError, OSError) as exc: - log.warn(grok_environment_error( - exc, "error removing %s: " % directory)) + log.warn("error removing %s: %s", directory, exc) def ensure_relative(path): """Take the full path 'path', and make it a relative path. diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -8,7 +8,8 @@ from distutils.errors import DistutilsPlatformError, DistutilsByteCompileError from distutils.util import (get_platform, convert_path, change_root, check_environ, split_quoted, strtobool, - rfc822_escape, byte_compile) + rfc822_escape, byte_compile, + grok_environment_error) from distutils import util # used to patch _environ_checked from distutils.sysconfig import get_config_vars from distutils import sysconfig @@ -285,6 +286,13 @@ finally: sys.dont_write_bytecode = old_dont_write_bytecode + def test_grok_environment_error(self): + # test obsolete function to ensure backward compat (#4931) + exc = IOError("Unable to find batch file") + msg = grok_environment_error(exc) + self.assertEqual(msg, "error: Unable to find batch file") + + def test_suite(): return unittest.makeSuite(UtilTestCase) diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -213,25 +213,10 @@ def grok_environment_error (exc, prefix="error: "): - """Generate a useful error message from an EnvironmentError (IOError or - OSError) exception object. Handles Python 1.5.1 and 1.5.2 styles, and - does what it can to deal with exception objects that don't have a - filename (which happens when the error is due to a two-file operation, - such as 'rename()' or 'link()'. Returns the error message as a string - prefixed with 'prefix'. - """ - # check for Python 1.5.2-style {IO,OS}Error exception objects - if hasattr(exc, 'filename') and hasattr(exc, 'strerror'): - if exc.filename: - error = prefix + "%s: %s" % (exc.filename, exc.strerror) - else: - # two-argument functions in posix module don't - # include the filename in the exception object! - error = prefix + "%s" % exc.strerror - else: - error = prefix + str(exc.args[-1]) - - return error + # Function kept for backward compatibility. + # Used to try clever things with EnvironmentErrors, + # but nowadays str(exception) produces good messages. + return prefix + str(exc) # Needed by 'split_quoted()' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -16,6 +16,9 @@ - Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. +- Issue #4931: distutils should not produce unhelpful "error: None" messages + anymore. distutils.util.grok_environment_error is kept but doc-deprecated. + - Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 09:11:12 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 09:11:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zICgjNDkzMSk=?= Message-ID: <3fkQq41R92z7Ljl@mail.python.org> http://hg.python.org/cpython/rev/c7bd0f953687 changeset: 89599:c7bd0f953687 parent: 89594:996652f3c136 parent: 89598:504eb00998f2 user: ?ric Araujo date: Wed Mar 12 04:10:51 2014 -0400 summary: Merge 3.3 (#4931) files: Doc/distutils/apiref.rst | 9 ------- Lib/distutils/core.py | 7 +---- Lib/distutils/dir_util.py | 6 +--- Lib/distutils/tests/test_util.py | 10 +++++++- Lib/distutils/util.py | 23 +++---------------- Misc/NEWS | 3 ++ 6 files changed, 20 insertions(+), 38 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1160,15 +1160,6 @@ underscore. No { } or ( ) style quoting is available. -.. function:: grok_environment_error(exc[, prefix='error: ']) - - Generate a useful error message from an :exc:`OSError` exception object. - Handles Python 1.5.1 and later styles, and does what it can to deal with - exception objects that don't have a filename (which happens when the error - is due to a two-file operation, such as :func:`~os.rename` or :func:`~os.link`). - Returns the error message as a string prefixed with *prefix*. - - .. function:: split_quoted(s) Split a string up according to Unix shell-like rules for quotes and backslashes. diff --git a/Lib/distutils/core.py b/Lib/distutils/core.py --- a/Lib/distutils/core.py +++ b/Lib/distutils/core.py @@ -11,7 +11,6 @@ from distutils.debug import DEBUG from distutils.errors import * -from distutils.util import grok_environment_error # Mainly import these so setup scripts can "from distutils.core import" them. from distutils.dist import Distribution @@ -150,13 +149,11 @@ except KeyboardInterrupt: raise SystemExit("interrupted") except OSError as exc: - error = grok_environment_error(exc) - if DEBUG: - sys.stderr.write(error + "\n") + sys.stderr.write("error: %s\n" % (exc,)) raise else: - raise SystemExit(error) + raise SystemExit("error: %s" % (exc,)) except (DistutilsError, CCompilerError) as msg: diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -2,7 +2,7 @@ Utility functions for manipulating directories and directory trees.""" -import os, sys +import os import errno from distutils.errors import DistutilsFileError, DistutilsInternalError from distutils import log @@ -182,7 +182,6 @@ Any errors are ignored (apart from being reported to stdout if 'verbose' is true). """ - from distutils.util import grok_environment_error global _path_created if verbose >= 1: @@ -199,8 +198,7 @@ if abspath in _path_created: del _path_created[abspath] except OSError as exc: - log.warn(grok_environment_error( - exc, "error removing %s: " % directory)) + log.warn("error removing %s: %s", directory, exc) def ensure_relative(path): """Take the full path 'path', and make it a relative path. diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -8,7 +8,8 @@ from distutils.errors import DistutilsPlatformError, DistutilsByteCompileError from distutils.util import (get_platform, convert_path, change_root, check_environ, split_quoted, strtobool, - rfc822_escape, byte_compile) + rfc822_escape, byte_compile, + grok_environment_error) from distutils import util # used to patch _environ_checked from distutils.sysconfig import get_config_vars from distutils import sysconfig @@ -285,6 +286,13 @@ finally: sys.dont_write_bytecode = old_dont_write_bytecode + def test_grok_environment_error(self): + # test obsolete function to ensure backward compat (#4931) + exc = IOError("Unable to find batch file") + msg = grok_environment_error(exc) + self.assertEqual(msg, "error: Unable to find batch file") + + def test_suite(): return unittest.makeSuite(UtilTestCase) diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -207,25 +207,10 @@ def grok_environment_error (exc, prefix="error: "): - """Generate a useful error message from an OSError - exception object. Handles Python 1.5.1 and 1.5.2 styles, and - does what it can to deal with exception objects that don't have a - filename (which happens when the error is due to a two-file operation, - such as 'rename()' or 'link()'. Returns the error message as a string - prefixed with 'prefix'. - """ - # check for Python 1.5.2-style {IO,OS}Error exception objects - if hasattr(exc, 'filename') and hasattr(exc, 'strerror'): - if exc.filename: - error = prefix + "%s: %s" % (exc.filename, exc.strerror) - else: - # two-argument functions in posix module don't - # include the filename in the exception object! - error = prefix + "%s" % exc.strerror - else: - error = prefix + str(exc.args[-1]) - - return error + # Function kept for backward compatibility. + # Used to try clever things with EnvironmentErrors, + # but nowadays str(exception) produces good messages. + return prefix + str(exc) # Needed by 'split_quoted()' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ - Issue #19157: Include the broadcast address in the usuable hosts for IPv6 in ipaddress. +- Issue #4931: distutils should not produce unhelpful "error: None" messages + anymore. distutils.util.grok_environment_error is kept but doc-deprecated. + - Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Mar 12 09:55:20 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 12 Mar 2014 09:55:20 +0100 Subject: [Python-checkins] Daily reference leaks (996652f3c136): sum=4 Message-ID: results for 996652f3c136 on branch "default" -------------------------------------------- test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 0, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogZe_tyR', '-x'] From python-checkins at python.org Wed Mar 12 10:04:07 2014 From: python-checkins at python.org (eric.araujo) Date: Wed, 12 Mar 2014 10:04:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_missing_im?= =?utf-8?q?port_in_bdist=5Frpm_=28=2318045=29?= Message-ID: <3fkS075gmWz7Ltt@mail.python.org> http://hg.python.org/cpython/rev/677327810121 changeset: 89600:677327810121 branch: 2.7 parent: 89597:0f1237b61f58 user: ?ric Araujo date: Wed Mar 12 05:01:25 2014 -0400 summary: Fix missing import in bdist_rpm (#18045) files: Lib/distutils/command/bdist_rpm.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -12,6 +12,7 @@ from distutils.core import Command from distutils.debug import DEBUG from distutils.file_util import write_file +from distutils.sysconfig import get_python_version from distutils.errors import (DistutilsOptionError, DistutilsPlatformError, DistutilsFileError, DistutilsExecError) from distutils import log -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 12:41:55 2014 From: python-checkins at python.org (victor.stinner) Date: Wed, 12 Mar 2014 12:41:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320896=3A_Workarou?= =?utf-8?q?nd_the_bug_temporarely_to_fix_buildbots?= Message-ID: <3fkWVC20L1z7Ljb@mail.python.org> http://hg.python.org/cpython/rev/c13398566409 changeset: 89601:c13398566409 parent: 89599:c7bd0f953687 user: Victor Stinner date: Wed Mar 12 12:41:44 2014 +0100 summary: Issue #20896: Workaround the bug temporarely to fix buildbots files: Lib/test/test_ssl.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1351,12 +1351,13 @@ def test_get_server_certificate(self): def _test_get_server_certificate(host, port, cert=None): with support.transient_internet(host): - pem = ssl.get_server_certificate((host, port)) + # FIXME: force PROTOCOL_SSLv23 for workaround bug #20896 + pem = ssl.get_server_certificate((host, port), ssl_version=ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) try: - pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE) + pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE, ssl_version=ssl.PROTOCOL_SSLv23) except ssl.SSLError as x: #should fail if support.verbose: @@ -1364,7 +1365,7 @@ else: self.fail("Got server certificate %s for %s:%s!" % (pem, host, port)) - pem = ssl.get_server_certificate((host, port), ca_certs=cert) + pem = ssl.get_server_certificate((host, port), ca_certs=cert, ssl_version=ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 19:14:33 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 19:14:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Backed_out_changeset_c1339?= =?utf-8?q?8566409?= Message-ID: <3fkhCF0sPPz7LjX@mail.python.org> http://hg.python.org/cpython/rev/efe527e0f802 changeset: 89602:efe527e0f802 user: Benjamin Peterson date: Wed Mar 12 13:14:19 2014 -0500 summary: Backed out changeset c13398566409 files: Lib/test/test_ssl.py | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1351,13 +1351,12 @@ def test_get_server_certificate(self): def _test_get_server_certificate(host, port, cert=None): with support.transient_internet(host): - # FIXME: force PROTOCOL_SSLv23 for workaround bug #20896 - pem = ssl.get_server_certificate((host, port), ssl_version=ssl.PROTOCOL_SSLv23) + pem = ssl.get_server_certificate((host, port)) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) try: - pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE, ssl_version=ssl.PROTOCOL_SSLv23) + pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -1365,7 +1364,7 @@ else: self.fail("Got server certificate %s for %s:%s!" % (pem, host, port)) - pem = ssl.get_server_certificate((host, port), ca_certs=cert, ssl_version=ssl.PROTOCOL_SSLv23) + pem = ssl.get_server_certificate((host, port), ca_certs=cert) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 21:07:36 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 21:07:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogdXNlIHN1cHBvcnQu?= =?utf-8?q?rmtree_instead_of_shutil_=28closes_=2319614=29?= Message-ID: <3fkkjh6zn0z7LjM@mail.python.org> http://hg.python.org/cpython/rev/a5f767bf9d1c changeset: 89603:a5f767bf9d1c branch: 3.3 parent: 89598:504eb00998f2 user: Benjamin Peterson date: Wed Mar 12 15:07:01 2014 -0500 summary: use support.rmtree instead of shutil (closes #19614) Patch by Sean Rodman. files: Lib/test/test_support.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -102,7 +102,7 @@ self.assertTrue(os.path.isdir(path)) self.assertFalse(os.path.isdir(path)) finally: - shutil.rmtree(parent_dir) + support.rmtree(parent_dir) def test_temp_dir__path_none(self): """Test passing no path.""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 21:07:38 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 21:07:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTk2MTQp?= Message-ID: <3fkkjk1jhrz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/8a77e22aff6c changeset: 89604:8a77e22aff6c parent: 89602:efe527e0f802 parent: 89603:a5f767bf9d1c user: Benjamin Peterson date: Wed Mar 12 15:07:22 2014 -0500 summary: merge 3.3 (#19614) files: Lib/test/test_support.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -102,7 +102,7 @@ self.assertTrue(os.path.isdir(path)) self.assertFalse(os.path.isdir(path)) finally: - shutil.rmtree(parent_dir) + support.rmtree(parent_dir) def test_temp_dir__path_none(self): """Test passing no path.""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 21:09:14 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 21:09:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_add_Sean_Rodma?= =?utf-8?q?n?= Message-ID: <3fkklZ4KdDz7LjN@mail.python.org> http://hg.python.org/cpython/rev/e682f87a6822 changeset: 89605:e682f87a6822 branch: 3.3 parent: 89603:a5f767bf9d1c user: Benjamin Peterson date: Wed Mar 12 15:08:43 2014 -0500 summary: add Sean Rodman files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1048,6 +1048,7 @@ Jim Robinson Mark Roddy Kevin Rodgers +Sean Rodman Giampaolo Rodola Elson Rodriguez Adi Roiban -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 21:09:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 21:09:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_Sean_Rodma?= =?utf-8?q?n?= Message-ID: <3fkklb61zkz7LjN@mail.python.org> http://hg.python.org/cpython/rev/9fe5894ef95e changeset: 89606:9fe5894ef95e branch: 2.7 parent: 89600:677327810121 user: Benjamin Peterson date: Wed Mar 12 15:08:43 2014 -0500 summary: add Sean Rodman files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -876,6 +876,7 @@ Jim Robinson Andy Robinson Kevin Rodgers +Sean Rodman Giampaolo Rodola Adi Roiban Mike Romberg -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 12 21:09:17 2014 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 12 Mar 2014 21:09:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fkkld0Fyjz7LjM@mail.python.org> http://hg.python.org/cpython/rev/fbdaca8bf3f5 changeset: 89607:fbdaca8bf3f5 parent: 89604:8a77e22aff6c parent: 89605:e682f87a6822 user: Benjamin Peterson date: Wed Mar 12 15:09:00 2014 -0500 summary: merge 3.3 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1098,6 +1098,7 @@ Jim Robinson Mark Roddy Kevin Rodgers +Sean Rodman Giampaolo Rodola Elson Rodriguez Adi Roiban -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:11:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 00:11:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4xKTogdXNlIHNzbC5QUk9U?= =?utf-8?q?OCOL=5FSSLv23_for_maximum_compatibility_=28closes_=2320896=29?= Message-ID: <3fkpnb15WRz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/23add5382fb3 changeset: 89608:23add5382fb3 branch: 3.1 parent: 89430:c25e1442529f user: Benjamin Peterson date: Wed Mar 12 18:05:53 2014 -0500 summary: use ssl.PROTOCOL_SSLv23 for maximum compatibility (closes #20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -195,12 +195,15 @@ def test_get_server_certificate(self): with support.transient_internet("svn.python.org"): - pem = ssl.get_server_certificate(("svn.python.org", 443)) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on svn.python.org:443!") try: - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=CERTFILE) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -208,7 +211,9 @@ else: self.fail("Got server certificate %s for svn.python.org!" % pem) - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=SVN_PYTHON_ORG_ROOT_CERT) if not pem: self.fail("No server certificate on svn.python.org:443!") if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:11:16 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 00:11:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogdXNlIHNzbC5QUk9U?= =?utf-8?q?OCOL=5FSSLv23_for_maximum_compatibility_=28closes_=2320896=29?= Message-ID: <3fkpnc381WzMbR@mail.python.org> http://hg.python.org/cpython/rev/789ca594960f changeset: 89609:789ca594960f branch: 3.2 parent: 89431:e82dcd700e8c user: Benjamin Peterson date: Wed Mar 12 18:05:53 2014 -0500 summary: use ssl.PROTOCOL_SSLv23 for maximum compatibility (closes #20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -690,12 +690,15 @@ def test_get_server_certificate(self): with support.transient_internet("svn.python.org"): - pem = ssl.get_server_certificate(("svn.python.org", 443)) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on svn.python.org:443!") try: - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=CERTFILE) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -703,7 +706,9 @@ else: self.fail("Got server certificate %s for svn.python.org!" % pem) - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=SVN_PYTHON_ORG_ROOT_CERT) if not pem: self.fail("No server certificate on svn.python.org:443!") if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:11:17 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 00:11:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogdXNlIHNzbC5QUk9U?= =?utf-8?q?OCOL=5FSSLv23_for_maximum_compatibility_=28closes_=2320896=29?= Message-ID: <3fkpnd546PzMbR@mail.python.org> http://hg.python.org/cpython/rev/de97d0334314 changeset: 89610:de97d0334314 branch: 2.7 parent: 89606:9fe5894ef95e user: Benjamin Peterson date: Wed Mar 12 18:05:53 2014 -0500 summary: use ssl.PROTOCOL_SSLv23 for maximum compatibility (closes #20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -390,19 +390,24 @@ def test_get_server_certificate(self): with test_support.transient_internet("svn.python.org"): - pem = ssl.get_server_certificate(("svn.python.org", 443)) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on svn.python.org:443!") try: - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=CERTFILE) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError: #should fail pass else: self.fail("Got server certificate %s for svn.python.org!" % pem) - pem = ssl.get_server_certificate(("svn.python.org", 443), ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + pem = ssl.get_server_certificate(("svn.python.org", 443), + ssl.PROTOCOL_SSLv23, + ca_certs=SVN_PYTHON_ORG_ROOT_CERT) if not pem: self.fail("No server certificate on svn.python.org:443!") if test_support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:11:18 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 00:11:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2_=28=2320896=29?= Message-ID: <3fkpnf73zFz7LkC@mail.python.org> http://hg.python.org/cpython/rev/12df02358137 changeset: 89611:12df02358137 branch: 3.3 parent: 89605:e682f87a6822 parent: 89609:789ca594960f user: Benjamin Peterson date: Wed Mar 12 18:10:47 2014 -0500 summary: merge 3.2 (#20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -964,12 +964,15 @@ def test_get_server_certificate(self): def _test_get_server_certificate(host, port, cert=None): with support.transient_internet(host): - pem = ssl.get_server_certificate((host, port)) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) try: - pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -977,7 +980,9 @@ else: self.fail("Got server certificate %s for %s:%s!" % (pem, host, port)) - pem = ssl.get_server_certificate((host, port), ca_certs=cert) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=cert) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:11:20 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 00:11:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMjA4OTYp?= Message-ID: <3fkpnh1tbbz7LjS@mail.python.org> http://hg.python.org/cpython/rev/0cba79667c7d changeset: 89612:0cba79667c7d parent: 89607:fbdaca8bf3f5 parent: 89611:12df02358137 user: Benjamin Peterson date: Wed Mar 12 18:10:57 2014 -0500 summary: merge 3.3 (#20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1351,12 +1351,15 @@ def test_get_server_certificate(self): def _test_get_server_certificate(host, port, cert=None): with support.transient_internet(host): - pem = ssl.get_server_certificate((host, port)) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) try: - pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -1364,7 +1367,9 @@ else: self.fail("Got server certificate %s for %s:%s!" % (pem, host, port)) - pem = ssl.get_server_certificate((host, port), ca_certs=cert) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=cert) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:36:18 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 00:36:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_note_marku?= =?utf-8?b?cCAoIzE2ODA1KS4=?= Message-ID: <3fkqLV4vK0zQvG@mail.python.org> http://hg.python.org/cpython/rev/ca43fa662b0d changeset: 89613:ca43fa662b0d branch: 2.7 parent: 89610:de97d0334314 user: ?ric Araujo date: Wed Mar 12 19:35:54 2014 -0400 summary: Fix note markup (#16805). Patch by Tshepang Lekhonkhobe, reviewed by Georg Brandl. files: Doc/c-api/init.rst | 1 + Doc/library/getopt.rst | 1 + Doc/library/io.rst | 1 + Doc/library/logging.config.rst | 8 ++++++-- Doc/library/os.rst | 4 +++- Doc/library/socket.rst | 1 + 6 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -534,6 +534,7 @@ .. index:: module: thread .. note:: + When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -10,6 +10,7 @@ -------------- .. note:: + The :mod:`getopt` module is a parser for command line options whose API is designed to be familiar to users of the C :c:func:`getopt` function. Users who are unfamiliar with the C :c:func:`getopt` function or who would like to write diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -644,6 +644,7 @@ :exc:`UnsupportedOperation`. .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -118,7 +118,9 @@ send it to the socket as a string of bytes preceded by a four-byte length string packed in binary using ``struct.pack('>L', n)``. - .. note:: Because portions of the configuration are passed through + .. note:: + + Because portions of the configuration are passed through :func:`eval`, use of this function may open its users to a security risk. While the function only binds to a socket on ``localhost``, and so does not accept connections from remote machines, there are scenarios where @@ -721,7 +723,9 @@ :class:`~logging.Formatter` can present exception tracebacks in an expanded or condensed format. -.. note:: Due to the use of :func:`eval` as described above, there are +.. note:: + + Due to the use of :func:`eval` as described above, there are potential security risks which result from using the :func:`listen` to send and receive configurations via sockets. The risks are limited to where multiple users with no mutual trust run code on the same machine; see the diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -157,7 +157,9 @@ Availability: Unix. - .. note:: On Mac OS X, :func:`getgroups` behavior differs somewhat from + .. note:: + + On Mac OS X, :func:`getgroups` behavior differs somewhat from other Unix platforms. If the Python interpreter was built with a deployment target of :const:`10.5` or earlier, :func:`getgroups` returns the list of effective group ids associated with the current user process; diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -562,6 +562,7 @@ automatically closed when they are garbage-collected. .. note:: + :meth:`close()` releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call :meth:`shutdown()` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:52:05 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 00:52:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_note_marku?= =?utf-8?b?cCAoIzE2ODA1KS4=?= Message-ID: <3fkqhj0q5cz7LjR@mail.python.org> http://hg.python.org/cpython/rev/9dc653f48357 changeset: 89614:9dc653f48357 branch: 3.3 parent: 89611:12df02358137 user: ?ric Araujo date: Wed Mar 12 19:51:00 2014 -0400 summary: Fix note markup (#16805). Patch by Tshepang Lekhonkhobe, reviewed by Georg Brandl. files: Doc/c-api/arg.rst | 1 + Doc/c-api/init.rst | 1 + Doc/faq/library.rst | 1 + Doc/library/fractions.rst | 8 ++++++-- Doc/library/getopt.rst | 1 + Doc/library/io.rst | 1 + Doc/library/logging.config.rst | 8 ++++++-- Doc/library/os.rst | 4 +++- Doc/library/pkgutil.rst | 2 ++ Doc/library/socket.rst | 1 + Doc/library/ssl.rst | 1 + Doc/library/sys.rst | 4 +++- Doc/library/test.rst | 1 + 13 files changed, 28 insertions(+), 6 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -45,6 +45,7 @@ Unless otherwise stated, buffers are not NUL-terminated. .. note:: + For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of the length argument (int or :c:type:`Py_ssize_t`) is controlled by defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -548,6 +548,7 @@ .. index:: module: _thread .. note:: + When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -509,6 +509,7 @@ you can also use the :mod:`array` module. .. note:: + To read and write binary data, it is mandatory to open the file in binary mode (here, passing ``"rb"`` to :func:`open`). If you use ``"r"`` instead (the default), the file will be open in text mode diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -99,7 +99,9 @@ value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`float`. @@ -108,7 +110,9 @@ This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`decimal.Decimal` instance. diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -10,6 +10,7 @@ -------------- .. note:: + The :mod:`getopt` module is a parser for command line options whose API is designed to be familiar to users of the C :c:func:`getopt` function. Users who are unfamiliar with the C :c:func:`getopt` function or who would like to write diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -681,6 +681,7 @@ :exc:`UnsupportedOperation`. .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -115,7 +115,9 @@ send it to the socket as a string of bytes preceded by a four-byte length string packed in binary using ``struct.pack('>L', n)``. - .. note:: Because portions of the configuration are passed through + .. note:: + + Because portions of the configuration are passed through :func:`eval`, use of this function may open its users to a security risk. While the function only binds to a socket on ``localhost``, and so does not accept connections from remote machines, there are scenarios where @@ -714,7 +716,9 @@ :class:`~logging.Formatter` can present exception tracebacks in an expanded or condensed format. -.. note:: Due to the use of :func:`eval` as described above, there are +.. note:: + + Due to the use of :func:`eval` as described above, there are potential security risks which result from using the :func:`listen` to send and receive configurations via sockets. The risks are limited to where multiple users with no mutual trust run code on the same machine; see the diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -260,7 +260,9 @@ Availability: Unix. - .. note:: On Mac OS X, :func:`getgroups` behavior differs somewhat from + .. note:: + + On Mac OS X, :func:`getgroups` behavior differs somewhat from other Unix platforms. If the Python interpreter was built with a deployment target of :const:`10.5` or earlier, :func:`getgroups` returns the list of effective group ids associated with the current user process; diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -146,6 +146,7 @@ *prefix* is a string to output on the front of every module name on output. .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and @@ -184,6 +185,7 @@ walk_packages(ctypes.__path__, ctypes.__name__ + '.') .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -755,6 +755,7 @@ :keyword:`with` statement around them. .. note:: + :meth:`close()` releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call :meth:`shutdown()` diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -623,6 +623,7 @@ 'version': 3} .. note:: + To validate a certificate for a particular service, you can use the :func:`match_hostname` function. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -227,7 +227,9 @@ installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* is the version number of Python, for example ``3.2``. - .. note:: If a :ref:`virtual environment ` is in effect, this + .. note:: + + If a :ref:`virtual environment ` is in effect, this value will be changed in ``site.py`` to point to the virtual environment. The value for the Python installation will still be available, via :data:`base_exec_prefix`. diff --git a/Doc/library/test.rst b/Doc/library/test.rst --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -199,6 +199,7 @@ test suite. .. note:: + :mod:`test.support` is not a public module. It is documented here to help Python developers write tests. The API of this module is subject to change without backwards compatibility concerns between releases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 00:52:06 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 00:52:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zICgjMTY4MDUp?= Message-ID: <3fkqhk49Tsz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/44e815a33ae3 changeset: 89615:44e815a33ae3 parent: 89612:0cba79667c7d parent: 89614:9dc653f48357 user: ?ric Araujo date: Wed Mar 12 19:51:50 2014 -0400 summary: Merge 3.3 (#16805) files: Doc/c-api/arg.rst | 1 + Doc/c-api/init.rst | 1 + Doc/faq/library.rst | 1 + Doc/library/fractions.rst | 8 ++++++-- Doc/library/getopt.rst | 1 + Doc/library/io.rst | 1 + Doc/library/logging.config.rst | 8 ++++++-- Doc/library/os.rst | 4 +++- Doc/library/pkgutil.rst | 2 ++ Doc/library/socket.rst | 1 + Doc/library/ssl.rst | 1 + Doc/library/sys.rst | 4 +++- Doc/library/test.rst | 1 + 13 files changed, 28 insertions(+), 6 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -45,6 +45,7 @@ Unless otherwise stated, buffers are not NUL-terminated. .. note:: + For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of the length argument (int or :c:type:`Py_ssize_t`) is controlled by defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -582,6 +582,7 @@ .. index:: module: _thread .. note:: + When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -513,6 +513,7 @@ you can also use the :mod:`array` module. .. note:: + To read and write binary data, it is mandatory to open the file in binary mode (here, passing ``"rb"`` to :func:`open`). If you use ``"r"`` instead (the default), the file will be open in text mode diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -99,7 +99,9 @@ value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`float`. @@ -108,7 +110,9 @@ This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`decimal.Decimal` instance. diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -10,6 +10,7 @@ -------------- .. note:: + The :mod:`getopt` module is a parser for command line options whose API is designed to be familiar to users of the C :c:func:`getopt` function. Users who are unfamiliar with the C :c:func:`getopt` function or who would like to write diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -686,6 +686,7 @@ :exc:`UnsupportedOperation`. .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -148,7 +148,9 @@ send it to the socket as a string of bytes preceded by a four-byte length string packed in binary using ``struct.pack('>L', n)``. - .. note:: Because portions of the configuration are passed through + .. note:: + + Because portions of the configuration are passed through :func:`eval`, use of this function may open its users to a security risk. While the function only binds to a socket on ``localhost``, and so does not accept connections from remote machines, there are scenarios where @@ -752,7 +754,9 @@ :class:`~logging.Formatter` can present exception tracebacks in an expanded or condensed format. -.. note:: Due to the use of :func:`eval` as described above, there are +.. note:: + + Due to the use of :func:`eval` as described above, there are potential security risks which result from using the :func:`listen` to send and receive configurations via sockets. The risks are limited to where multiple users with no mutual trust run code on the same machine; see the diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -260,7 +260,9 @@ Availability: Unix. - .. note:: On Mac OS X, :func:`getgroups` behavior differs somewhat from + .. note:: + + On Mac OS X, :func:`getgroups` behavior differs somewhat from other Unix platforms. If the Python interpreter was built with a deployment target of :const:`10.5` or earlier, :func:`getgroups` returns the list of effective group ids associated with the current user process; diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -147,6 +147,7 @@ *prefix* is a string to output on the front of every module name on output. .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and @@ -185,6 +186,7 @@ walk_packages(ctypes.__path__, ctypes.__name__ + '.') .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -799,6 +799,7 @@ :keyword:`with` statement around them. .. note:: + :meth:`close()` releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call :meth:`shutdown()` diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -818,6 +818,7 @@ 'version': 3} .. note:: + To validate a certificate for a particular service, you can use the :func:`match_hostname` function. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -227,7 +227,9 @@ installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* is the version number of Python, for example ``3.2``. - .. note:: If a :ref:`virtual environment ` is in effect, this + .. note:: + + If a :ref:`virtual environment ` is in effect, this value will be changed in ``site.py`` to point to the virtual environment. The value for the Python installation will still be available, via :data:`base_exec_prefix`. diff --git a/Doc/library/test.rst b/Doc/library/test.rst --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -199,6 +199,7 @@ test suite. .. note:: + :mod:`test.support` is not a public module. It is documented here to help Python developers write tests. The API of this module is subject to change without backwards compatibility concerns between releases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:20:02 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 03:20:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Make_distutils?= =?utf-8?q?_error_messages_more_helpful_=28=2311599=29=2E?= Message-ID: <3fktzQ3wJWz7LjN@mail.python.org> http://hg.python.org/cpython/rev/37bca30c3e19 changeset: 89616:37bca30c3e19 branch: 2.7 parent: 89613:ca43fa662b0d user: ?ric Araujo date: Wed Mar 12 22:19:39 2014 -0400 summary: Make distutils error messages more helpful (#11599). When running external programs such as a C compiler and getting an error code, distutils only prints the program name. With this change, one can get the full command line by setting the DISTUTILS_DEBUG environment variable. This should have no compatibility issues, unless there are tools that depend on the exact format of distutils debug messages. files: Doc/distutils/setupscript.rst | 9 ++- Doc/install/index.rst | 4 +- Lib/distutils/spawn.py | 63 ++++++++++++++++------ Misc/NEWS | 4 + 4 files changed, 58 insertions(+), 22 deletions(-) diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -684,6 +684,8 @@ DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -699,7 +701,8 @@ and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -58,7 +58,9 @@ document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. + .. _inst-trivial-install: diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -12,6 +12,7 @@ import os from distutils.errors import DistutilsPlatformError, DistutilsExecError +from distutils.debug import DEBUG from distutils import log def spawn(cmd, search_path=1, verbose=0, dry_run=0): @@ -30,6 +31,9 @@ Raise DistutilsExecError if running the program fails in any way; just return on success. """ + # cmd is documented as a list, but just in case some code passes a tuple + # in, protect our %-formatting code against horrible death + cmd = list(cmd) if os.name == 'posix': _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': @@ -69,12 +73,16 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError, exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' failed: %s" % (cmd[0], exc[-1]) + "command %r failed: %s" % (cmd, exc[-1]) if rc != 0: # and this reflects the command running but failing + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' failed with exit status %d" % (cmd[0], rc) + "command %r failed with exit status %d" % (cmd, rc) def _spawn_os2(cmd, search_path=1, verbose=0, dry_run=0): executable = cmd[0] @@ -88,13 +96,17 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError, exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' failed: %s" % (cmd[0], exc[-1]) + "command %r failed: %s" % (cmd, exc[-1]) if rc != 0: # and this reflects the command running but failing - log.debug("command '%s' failed with exit status %d" % (cmd[0], rc)) + if not DEBUG: + cmd = executable + log.debug("command %r failed with exit status %d" % (cmd, rc)) raise DistutilsExecError, \ - "command '%s' failed with exit status %d" % (cmd[0], rc) + "command %r failed with exit status %d" % (cmd, rc) if sys.platform == 'darwin': from distutils import sysconfig @@ -105,8 +117,9 @@ log.info(' '.join(cmd)) if dry_run: return + executable = cmd[0] exec_fn = search_path and os.execvp or os.execv - exec_args = [cmd[0], cmd] + env = None if sys.platform == 'darwin': global _cfg_target, _cfg_target_split if _cfg_target is None: @@ -127,18 +140,24 @@ env = dict(os.environ, MACOSX_DEPLOYMENT_TARGET=cur_target) exec_fn = search_path and os.execvpe or os.execve - exec_args.append(env) pid = os.fork() if pid == 0: # in the child try: - exec_fn(*exec_args) + if env is None: + exec_fn(executable, cmd) + else: + exec_fn(executable, cmd, env) except OSError, e: - sys.stderr.write("unable to execute %s: %s\n" % - (cmd[0], e.strerror)) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r: %s\n" % + (cmd, e.strerror)) os._exit(1) - sys.stderr.write("unable to execute %s for unknown reasons" % cmd[0]) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r for unknown reasons" % cmd) os._exit(1) else: # in the parent # Loop until the child either exits or is terminated by a signal @@ -150,29 +169,37 @@ import errno if exc.errno == errno.EINTR: continue + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' failed: %s" % (cmd[0], exc[-1]) + "command %r failed: %s" % (cmd, exc[-1]) if os.WIFSIGNALED(status): + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' terminated by signal %d" % \ - (cmd[0], os.WTERMSIG(status)) + "command %r terminated by signal %d" % \ + (cmd, os.WTERMSIG(status)) elif os.WIFEXITED(status): exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! else: + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "command '%s' failed with exit status %d" % \ - (cmd[0], exit_status) + "command %r failed with exit status %d" % \ + (cmd, exit_status) elif os.WIFSTOPPED(status): continue else: + if not DEBUG: + cmd = executable raise DistutilsExecError, \ - "unknown error executing '%s': termination status %d" % \ - (cmd[0], status) + "unknown error executing %r: termination status %d" % \ + (cmd, status) def find_executable(executable, path=None): """Tries to find 'executable' in the directories listed in 'path'. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ as documented. The pattern and source keyword parameters are left as deprecated aliases. +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + - Issue #4931: distutils should not produce unhelpful "error: None" messages anymore. distutils.util.grok_environment_error is kept but doc-deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:42:21 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:42:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_unneces?= =?utf-8?q?sary_word_=28closes_=2319060=29?= Message-ID: <3fkvT91vpBz7LjT@mail.python.org> http://hg.python.org/cpython/rev/1d31060f8a5c changeset: 89617:1d31060f8a5c branch: 2.7 user: Benjamin Peterson date: Wed Mar 12 21:41:35 2014 -0500 summary: remove unnecessary word (closes #19060) Patch by Anastasia Filatova. files: Doc/library/subprocess.rst | 2 +- Lib/subprocess.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -12,7 +12,7 @@ The :mod:`subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to -replace several other, older modules and functions, such as:: +replace several older modules and functions:: os.system os.spawn* diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -11,7 +11,7 @@ This module allows you to spawn processes, connect to their input/output/error pipes, and obtain their return codes. This module -intends to replace several other, older modules and functions, like: +intends to replace several older modules and functions: os.system os.spawn* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:42:22 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:42:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_unneces?= =?utf-8?q?sary_word_=28closes_=2319060=29?= Message-ID: <3fkvTB3kZnz7LjY@mail.python.org> http://hg.python.org/cpython/rev/4d8a9d12edfa changeset: 89618:4d8a9d12edfa branch: 3.3 parent: 89614:9dc653f48357 user: Benjamin Peterson date: Wed Mar 12 21:41:35 2014 -0500 summary: remove unnecessary word (closes #19060) Patch by Anastasia Filatova. files: Doc/library/subprocess.rst | 2 +- Lib/subprocess.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -9,7 +9,7 @@ The :mod:`subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to -replace several other, older modules and functions, such as:: +replace several older modules and functions:: os.system os.spawn* diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -11,7 +11,7 @@ This module allows you to spawn processes, connect to their input/output/error pipes, and obtain their return codes. This module -intends to replace several other, older modules and functions, like: +intends to replace several older modules and functions: os.system os.spawn* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:42:23 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:42:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTkwNjAp?= Message-ID: <3fkvTC5gBFz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/6f93ab911d5d changeset: 89619:6f93ab911d5d parent: 89615:44e815a33ae3 parent: 89618:4d8a9d12edfa user: Benjamin Peterson date: Wed Mar 12 21:42:04 2014 -0500 summary: merge 3.3 (#19060) files: Doc/library/subprocess.rst | 2 +- Lib/subprocess.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -9,7 +9,7 @@ The :mod:`subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to -replace several other, older modules and functions, such as:: +replace several older modules and functions:: os.system os.spawn* diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -11,7 +11,7 @@ This module allows you to spawn processes, connect to their input/output/error pipes, and obtain their return codes. This module -intends to replace several other, older modules and functions, like: +intends to replace several older modules and functions: os.system os.spawn* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:52:33 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:52:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_weaken_callbac?= =?utf-8?q?k_count_inequality_=28closes_=2320901=29?= Message-ID: <3fkvhx0d6kz7LjT@mail.python.org> http://hg.python.org/cpython/rev/1763e27a182d changeset: 89620:1763e27a182d branch: 2.7 parent: 89617:1d31060f8a5c user: Benjamin Peterson date: Wed Mar 12 21:51:52 2014 -0500 summary: weaken callback count inequality (closes #20901) files: Lib/sqlite3/test/hooks.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -162,7 +162,7 @@ create table bar (a, b) """) second_count = len(progress_calls) - self.assertGreater(first_count, second_count) + self.assertGreaterEqual(first_count, second_count) def CheckCancelOperation(self): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:52:34 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:52:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_weaken_callbac?= =?utf-8?q?k_count_inequality_=28closes_=2320901=29?= Message-ID: <3fkvhy236gz7LjT@mail.python.org> http://hg.python.org/cpython/rev/dbc9e3ed5e9f changeset: 89621:dbc9e3ed5e9f branch: 3.3 parent: 89618:4d8a9d12edfa user: Benjamin Peterson date: Wed Mar 12 21:51:52 2014 -0500 summary: weaken callback count inequality (closes #20901) files: Lib/sqlite3/test/hooks.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -162,7 +162,7 @@ create table bar (a, b) """) second_count = len(progress_calls) - self.assertGreater(first_count, second_count) + self.assertGreaterEqual(first_count, second_count) def CheckCancelOperation(self): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 03:52:35 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 13 Mar 2014 03:52:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMjA5MDEp?= Message-ID: <3fkvhz3SLjz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/4d626a9df062 changeset: 89622:4d626a9df062 parent: 89619:6f93ab911d5d parent: 89621:dbc9e3ed5e9f user: Benjamin Peterson date: Wed Mar 12 21:52:19 2014 -0500 summary: merge 3.3 (#20901) files: Lib/sqlite3/test/hooks.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -162,7 +162,7 @@ create table bar (a, b) """) second_count = len(progress_calls) - self.assertGreater(first_count, second_count) + self.assertGreaterEqual(first_count, second_count) def CheckCancelOperation(self): """ -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Mar 13 09:53:17 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 13 Mar 2014 09:53:17 +0100 Subject: [Python-checkins] Daily reference leaks (44e815a33ae3): sum=4 Message-ID: results for 44e815a33ae3 on branch "default" -------------------------------------------- test_site leaked [2, -2, 2] references, sum=2 test_site leaked [2, -2, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog3DbgH5', '-x'] From python-checkins at python.org Thu Mar 13 10:51:32 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 13 Mar 2014 10:51:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2320787=3A_asyncio_?= =?utf-8?q?doc=3A_fix_typo=2E_Patch_written_by_akira=2E?= Message-ID: <3fl50N2L6Kz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/7a42fb12a736 changeset: 89623:7a42fb12a736 user: Victor Stinner date: Thu Mar 13 10:50:01 2014 +0100 summary: Close #20787: asyncio doc: fix typo. Patch written by akira. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -488,7 +488,7 @@ * *stdout*: Either a file-like object representing the pipe to be connected to the subprocess's standard output stream using - :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :meth:`~BaseEventLoop.connect_read_pipe`, or the constant :const:`subprocess.PIPE` (the default). By default a new pipe will be created and connected. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 10:54:32 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 13 Mar 2014 10:54:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2320889=3A_asyncio_?= =?utf-8?q?doc=3A_Document_acquire=28=29=2C_locked=28=29_and_release=28=29?= =?utf-8?q?_method_of?= Message-ID: <3fl53r5KQ0z7LjV@mail.python.org> http://hg.python.org/cpython/rev/f22e1d89486d changeset: 89624:f22e1d89486d user: Victor Stinner date: Thu Mar 13 10:54:18 2014 +0100 summary: Close #20889: asyncio doc: Document acquire(), locked() and release() method of Condition files: Doc/library/asyncio-sync.rst | 27 +++++++++++++++++++++++- 1 files changed, 26 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -64,7 +64,7 @@ .. method:: locked() - Return ``True`` if lock is acquired. + Return ``True`` if the lock is acquired. .. method:: acquire() @@ -141,6 +141,15 @@ A new :class:`Lock` object is created and used as the underlying lock. + .. method:: acquire() + + Acquire the underlying lock. + + This method blocks until the lock is unlocked, then sets it to locked and + returns ``True``. + + This method is a :ref:`coroutine `. + .. method:: notify(n=1) By default, wake up one coroutine waiting on this condition, if any. @@ -156,6 +165,10 @@ call until it can reacquire the lock. Since :meth:`notify` does not release the lock, its caller should. + .. method:: locked() + + Return ``True`` if the underlying lock is acquired. + .. method:: notify_all() Wake up all threads waiting on this condition. This method acts like @@ -163,6 +176,18 @@ calling thread has not acquired the lock when this method is called, a :exc:`RuntimeError` is raised. + .. method:: release() + + Release the underlying lock. + + When the lock is locked, reset it to unlocked, and return. If any other + coroutines are blocked waiting for the lock to become unlocked, allow + exactly one of them to proceed. + + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. + + There is no return value. + .. method:: wait() Wait until notified. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 10:58:12 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 13 Mar 2014 10:58:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_functions_a?= =?utf-8?q?re_coroutine=2C_they_don=27t_return_a_coroutine?= Message-ID: <3fl5840pt2z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/077dce5c4196 changeset: 89625:077dce5c4196 user: Victor Stinner date: Thu Mar 13 10:58:03 2014 +0100 summary: asyncio doc: functions are coroutine, they don't return a coroutine It's not exact, but easier to understand. files: Doc/library/asyncio-protocol.rst | 2 +- Doc/library/asyncio-subprocess.rst | 4 ++-- Doc/library/asyncio-sync.rst | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -497,6 +497,6 @@ :meth:`Transport.close` can be called immediately after :meth:`WriteTransport.write` even if data are not sent yet on the socket: both methods are asynchronous. ``yield from`` is not needed because these transport -methods don't return coroutines. +methods are not coroutines. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -11,13 +11,13 @@ Run the shell command *cmd* given as a string. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. .. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) Create a subprocess. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. Use the :meth:`BaseEventLoop.connect_read_pipe` and :meth:`BaseEventLoop.connect_write_pipe` methods to connect pipes. diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -73,7 +73,7 @@ This method blocks until the lock is unlocked, then sets it to locked and returns ``True``. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: release() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 11:02:29 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 11:02:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Make_distutils?= =?utf-8?q?_error_messages_more_helpful_=28=2311599=29=2E?= Message-ID: <3fl5F139lZz7LjV@mail.python.org> http://hg.python.org/cpython/rev/43b06352221b changeset: 89626:43b06352221b branch: 3.3 parent: 89621:dbc9e3ed5e9f user: ?ric Araujo date: Thu Mar 13 04:55:35 2014 -0400 summary: Make distutils error messages more helpful (#11599). When running external programs such as a C compiler and getting an error code, distutils only prints the program name. With this change, one can get the full command line by setting the DISTUTILS_DEBUG environment variable. This should have no compatibility issues, unless there are tools that depend on the exact format of distutils debug messages. files: Doc/distutils/setupscript.rst | 9 ++- Doc/install/index.rst | 3 +- Lib/distutils/spawn.py | 63 ++++++++++++++++------ Misc/NEWS | 4 + 4 files changed, 57 insertions(+), 22 deletions(-) diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -685,6 +685,8 @@ DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -700,7 +702,8 @@ and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -58,7 +58,8 @@ document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. .. _inst-trivial-install: diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -10,6 +10,7 @@ import os from distutils.errors import DistutilsPlatformError, DistutilsExecError +from distutils.debug import DEBUG from distutils import log def spawn(cmd, search_path=1, verbose=0, dry_run=0): @@ -28,6 +29,9 @@ Raise DistutilsExecError if running the program fails in any way; just return on success. """ + # cmd is documented as a list, but just in case some code passes a tuple + # in, protect our %-formatting code against horrible death + cmd = list(cmd) if os.name == 'posix': _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': @@ -67,12 +71,16 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError as exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if rc != 0: # and this reflects the command running but failing + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" % (cmd[0], rc)) + "command %r failed with exit status %d" % (cmd, rc)) def _spawn_os2(cmd, search_path=1, verbose=0, dry_run=0): executable = cmd[0] @@ -86,13 +94,17 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError as exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if rc != 0: # and this reflects the command running but failing - log.debug("command '%s' failed with exit status %d" % (cmd[0], rc)) + if not DEBUG: + cmd = executable + log.debug("command %r failed with exit status %d" % (cmd, rc)) raise DistutilsExecError( - "command '%s' failed with exit status %d" % (cmd[0], rc)) + "command %r failed with exit status %d" % (cmd, rc)) if sys.platform == 'darwin': from distutils import sysconfig @@ -103,8 +115,9 @@ log.info(' '.join(cmd)) if dry_run: return + executable = cmd[0] exec_fn = search_path and os.execvp or os.execv - exec_args = [cmd[0], cmd] + env = None if sys.platform == 'darwin': global _cfg_target, _cfg_target_split if _cfg_target is None: @@ -125,17 +138,23 @@ env = dict(os.environ, MACOSX_DEPLOYMENT_TARGET=cur_target) exec_fn = search_path and os.execvpe or os.execve - exec_args.append(env) pid = os.fork() if pid == 0: # in the child try: - exec_fn(*exec_args) + if env is None: + exec_fn(executable, cmd) + else: + exec_fn(executable, cmd, env) except OSError as e: - sys.stderr.write("unable to execute %s: %s\n" - % (cmd[0], e.strerror)) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r: %s\n" + % (cmd, e.strerror)) os._exit(1) - sys.stderr.write("unable to execute %s for unknown reasons" % cmd[0]) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r for unknown reasons" % cmd) os._exit(1) else: # in the parent # Loop until the child either exits or is terminated by a signal @@ -147,26 +166,34 @@ import errno if exc.errno == errno.EINTR: continue + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if os.WIFSIGNALED(status): + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' terminated by signal %d" - % (cmd[0], os.WTERMSIG(status))) + "command %r terminated by signal %d" + % (cmd, os.WTERMSIG(status))) elif os.WIFEXITED(status): exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" - % (cmd[0], exit_status)) + "command %r failed with exit status %d" + % (cmd, exit_status)) elif os.WIFSTOPPED(status): continue else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "unknown error executing '%s': termination status %d" - % (cmd[0], status)) + "unknown error executing %r: termination status %d" + % (cmd, status)) def find_executable(executable, path=None): """Tries to find 'executable' in the directories listed in 'path'. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -16,6 +16,10 @@ - Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + - Issue #4931: distutils should not produce unhelpful "error: None" messages anymore. distutils.util.grok_environment_error is kept but doc-deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 11:02:30 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 11:02:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zICgjMTE1OTkp?= Message-ID: <3fl5F26FWWz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/c04260b59e8c changeset: 89627:c04260b59e8c parent: 89624:f22e1d89486d parent: 89626:43b06352221b user: ?ric Araujo date: Thu Mar 13 05:59:31 2014 -0400 summary: Merge 3.3 (#11599) files: Doc/distutils/setupscript.rst | 9 ++- Doc/install/index.rst | 3 +- Lib/distutils/spawn.py | 55 ++++++++++++++++------ Misc/NEWS | 4 + 4 files changed, 52 insertions(+), 19 deletions(-) diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -685,6 +685,8 @@ DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -700,7 +702,8 @@ and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -58,7 +58,8 @@ document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. .. _inst-trivial-install: diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -10,6 +10,7 @@ import os from distutils.errors import DistutilsPlatformError, DistutilsExecError +from distutils.debug import DEBUG from distutils import log def spawn(cmd, search_path=1, verbose=0, dry_run=0): @@ -28,10 +29,15 @@ Raise DistutilsExecError if running the program fails in any way; just return on success. """ + # cmd is documented as a list, but just in case some code passes a tuple + # in, protect our %-formatting code against horrible death + cmd = list(cmd) if os.name == 'posix': _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': _spawn_nt(cmd, search_path, dry_run=dry_run) + elif os.name == 'os2': + _spawn_os2(cmd, search_path, dry_run=dry_run) else: raise DistutilsPlatformError( "don't know how to spawn programs on platform '%s'" % os.name) @@ -65,12 +71,16 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError as exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if rc != 0: # and this reflects the command running but failing + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" % (cmd[0], rc)) + "command %r failed with exit status %d" % (cmd, rc)) if sys.platform == 'darwin': from distutils import sysconfig @@ -81,8 +91,9 @@ log.info(' '.join(cmd)) if dry_run: return + executable = cmd[0] exec_fn = search_path and os.execvp or os.execv - exec_args = [cmd[0], cmd] + env = None if sys.platform == 'darwin': global _cfg_target, _cfg_target_split if _cfg_target is None: @@ -103,17 +114,23 @@ env = dict(os.environ, MACOSX_DEPLOYMENT_TARGET=cur_target) exec_fn = search_path and os.execvpe or os.execve - exec_args.append(env) pid = os.fork() if pid == 0: # in the child try: - exec_fn(*exec_args) + if env is None: + exec_fn(executable, cmd) + else: + exec_fn(executable, cmd, env) except OSError as e: - sys.stderr.write("unable to execute %s: %s\n" - % (cmd[0], e.strerror)) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r: %s\n" + % (cmd, e.strerror)) os._exit(1) - sys.stderr.write("unable to execute %s for unknown reasons" % cmd[0]) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r for unknown reasons" % cmd) os._exit(1) else: # in the parent # Loop until the child either exits or is terminated by a signal @@ -125,26 +142,34 @@ import errno if exc.errno == errno.EINTR: continue + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if os.WIFSIGNALED(status): + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' terminated by signal %d" - % (cmd[0], os.WTERMSIG(status))) + "command %r terminated by signal %d" + % (cmd, os.WTERMSIG(status))) elif os.WIFEXITED(status): exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" - % (cmd[0], exit_status)) + "command %r failed with exit status %d" + % (cmd, exit_status)) elif os.WIFSTOPPED(status): continue else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "unknown error executing '%s': termination status %d" - % (cmd[0], status)) + "unknown error executing %r: termination status %d" + % (cmd, status)) def find_executable(executable, path=None): """Tries to find 'executable' in the directories listed in 'path'. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,10 @@ - Issue #19157: Include the broadcast address in the usuable hosts for IPv6 in ipaddress. +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + - Issue #4931: distutils should not produce unhelpful "error: None" messages anymore. distutils.util.grok_environment_error is kept but doc-deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 11:02:32 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 11:02:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3fl5F42hGyz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/69d503b12432 changeset: 89628:69d503b12432 parent: 89625:077dce5c4196 parent: 89627:c04260b59e8c user: ?ric Araujo date: Thu Mar 13 06:02:15 2014 -0400 summary: Merge heads files: Doc/distutils/setupscript.rst | 9 ++- Doc/install/index.rst | 3 +- Lib/distutils/spawn.py | 55 ++++++++++++++++------ Misc/NEWS | 4 + 4 files changed, 52 insertions(+), 19 deletions(-) diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -685,6 +685,8 @@ DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -700,7 +702,8 @@ and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -58,7 +58,8 @@ document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. .. _inst-trivial-install: diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -10,6 +10,7 @@ import os from distutils.errors import DistutilsPlatformError, DistutilsExecError +from distutils.debug import DEBUG from distutils import log def spawn(cmd, search_path=1, verbose=0, dry_run=0): @@ -28,10 +29,15 @@ Raise DistutilsExecError if running the program fails in any way; just return on success. """ + # cmd is documented as a list, but just in case some code passes a tuple + # in, protect our %-formatting code against horrible death + cmd = list(cmd) if os.name == 'posix': _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': _spawn_nt(cmd, search_path, dry_run=dry_run) + elif os.name == 'os2': + _spawn_os2(cmd, search_path, dry_run=dry_run) else: raise DistutilsPlatformError( "don't know how to spawn programs on platform '%s'" % os.name) @@ -65,12 +71,16 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError as exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if rc != 0: # and this reflects the command running but failing + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" % (cmd[0], rc)) + "command %r failed with exit status %d" % (cmd, rc)) if sys.platform == 'darwin': from distutils import sysconfig @@ -81,8 +91,9 @@ log.info(' '.join(cmd)) if dry_run: return + executable = cmd[0] exec_fn = search_path and os.execvp or os.execv - exec_args = [cmd[0], cmd] + env = None if sys.platform == 'darwin': global _cfg_target, _cfg_target_split if _cfg_target is None: @@ -103,17 +114,23 @@ env = dict(os.environ, MACOSX_DEPLOYMENT_TARGET=cur_target) exec_fn = search_path and os.execvpe or os.execve - exec_args.append(env) pid = os.fork() if pid == 0: # in the child try: - exec_fn(*exec_args) + if env is None: + exec_fn(executable, cmd) + else: + exec_fn(executable, cmd, env) except OSError as e: - sys.stderr.write("unable to execute %s: %s\n" - % (cmd[0], e.strerror)) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r: %s\n" + % (cmd, e.strerror)) os._exit(1) - sys.stderr.write("unable to execute %s for unknown reasons" % cmd[0]) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r for unknown reasons" % cmd) os._exit(1) else: # in the parent # Loop until the child either exits or is terminated by a signal @@ -125,26 +142,34 @@ import errno if exc.errno == errno.EINTR: continue + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if os.WIFSIGNALED(status): + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' terminated by signal %d" - % (cmd[0], os.WTERMSIG(status))) + "command %r terminated by signal %d" + % (cmd, os.WTERMSIG(status))) elif os.WIFEXITED(status): exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" - % (cmd[0], exit_status)) + "command %r failed with exit status %d" + % (cmd, exit_status)) elif os.WIFSTOPPED(status): continue else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "unknown error executing '%s': termination status %d" - % (cmd[0], status)) + "unknown error executing %r: termination status %d" + % (cmd, status)) def find_executable(executable, path=None): """Tries to find 'executable' in the directories listed in 'path'. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,10 @@ - Issue #19157: Include the broadcast address in the usuable hosts for IPv6 in ipaddress. +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + - Issue #4931: distutils should not produce unhelpful "error: None" messages anymore. distutils.util.grok_environment_error is kept but doc-deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 13:14:06 2014 From: python-checkins at python.org (nick.coghlan) Date: Thu, 13 Mar 2014 13:14:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319407=3A_New_inst?= =?utf-8?q?allation_=26_distribution_guides?= Message-ID: <3fl88t0kQHz7LjY@mail.python.org> http://hg.python.org/cpython/rev/d22ef969cb82 changeset: 89629:d22ef969cb82 user: Nick Coghlan date: Thu Mar 13 22:13:45 2014 +1000 summary: Close #19407: New installation & distribution guides - based on pip and other PyPA tools - includes references to the new Python Packaging User Guide where appropriate (and the relevant section is at least partially filled in) - started new FAQ sections - both guides aim to introduce users to basic open source concepts if they aren't aware of them - existing guides have been relocated (now linked from the distutils docs) rather then removed, since there is some needed material that has yet to be relocated to the distutils docs as a reference for the legacy formats files: Doc/conf.py | 4 +- Doc/contents.rst | 12 +- Doc/distributing/index.rst | 143 +++++++++ Doc/distutils/index.rst | 6 +- Doc/extending/index.rst | 42 ++- Doc/install/index.rst | 6 +- Doc/installing/index.rst | 211 ++++++++++++++ Doc/library/distutils.rst | 19 +- Doc/library/ensurepip.rst | 2 +- Doc/tools/sphinxext/indexcontent.html | 8 +- Doc/tutorial/whatnow.rst | 4 +- Doc/using/venv-create.inc | 5 + Doc/using/windows.rst | 4 + Doc/whatsnew/3.4.rst | 19 + Misc/NEWS | 6 + 15 files changed, 459 insertions(+), 32 deletions(-) diff --git a/Doc/conf.py b/Doc/conf.py --- a/Doc/conf.py +++ b/Doc/conf.py @@ -120,11 +120,11 @@ latex_documents = [ ('c-api/index', 'c-api.tex', 'The Python/C API', _stdauthor, 'manual'), - ('distutils/index', 'distutils.tex', + ('distributing/index', 'distributing.tex', 'Distributing Python Modules', _stdauthor, 'manual'), ('extending/index', 'extending.tex', 'Extending and Embedding Python', _stdauthor, 'manual'), - ('install/index', 'install.tex', + ('installing/index', 'installing.tex', 'Installing Python Modules', _stdauthor, 'manual'), ('library/index', 'library.tex', 'The Python Library Reference', _stdauthor, 'manual'), diff --git a/Doc/contents.rst b/Doc/contents.rst --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -11,8 +11,8 @@ library/index.rst extending/index.rst c-api/index.rst - distutils/index.rst - install/index.rst + distributing/index.rst + installing/index.rst howto/index.rst faq/index.rst glossary.rst @@ -21,3 +21,11 @@ bugs.rst copyright.rst license.rst + +.. include legacy packaging docs in build + +.. toctree:: + :hidden: + + distutils/index.rst + install/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst new file mode 100644 --- /dev/null +++ b/Doc/distributing/index.rst @@ -0,0 +1,143 @@ +.. _distributing-index: + +############################### + Distributing Python Modules +############################### + +:Email: distutils-sig at python.org + + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the distribution part of the process. For a guide to +installing other Python projects, refer to the +:ref:`installation guide `. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* the `Python Package Index `__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + `__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub ` and + `BitBucket `__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Open source licensing and collaboration +======================================= + +In most parts of the world, software is automatically covered by copyright. +This means that other developers require explicit permission to copy, use, +modify and redistribute the software. + +Open source licensing is a way of explicitly granting such permission in a +relatively consistent way, allowing developers to share and collaborate +efficiently by making common solutions to various problems freely available. +This leaves many developers free to spend more time focusing on the problems +that are relatively unique to their specific situation. + +The distribution tools provided with Python are designed to make it +reasonably straightforward for developers to make their own contributions +back to that common pool of software if they choose to do so. + +The same distribution tools can also be used to distribute software within +an organisation, regardless of whether that software is published as open +source software or not. + + +Installing the tools +==================== + +The standard library does not include build tools that support modern +Python packaging standards, as the core development team has found that it +is important to have standard tools that work consistently, even on older +versions of Python. + +The currently recommended build and distribution tools can be installed +using ``pip``:: + + pip install setuptools wheel twine + + +Reading the guide +================= + +The Python Packaging User Guide covers the various key steps and elements +involved in creating a project + +* `Project structure`_ +* `Building and packaging the project`_ +* `Uploading the project to the Python Package Index`_ + +.. _Project structure: \ + http://packaging.python.org/en/latest/tutorial.html#creating-your-own-project +.. _Building and packaging the project: \ + http://packaging.python.org/en/latest/tutorial.html#building-packaging-your-project +.. _Uploading the project to the Python Package Index: \ + http://packaging.python.org/en/latest/tutorial.html#building-uploading-your-project-to-pypi + + +How do I...? +============ + +These are quick answers or links for some common tasks. + +... choose a name for my project? +--------------------------------- + +This isn't an easy topic, but here are a few tips: + +* check the Python Package Index to see if the name is already in use +* check popular hosting sites like GitHub, BitBucket, etc to see if there + is already a project with that name +* check what comes up in a web search for the name you're considering +* avoid particularly common words, especially ones with multiple meanings, + as they can make it difficult for users to find your software when + searching for it + + +... create and distribute binary extensions? +-------------------------------------------- + +This is actually quite a complex topic, with a variety of alternatives +available depending on exactly what you're aiming to achieve. See the +Python Packaging User Guide for more information and recommendations. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + `__ + +.. other topics: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -1,8 +1,8 @@ .. _distutils-index: -############################### - Distributing Python Modules -############################### +############################################## + Distributing Python Modules (Legacy version) +############################################## :Authors: Greg Ward, Anthony Baxter :Email: distutils-sig at python.org diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,14 +21,31 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. -.. note:: - This guide only covers the basic tools for creating extensions provided - as part of this version of CPython. Third party tools may offer simpler - alternatives. Refer to the `binary extensions section - `__ - in the Python Packaging User Guide for more information. +Recommended third party tools +============================= +This guide only covers the basic tools for creating extensions provided +as part of this version of CPython. Third party tools like Cython, +``cffi``, SWIG and Numba offer both simpler and more sophisticated +approaches to creating C and C++ extensions for Python. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions `_ + The Python Packaging User Guide not only covers several available + tools that simplify the creation of binary extensions, but also + discusses the various reasons why creating an extension module may be + desirable in the first place. + + +Creating extensions without third party tools +============================================= + +This section of the guide covers creating C and C++ extensions without +assistance from third party tools. It is intended primarily for creators +of those tools, rather than being a recommended way to create your own +C extensions. .. toctree:: :maxdepth: 2 @@ -38,4 +55,17 @@ newtypes.rst building.rst windows.rst + +Embedding the CPython runtime in a larger application +===================================================== + +Sometimes, rather than creating an extension that runs inside the Python +interpreter as the main application, it is desirable to instead embed +the CPython runtime inside a larger application. This section covers +some of the details involved in doing that successfully. + +.. toctree:: + :maxdepth: 2 + :numbered: + embedding.rst diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -2,9 +2,9 @@ .. _install-index: -***************************** - Installing Python Modules -***************************** +******************************************** + Installing Python Modules (Legacy version) +******************************************** :Author: Greg Ward diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst new file mode 100644 --- /dev/null +++ b/Doc/installing/index.rst @@ -0,0 +1,211 @@ +.. highlightlang:: none + +.. _installing-index: + +***************************** + Installing Python Modules +***************************** + +:Email: distutils-sig at python.org + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the installation part of the process. For a guide to +creating and sharing your own Python projects, refer to the +:ref:`distribution guide `. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* ``pip`` is the preferred installer program. Starting with Python 3.4, it + is included by default with the Python binary installers. + installed into virtual environments +* a virtual environment is a semi-isolated Python environment that allows + packages to be installed for use by a particular application, rather than + being installed system wide +* ``pyvenv`` is the standard tool for creating virtual environments, and has + been part of Python since Python 3.3. Starting with Python 3.4, it + defaults to installing ``pip`` into all created virtual environments +* the `Python Package Index `__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + `__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub ` and + `BitBucket `__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Basic usage +=========== + +The standard packaging tools are all designed to be used from the command +line. For Windows users, the examples below assume that the option to +adjust the system PATH environment variable was selected when installing +Python. For Linux users, the command to install into the system version of +Python 3 is likely to be ``pip3`` rather than ``pip``. + +The following command will install the latest version of a module and its +dependencies from the Python Package Index:: + + pip install SomePackage + +It's also possible to specify an exact or minimum version directly on the +command line:: + + pip install SomePackage==1.0.4 # specific version + pip install 'SomePackage>=1.0.4' # minimum version + +Normally, if a suitable module is already installed, attempting to install +it again will have no effect. Upgrading existing modules must be requested +explicitly:: + + pip install --upgrade SomePackage + +More information and resources regarding ``pip`` and its capabilities can be +found in the `Python Packaging User Guide `__. + +``pyvenv`` has its own documentation at :ref:`scripts-pyvenv`. Installing +into an active virtual environment uses the commands shown above. + +.. seealso:: + + `Python Packaging User Guide: Installing Python packages + `__ + + +How do I ...? +============= + +These are quick answers or links for some common tasks. + +... install ``pip`` in versions of Python prior to Python 3.4? +-------------------------------------------------------------- + +Python only started bundling ``pip`` with Python 3.4. For earlier versions, +``pip`` needs to be "bootstrapped" as described in the Python Packaging +User Guide. + +.. seealso:: + + `Python Packaging User Guide: Installing the Tools + `__ + + +.. installing-per-user-installation: + +... install packages just for the current user? +----------------------------------------------- + +Passing the ``--user`` option to ``pip install`` will install a package +just for the current user, rather than for all users of the system. + + +... install scientific Python packages? +--------------------------------------- + +A number of scientific Python packages have complex binary dependencies, and +aren't currently easy to install using ``pip`` directly. At this point in +time, it will often be easier for users to install these packages by +`other means +`__ +rather than attempting to install them with ``pip``. + +.. seealso:: + + `Python Packaging User Guide: Installing Scientific Packages + `__ + + +... work with multiple versions of Python installed in parallel? +---------------------------------------------------------------- + +On Linux, Mac OS X and other POSIX systems, use the versioned Python commands +in combination with the ``-m`` switch to run the appropriate copy of +``pip``:: + + python2 -m pip install SomePackage # default Python 2 + python2.7 -m pip install SomePackage # specifically Python 2.7 + python3 -m pip install SomePackage # default Python 3 + python3.4 -m pip install SomePackage # specifically Python 3.4 + +(appropriately versioned ``pip`` commands may also be available) + +On Windows, use the ``py`` Python launcher in combination with the ``-m`` +switch:: + + py -2 -m pip install SomePackage # default Python 2 + py -2.7 -m pip install SomePackage # specifically Python 2.7 + py -3 -m pip install SomePackage # default Python 3 + py -3.4 -m pip install SomePackage # specifically Python 3.4 + +.. other questions: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) + + +Common installation issues +========================== + +Installing into the system Python on Linux +------------------------------------------ + +On Linux systems, a Python installation will typically be included as part +of the distribution. Installing into this Python installation requires +root access to the system, and may interfere with the operation of the +system package manager and other components of the system if a component +is unexpectedly upgraded using ``pip``. + +On such systems, it is often better to use a virtual environment or a +per-user installation when installing packages with ``pip``. + + +Installing binary extensions +---------------------------- + +Python has typically relied heavily on source based distribution, with end +users being expected to compile extension modules from source as part of +the installation process. + +With the introduction of support for the binary ``wheel`` format, and the +ability to publish wheels for at least Windows and Mac OS X through the +Python Package Index, this problem is expected to diminish over time, +as users are more regularly able to install pre-built extensions rather +than needing to build them themselves. + +Some of the solutions for installing `scientific software +`__ +that is not yet available as pre-built ``wheel`` files may also help with +obtaining other binary extensions without needing to build them locally. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + `__ diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -12,14 +12,15 @@ 100%-pure Python, or may be extension modules written in C, or may be collections of Python packages which include modules coded in both Python and C. +Most Python users will *not* want to use this module directly, but instead +use the cross-version tools maintained by the Python Packaging Authority. +Refer to the `Python Packaging User Guide `_ +for more information. -User documentation and API reference are provided in another document: +For the benefits of packaging tool authors and users seeking a deeper +understanding of the details of the current packaging and distribution +system, the legacy :mod:`distutils` based user documentation and API +reference remain available: -.. seealso:: - - :ref:`distutils-index` - The manual for developers and packagers of Python modules. This describes - how to prepare :mod:`distutils`\ -based packages so that they may be - easily installed into an existing Python installation. It also contains - instructions for end-users wanting to install a distutils-based package, - :ref:`install-index`. +* :ref:`install-index` +* :ref:`distutils-index` diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -28,7 +28,7 @@ .. seealso:: - :ref:`install-index` + :ref:`installing-index` The end user guide for installing Python packages :pep:`453`: Explicit bootstrapping of pip in Python installations diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -16,14 +16,14 @@

Python HOWTOs
in-depth documents on specific topics

+

Installing Python Modules
+ installing from the Python Package Index & other sources

+

Distributing Python Modules
+ publishing modules for installation by others

Extending and Embedding
tutorial for C/C++ programmers

Python/C API
reference for C/C++ programmers

-

Installing Python Modules
- information for installers & sys-admins

-

Distributing Python Modules
- sharing modules with others

FAQs
frequently asked questions (with answers!)

diff --git a/Doc/tutorial/whatnow.rst b/Doc/tutorial/whatnow.rst --- a/Doc/tutorial/whatnow.rst +++ b/Doc/tutorial/whatnow.rst @@ -21,8 +21,8 @@ and many other tasks. Skimming through the Library Reference will give you an idea of what's available. -* :ref:`install-index` explains how to install external modules written by other - Python users. +* :ref:`installing-index` explains how to install additional modules written + by other Python users. * :ref:`reference-index`: A detailed explanation of Python's syntax and semantics. It's heavy reading, but is useful as a complete guide to the diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -11,6 +11,11 @@ Windows). It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory (on Windows, this is ``Lib\site-packages``). +.. seealso:: + + `Python Packaging User Guide: Creating and using virtual environments + `__ + .. highlight:: none On Windows, you may have to invoke the ``pyvenv`` script as follows, if you diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -11,6 +11,10 @@ This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. +.. XXX (ncoghlan) + + This looks rather stale to me... + Installing Python ================= diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -180,6 +180,9 @@ PEP 453: Explicit Bootstrapping of PIP in Python Installations -------------------------------------------------------------- +Bootstrapping pip by default +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard cross-platform mechanism to bootstrap the pip installer into Python installations and virtual environments. @@ -207,6 +210,22 @@ __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors + +Documentation changes +~~~~~~~~~~~~~~~~~~~~~ + +As part of this change, the :ref:`installing-index` and +:ref:`distributing-index` sections of the documentation have been +completely redesigned as short getting started and FAQ documents. Most +packaging documentation has now been moved out to the Python Packaging +Authority maintained `Python Packaging User Guide +`__ and the documentation of the individual +projects. + +However, as this migration is currently still incomplete, the legacy +versions of those guides remaining available as :ref:`install-index` +and :ref:`distutils-index`. + .. note:: To avoid conflicts between parallel Python 2 and Python 3 installations, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -59,6 +59,12 @@ - Issue #20765: Add missing documentation for PurePath.with_name() and PurePath.with_suffix(). +- Issue #19407: New package installation and distribution guides based on + the Python Packaging Authority tools. Existing guides have been retained + as legacy links from the distutils docs, as they still contain some + required reference material for tool developers that isn't recorded + anywhere else. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 13:33:53 2014 From: python-checkins at python.org (nick.coghlan) Date: Thu, 13 Mar 2014 13:33:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319407=3A_fix_link?= Message-ID: <3fl8bj05V0z7Lk7@mail.python.org> http://hg.python.org/cpython/rev/f5be4ea5b43e changeset: 89630:f5be4ea5b43e user: Nick Coghlan date: Thu Mar 13 22:33:33 2014 +1000 summary: Issue #19407: fix link files: Doc/distributing/index.rst | 2 +- Doc/installing/index.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -39,7 +39,7 @@ developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation - and issue trackers on both `GitHub ` and + and issue trackers on both `GitHub `__ and `BitBucket `__. * ``distutils`` is the original build and distribution system first added to the Python standard library in 1998. While direct use of ``distutils`` is diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -49,7 +49,7 @@ developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation - and issue trackers on both `GitHub ` and + and issue trackers on both `GitHub `__ and `BitBucket `__. * ``distutils`` is the original build and distribution system first added to the Python standard library in 1998. While direct use of ``distutils`` is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 14:54:54 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 13 Mar 2014 14:54:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_What=27s_New_in_Python_3?= =?utf-8?q?=2E4=3A_more_security_changes?= Message-ID: <3flBPB2vDKzMfc@mail.python.org> http://hg.python.org/cpython/rev/78186a0affd3 changeset: 89631:78186a0affd3 user: Victor Stinner date: Thu Mar 13 14:54:46 2014 +0100 summary: What's New in Python 3.4: more security changes files: Doc/whatsnew/3.4.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -143,6 +143,12 @@ (:pep:`456`). * :ref:`Make newly created file descriptors non-inheritable ` (:pep:`446`) to avoid leaking file descriptors to child processes. +* New command line option for :ref:`isolated mode `, + (:issue:`16499`). +* All modules of the standard library now support server certificate + verification including hostname matching (:func:`ssl.match_hostname`) and CRL + (Certificate Revocation list, see + :func:`ssl.SSLContext.load_verify_locations`). * A new :func:`hashlib.pbkdf2_hmac` function provides the `PKCS#5 password-based key derivation function 2 `_. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 16:04:42 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 13 Mar 2014 16:04:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_fix_unittest_s?= =?utf-8?q?ubtest_example=2E?= Message-ID: <3flCxk0ZHSz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/9c641c97db36 changeset: 89632:9c641c97db36 user: R David Murray date: Wed Mar 12 09:31:50 2014 -0400 summary: whatsnew: fix unittest subtest example. files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1642,7 +1642,7 @@ class NumbersTest(unittest.TestCase): def test_even(self): for i in range(6): - with self.subTest(i=1): + with self.subTest(i=i): self.assertEqual(i % 2, 0) will result in six subtests, each identified in the unittest verbose output -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 16:11:08 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 13 Mar 2014 16:11:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_make_it_clear_?= =?utf-8?q?I=27m_the_editor=2C_not_author_of_all_text=2E?= Message-ID: <3flD5823Pjz7Llc@mail.python.org> http://hg.python.org/cpython/rev/8ca08f43eb1e changeset: 89633:8ca08f43eb1e user: R David Murray date: Thu Mar 13 11:10:01 2014 -0400 summary: whatsnew: make it clear I'm the editor, not author of all text. files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2,7 +2,7 @@ What's New In Python 3.4 **************************** -:Author: R. David Murray +:Author: R. David Murray (Editor) .. Rules for maintenance: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 17:01:11 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 13 Mar 2014 17:01:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_HTTPError=2Ehe?= =?utf-8?b?YWRlcnMgKCMxNTcwMSku?= Message-ID: <3flFBv4Fz6z7Lk7@mail.python.org> http://hg.python.org/cpython/rev/361c10d06b9c changeset: 89634:361c10d06b9c user: R David Murray date: Thu Mar 13 11:33:29 2014 -0400 summary: whatsnew: HTTPError.headers (#15701). files: Doc/library/urllib.error.rst | 2 +- Doc/whatsnew/3.4.rst | 5 +++++ 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -48,7 +48,7 @@ .. attribute:: headers - The HTTP response headers for the HTTP request that cause the + The HTTP response headers for the HTTP request that caused the :exc:`HTTPError`. .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1627,6 +1627,11 @@ Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht and Senthil Kumaran in :issue:`17272`.) +:class:`~urllib.error.HTTPError` objects now have a +:attr:`~urllib.error.HTTPError.headers` attribute that provides access to the +HTTP response headers associated with the error. (Contributed by +Berker Peksag in :issue:`15701`.) + unittest -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 17:01:12 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 13 Mar 2014 17:01:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_collections_no?= =?utf-8?q?_longer_implicitly_imports_=27abc=27_=28=2320784=29=2E?= Message-ID: <3flFBw63s5z7Lkp@mail.python.org> http://hg.python.org/cpython/rev/d575398d1916 changeset: 89635:d575398d1916 user: R David Murray date: Thu Mar 13 12:00:17 2014 -0400 summary: whatsnew: collections no longer implicitly imports 'abc' (#20784). files: Doc/whatsnew/3.4.rst | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2436,6 +2436,13 @@ have things that look like doctests in them you may see test failures you've never seen before when running your tests (:issue:`3158`). +* The :mod:`collections.abc` module has been slightly refactored as + part of the Python startup improvements. As a consequence of this, it is no + longer the case that importing :mod:`collections` automatically imports + :mod:`collections.abc`. If your program depended on the (undocumented) + implicit import, you will need to add an explicit ``import collections.abc`` + (:issue:`20784`). + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 17:38:40 2014 From: python-checkins at python.org (jesus.cea) Date: Thu, 13 Mar 2014 17:38:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMDkw?= =?utf-8?q?8=3A_Memory_leak_in_Reg2Py=28=29?= Message-ID: <3flG2826wdz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/b44a3f6676b8 changeset: 89636:b44a3f6676b8 branch: 2.7 parent: 89620:1763e27a182d user: Jesus Cea date: Thu Mar 13 17:33:43 2014 +0100 summary: Closes #20908: Memory leak in Reg2Py() files: PC/_winreg.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/PC/_winreg.c b/PC/_winreg.c --- a/PC/_winreg.c +++ b/PC/_winreg.c @@ -948,8 +948,10 @@ fixupMultiSZ(str, retDataBuf, retDataSize); obData = PyList_New(s); - if (obData == NULL) + if (obData == NULL) { + free(str); return NULL; + } for (index = 0; index < s; index++) { size_t len = _mbstrlen(str[index]); @@ -957,6 +959,7 @@ PyErr_SetString(PyExc_OverflowError, "registry string is too long for a Python string"); Py_DECREF(obData); + free(str); return NULL; } PyList_SetItem(obData, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 17:38:41 2014 From: python-checkins at python.org (jesus.cea) Date: Thu, 13 Mar 2014 17:38:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQ2xvc2VzICMyMDkw?= =?utf-8?q?8=3A_Memory_leak_in_Reg2Py=28=29?= Message-ID: <3flG295GTZz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/07968254be96 changeset: 89637:07968254be96 branch: 3.3 parent: 89626:43b06352221b user: Jesus Cea date: Thu Mar 13 17:35:32 2014 +0100 summary: Closes #20908: Memory leak in Reg2Py() files: PC/winreg.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/PC/winreg.c b/PC/winreg.c --- a/PC/winreg.c +++ b/PC/winreg.c @@ -943,8 +943,10 @@ fixupMultiSZ(str, data, len); obData = PyList_New(s); - if (obData == NULL) + if (obData == NULL) { + free(str); return NULL; + } for (index = 0; index < s; index++) { size_t len = wcslen(str[index]); @@ -952,6 +954,7 @@ PyErr_SetString(PyExc_OverflowError, "registry string is too long for a Python string"); Py_DECREF(obData); + free(str); return NULL; } PyList_SetItem(obData, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 17:38:43 2014 From: python-checkins at python.org (jesus.cea) Date: Thu, 13 Mar 2014 17:38:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_MERGE=3A_Closes_=2320908=3A_Memory_leak_in_Reg2Py=28=29?= Message-ID: <3flG2C19gvz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/227405961789 changeset: 89638:227405961789 parent: 89635:d575398d1916 parent: 89637:07968254be96 user: Jesus Cea date: Thu Mar 13 17:38:20 2014 +0100 summary: MERGE: Closes #20908: Memory leak in Reg2Py() files: PC/winreg.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/PC/winreg.c b/PC/winreg.c --- a/PC/winreg.c +++ b/PC/winreg.c @@ -943,8 +943,10 @@ fixupMultiSZ(str, data, len); obData = PyList_New(s); - if (obData == NULL) + if (obData == NULL) { + free(str); return NULL; + } for (index = 0; index < s; index++) { size_t len = wcslen(str[index]); @@ -952,6 +954,7 @@ PyErr_SetString(PyExc_OverflowError, "registry string is too long for a Python string"); Py_DECREF(obData); + free(str); return NULL; } PyList_SetItem(obData, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 19:09:11 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 13 Mar 2014 19:09:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320908=3A_PyMem=5F?= =?utf-8?q?Malloc=28=29_must_be_used_with_PyMem=5FFree=28=29=2C_not_with_f?= =?utf-8?b?cmVlKCk=?= Message-ID: <3flJ2b2bgmz7LjT@mail.python.org> http://hg.python.org/cpython/rev/6e3e91662216 changeset: 89639:6e3e91662216 user: Victor Stinner date: Thu Mar 13 19:08:10 2014 +0100 summary: Issue #20908: PyMem_Malloc() must be used with PyMem_Free(), not with free() files: PC/winreg.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/PC/winreg.c b/PC/winreg.c --- a/PC/winreg.c +++ b/PC/winreg.c @@ -944,7 +944,7 @@ fixupMultiSZ(str, data, len); obData = PyList_New(s); if (obData == NULL) { - free(str); + PyMem_Free(str); return NULL; } for (index = 0; index < s; index++) @@ -954,7 +954,7 @@ PyErr_SetString(PyExc_OverflowError, "registry string is too long for a Python string"); Py_DECREF(obData); - free(str); + PyMem_Free(str); return NULL; } PyList_SetItem(obData, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 19:37:06 2014 From: python-checkins at python.org (r.david.murray) Date: Thu, 13 Mar 2014 19:37:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_remove_=27draf?= =?utf-8?q?t=27_note=2C_tidy_up_summary=2C_collect_SSLContext_stuff=2E?= Message-ID: <3flJfp5SyhzRdL@mail.python.org> http://hg.python.org/cpython/rev/5e27593aa215 changeset: 89640:5e27593aa215 user: R David Murray date: Thu Mar 13 14:36:09 2014 -0400 summary: whatsnew: remove 'draft' note, tidy up summary, collect SSLContext stuff. Since Victor linked to it in a block, it seems to make sense to have all the SSLContext changes next to each other. I also sorted all the SSL security enhancements next to each other in the security enhancements summary. files: Doc/whatsnew/3.4.rst | 70 +++++++++++++++---------------- 1 files changed, 33 insertions(+), 37 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -67,11 +67,6 @@ For full details, see the `changelog `_. -.. note:: Prerelease users should be aware that this document is currently in - draft form. While it should be close to complete for the Python 3.4 - release candidates, adjustments and additions to the document may be made - up until the final release. - .. seealso:: @@ -92,9 +87,9 @@ New expected features for Python implementations: * :ref:`pip should always be "available" ` (:pep:`453`). -* :ref:`Make newly created file descriptors non-inheritable ` +* :ref:`Newly created file descriptors are non-inheritable ` (:pep:`446`). -* command line option for :ref:`isolated mode `, +* command line option for :ref:`isolated mode ` (:issue:`16499`). * :ref:`improvements in the handling of codecs ` that are not text encodings (multiple issues). @@ -145,10 +140,11 @@ (:pep:`446`) to avoid leaking file descriptors to child processes. * New command line option for :ref:`isolated mode `, (:issue:`16499`). -* All modules of the standard library now support server certificate - verification including hostname matching (:func:`ssl.match_hostname`) and CRL - (Certificate Revocation list, see - :func:`ssl.SSLContext.load_verify_locations`). +* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork + on Unix `. *spawn* and *forkserver* are + more secure because they avoid sharing data with child processes. +* :mod:`multiprocessing` child processes on Windows no longer inherit + all of the parent's inheritable handles, only the necessary ones. * A new :func:`hashlib.pbkdf2_hmac` function provides the `PKCS#5 password-based key derivation function 2 `_. @@ -157,18 +153,18 @@ ` for :mod:`ssl`. * :ref:`Server-side SNI (Server Name Indication) support ` for :mod:`ssl`. -* The :class:`ssl.SSLContext` class got a :ref:`lot of improvements +* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements `. -* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork - on Unix `: *spawn* and *forkserver* avoid - sharing data with child processes; child processes no longer inherit all of - the parents inheritable handles on Windows. +* All modules in the standard library that support SSL now support server + certificate verification, including hostname matching + (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see + :func:`ssl.SSLContext.load_verify_locations`). CPython implementation improvements: * :ref:`Safe object finalization ` (:pep:`442`). -* Leveraging :pep:`442`, :ref:`module globals are no longer set to None - during finalization `, in most cases (:issue:`18214`). +* Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set + to None during finalization ` (:issue:`18214`). * :ref:`Configurable memory allocators ` (:pep:`445`). * :ref:`Argument Clinic ` (:pep:`436`). @@ -251,8 +247,8 @@ .. _whatsnew-pep-446: -PEP 446: Make Newly Created File Descriptors Non-Inheritable ------------------------------------------------------------- +PEP 446: Newly Created File Descriptors Are Non-Inheritable +----------------------------------------------------------- :pep:`446` makes newly created file descriptors :ref:`non-inheritable `. New functions and methods: @@ -1432,23 +1428,6 @@ list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in and :issue:`18147`.) -.. _whatsnew34-win-cert-store: - -Two new windows-only functions, :func:`~ssl.enum_certificates` and -:func:`~ssl.enum_crls` provide the ability to retrieve certificates, -certificate information, and CRLs from the Windows cert store. (Contributed -by Christian Heimes in :issue:`17134`.) - -.. _whatsnew34-sni: - -Support for server-side SNI (Server Name Indication) using the new -:meth:`ssl.SSLContext.set_servername_callback` method. -(Contributed by Daniel Black in :issue:`8109`.) - -The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional -``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and -``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) - If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the certificate verification process by setting it to some combination of the new @@ -1474,6 +1453,23 @@ best-practice security requirements change. (Contributed by Christian Heimes in :issue:`19689`.) +.. _whatsnew34-win-cert-store: + +Two new windows-only functions, :func:`~ssl.enum_certificates` and +:func:`~ssl.enum_crls` provide the ability to retrieve certificates, +certificate information, and CRLs from the Windows cert store. (Contributed +by Christian Heimes in :issue:`17134`.) + +.. _whatsnew34-sni: + +Support for server-side SNI (Server Name Indication) using the new +:meth:`ssl.SSLContext.set_servername_callback` method. +(Contributed by Daniel Black in :issue:`8109`.) + +The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional +``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and +``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) + stat ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 20:43:19 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 13 Mar 2014 20:43:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319407=3A_Fix_typo?= =?utf-8?q?s_in_new_distribution_=26_installation_guides=2E?= Message-ID: <3flL7C0Prtz7LjM@mail.python.org> http://hg.python.org/cpython/rev/25dc02a2acae changeset: 89641:25dc02a2acae user: Ned Deily date: Thu Mar 13 12:42:36 2014 -0700 summary: Issue #19407: Fix typos in new distribution & installation guides. Also applies to cherry pick Issue #20909. files: Doc/distributing/index.rst | 2 +- Doc/installing/index.rst | 1 - 2 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -101,7 +101,7 @@ .. _Building and packaging the project: \ http://packaging.python.org/en/latest/tutorial.html#building-packaging-your-project .. _Uploading the project to the Python Package Index: \ - http://packaging.python.org/en/latest/tutorial.html#building-uploading-your-project-to-pypi + http://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi How do I...? diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -34,7 +34,6 @@ * ``pip`` is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers. - installed into virtual environments * a virtual environment is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than being installed system wide -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 21:37:53 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 21:37:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Clarify_distut?= =?utf-8?q?ils=E2=80=99_clean_command_=28ref_=236142=29?= Message-ID: <3flML91dCxz7LjM@mail.python.org> http://hg.python.org/cpython/rev/67ada6ab7fe2 changeset: 89642:67ada6ab7fe2 branch: 2.7 parent: 89636:b44a3f6676b8 user: ?ric Araujo date: Thu Mar 13 16:17:11 2014 -0400 summary: Clarify distutils? clean command (ref #6142) files: Doc/distutils/apiref.rst | 8 ++++++-- Doc/distutils/configfile.rst | 2 ++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1907,8 +1907,12 @@ .. module:: distutils.command.clean :synopsis: Clean a package build area - -.. % todo +This command removes the temporary files created by :command:`build` +and its subcommands, like intermediary compiled object files. With +the ``--all`` option, the complete build directory will be removed. + +Extension modules built :ref:`in place ` +will not be cleaned, as they are not in the build directory. :mod:`distutils.command.config` --- Perform package configuration diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -69,6 +69,8 @@ Note that an option spelled :option:`--foo-bar` on the command-line is spelled :option:`foo_bar` in configuration files. +.. _distutils-build-ext-inplace: + For example, say you want your extensions to be built "in-place"---that is, you have an extension :mod:`pkg.ext`, and you want the compiled extension file (:file:`ext.so` on Unix, say) to be put in the same source directory as your -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 23:32:44 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 23:32:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Clarify_distut?= =?utf-8?q?ils=E2=80=99_clean_command_=28ref_=236142=29?= Message-ID: <3flPth3jLSz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/066b896b6132 changeset: 89643:066b896b6132 branch: 3.3 parent: 89637:07968254be96 user: ?ric Araujo date: Thu Mar 13 16:17:11 2014 -0400 summary: Clarify distutils? clean command (ref #6142) files: Doc/distutils/apiref.rst | 8 ++++++-- Doc/distutils/configfile.rst | 2 ++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1934,8 +1934,12 @@ .. module:: distutils.command.clean :synopsis: Clean a package build area - -.. % todo +This command removes the temporary files created by :command:`build` +and its subcommands, like intermediary compiled object files. With +the ``--all`` option, the complete build directory will be removed. + +Extension modules built :ref:`in place ` +will not be cleaned, as they are not in the build directory. :mod:`distutils.command.config` --- Perform package configuration diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -69,6 +69,8 @@ Note that an option spelled :option:`--foo-bar` on the command-line is spelled :option:`foo_bar` in configuration files. +.. _distutils-build-ext-inplace: + For example, say you want your extensions to be built "in-place"---that is, you have an extension :mod:`pkg.ext`, and you want the compiled extension file (:file:`ext.so` on Unix, say) to be put in the same source directory as your -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 13 23:32:45 2014 From: python-checkins at python.org (eric.araujo) Date: Thu, 13 Mar 2014 23:32:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Clarify_distutils=E2=80=99_clean_command_=28ref_=236142?= =?utf-8?q?=29?= Message-ID: <3flPtj5b7Nz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/50ee8fe8e841 changeset: 89644:50ee8fe8e841 parent: 89641:25dc02a2acae parent: 89643:066b896b6132 user: ?ric Araujo date: Thu Mar 13 18:32:18 2014 -0400 summary: Clarify distutils? clean command (ref #6142) files: Doc/distutils/apiref.rst | 8 ++++++-- Doc/distutils/configfile.rst | 2 ++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1923,8 +1923,12 @@ .. module:: distutils.command.clean :synopsis: Clean a package build area - -.. % todo +This command removes the temporary files created by :command:`build` +and its subcommands, like intermediary compiled object files. With +the ``--all`` option, the complete build directory will be removed. + +Extension modules built :ref:`in place ` +will not be cleaned, as they are not in the build directory. :mod:`distutils.command.config` --- Perform package configuration diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -69,6 +69,8 @@ Note that an option spelled :option:`--foo-bar` on the command-line is spelled :option:`foo_bar` in configuration files. +.. _distutils-build-ext-inplace: + For example, say you want your extensions to be built "in-place"---that is, you have an extension :mod:`pkg.ext`, and you want the compiled extension file (:file:`ext.so` on Unix, say) to be put in the same source directory as your -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 00:31:11 2014 From: python-checkins at python.org (eric.araujo) Date: Fri, 14 Mar 2014 00:31:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_references_to_make_?= =?utf-8?q?targets_deleted_in_eef7899ea7ab?= Message-ID: <3flRB76CyBz7LjX@mail.python.org> http://hg.python.org/cpython/rev/74a6b3c9c6cd changeset: 89645:74a6b3c9c6cd user: ?ric Araujo date: Thu Mar 13 19:30:43 2014 -0400 summary: Remove references to make targets deleted in eef7899ea7ab files: Doc/Makefile | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -14,7 +14,7 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ $(SPHINXOPTS) . build/$(BUILDER) $(SOURCES) -.PHONY: help checkout update build html htmlhelp latex text changes linkcheck \ +.PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ autobuild-dev autobuild-stable @@ -36,7 +36,7 @@ @echo " check to run a check for frequent markup errors" @echo " serve to serve the documentation on the localhost (8000)" -build: checkout +build: $(SPHINXBUILD) $(ALLSPHINXOPTS) @echo @@ -157,7 +157,6 @@ # for development releases: always build autobuild-dev: - make update make dist SPHINXOPTS='-A daily=1 -A versionswitcher=1' -make suspicious -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 00:31:55 2014 From: python-checkins at python.org (eric.araujo) Date: Fri, 14 Mar 2014 00:31:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Fix_typo?= Message-ID: <3flRBz089Tz7LjX@mail.python.org> http://hg.python.org/devguide/rev/1a01870ccfa7 changeset: 674:1a01870ccfa7 user: ?ric Araujo date: Thu Mar 13 19:31:20 2014 -0400 summary: Fix typo files: coverage.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/coverage.rst b/coverage.rst --- a/coverage.rst +++ b/coverage.rst @@ -133,7 +133,7 @@ But one of the strengths of coverage.py is its HTML-based reports which let you visually see what lines of code were not tested:: - ./python COVERAGDIR html -i --include=`pwd`/Lib/* --omit="Lib/test/*,Lib/*/tests/*" + ./python COVERAGEDIR html -i --include=`pwd`/Lib/* --omit="Lib/test/*,Lib/*/tests/*" This will generate an HTML report in a directory named ``htmlcov`` which ignores any errors that may arise and ignores modules for which test coverage is -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Fri Mar 14 00:31:56 2014 From: python-checkins at python.org (eric.araujo) Date: Fri, 14 Mar 2014 00:31:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Update_maintainers_entrie?= =?utf-8?q?s_for_distutils?= Message-ID: <3flRC01nZNz7LjY@mail.python.org> http://hg.python.org/devguide/rev/6f21b877f2b5 changeset: 675:6f21b877f2b5 user: ?ric Araujo date: Thu Mar 13 19:31:49 2014 -0400 summary: Update maintainers entries for distutils files: experts.rst | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -96,7 +96,7 @@ decimal facundobatista, rhettinger, mark.dickinson difflib tim.peters (inactive) dis ncoghlan* -distutils tarek, eric.araujo +distutils eric.araujo doctest tim.peters (inactive) dummy_threading brett.cannon email barry, r.david.murray* @@ -160,7 +160,6 @@ optparse aronacher os loewis ossaudiodev -packaging tarek*, eric.araujo*, alexis parser benjamin.peterson pathlib pitrou* pdb georg.brandl* -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Fri Mar 14 01:19:35 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 01:19:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_linkify_some_p?= =?utf-8?q?ip_things=2C_and_explain_why_non-inheriting_is_good=2E?= Message-ID: <3flSFz6Zkkz7LjR@mail.python.org> http://hg.python.org/cpython/rev/ee635e7d213d changeset: 89646:ee635e7d213d user: R David Murray date: Thu Mar 13 19:53:38 2014 -0400 summary: whatsnew: linkify some pip things, and explain why non-inheriting is good. files: Doc/library/venv.rst | 2 ++ Doc/using/unix.rst | 2 ++ Doc/whatsnew/3.4.rst | 27 +++++++++++++++++---------- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -76,6 +76,8 @@ without there needing to be any reference to its venv in ``PATH``. +.. _venv-api: + API --- diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -60,6 +60,8 @@ prompt. +.. _building-python-on-unix: + Building Python =============== diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -193,15 +193,16 @@ X.Y stands for the version of the Python installation), along with the ``pip`` Python package and its dependencies. -The :mod:`venv` module and the :command:`pyvenv` utility make use of this -module to make ``pip`` readily available in virtual environments. When -using the command line interface, ``pip`` is installed by default, while -for the module API installation of ``pip`` must be requested explicitly. - -For CPython source builds on POSIX systems, the ``make install`` and -``make altinstall`` commands bootstrap ``pip`` by default. This behaviour -can be controlled through configure options, and overridden through -Makefile options. +The :ref:`pyvenv ` command line utility and the :mod:`venv` +module make use of the :mod:`ensurepip` module to make ``pip`` readily +available in virtual environments. When using the command line utility, ``pip`` +is installed by default, while when using the :mod:`venv` module +:ref:`venv-api` installation of ``pip`` must be requested explicitly. + +For CPython :ref:`source builds on POSIX systems `, +the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by +default. This behaviour can be controlled through configure options, and +overridden through Makefile options. On Windows and Mac OS X, the CPython installers now offer the option to install ``pip`` along with CPython itself. @@ -251,7 +252,13 @@ ----------------------------------------------------------- :pep:`446` makes newly created file descriptors :ref:`non-inheritable -`. New functions and methods: +`. In general, this is the behavior an application will +want: when launching a new process, having currently open files also +open in the new process can lead to all sorts of hard to find bugs, +and potentially to security issues. + +However, there are occasions when inheritance is desired. To support +these cases, the following new functions and methods are available: * :func:`os.get_inheritable`, :func:`os.set_inheritable` * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 01:55:23 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 01:55:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzc0NzU6IFJlbW92?= =?utf-8?q?e_references_to_=27=2Etransform=27_from_transform_codec_docstri?= =?utf-8?q?ngs=2E?= Message-ID: <3flT3H6hqCz7LjY@mail.python.org> http://hg.python.org/cpython/rev/d7950e916f20 changeset: 89647:d7950e916f20 branch: 3.3 parent: 89643:066b896b6132 user: R David Murray date: Thu Mar 13 20:54:30 2014 -0400 summary: #7475: Remove references to '.transform' from transform codec docstrings. files: Lib/encodings/base64_codec.py | 3 +-- Lib/encodings/hex_codec.py | 3 +-- Lib/encodings/quopri_codec.py | 3 +-- Lib/encodings/rot_13.py | 3 +-- Lib/encodings/uu_codec.py | 3 +-- Lib/encodings/zlib_codec.py | 3 +-- 6 files changed, 6 insertions(+), 12 deletions(-) diff --git a/Lib/encodings/base64_codec.py b/Lib/encodings/base64_codec.py --- a/Lib/encodings/base64_codec.py +++ b/Lib/encodings/base64_codec.py @@ -1,7 +1,6 @@ """Python 'base64_codec' Codec - base64 content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/hex_codec.py b/Lib/encodings/hex_codec.py --- a/Lib/encodings/hex_codec.py +++ b/Lib/encodings/hex_codec.py @@ -1,7 +1,6 @@ """Python 'hex_codec' Codec - 2-digit hex content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/quopri_codec.py b/Lib/encodings/quopri_codec.py --- a/Lib/encodings/quopri_codec.py +++ b/Lib/encodings/quopri_codec.py @@ -1,7 +1,6 @@ """Codec for quoted-printable encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. """ import codecs diff --git a/Lib/encodings/rot_13.py b/Lib/encodings/rot_13.py --- a/Lib/encodings/rot_13.py +++ b/Lib/encodings/rot_13.py @@ -1,8 +1,7 @@ #!/usr/bin/env python """ Python Character Mapping Codec for ROT13. -This codec de/encodes from str to str and is therefore usable with -str.transform() and str.untransform(). +This codec de/encodes from str to str. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/uu_codec.py b/Lib/encodings/uu_codec.py --- a/Lib/encodings/uu_codec.py +++ b/Lib/encodings/uu_codec.py @@ -1,7 +1,6 @@ """Python 'uu_codec' Codec - UU content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). Some details were adapted from uu.py which was written by Lance Ellinghouse and diff --git a/Lib/encodings/zlib_codec.py b/Lib/encodings/zlib_codec.py --- a/Lib/encodings/zlib_codec.py +++ b/Lib/encodings/zlib_codec.py @@ -1,7 +1,6 @@ """Python 'zlib_codec' Codec - zlib compression encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 01:55:25 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 01:55:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_=237475=3A_Remove_references_to_=27=2Etransform=27?= =?utf-8?q?_from_transform_codec_docstrings=2E?= Message-ID: <3flT3K0s5Sz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/83d54ab5c696 changeset: 89648:83d54ab5c696 parent: 89646:ee635e7d213d parent: 89647:d7950e916f20 user: R David Murray date: Thu Mar 13 20:55:09 2014 -0400 summary: Merge #7475: Remove references to '.transform' from transform codec docstrings. files: Lib/encodings/base64_codec.py | 3 +-- Lib/encodings/hex_codec.py | 3 +-- Lib/encodings/quopri_codec.py | 3 +-- Lib/encodings/rot_13.py | 3 +-- Lib/encodings/uu_codec.py | 3 +-- Lib/encodings/zlib_codec.py | 3 +-- 6 files changed, 6 insertions(+), 12 deletions(-) diff --git a/Lib/encodings/base64_codec.py b/Lib/encodings/base64_codec.py --- a/Lib/encodings/base64_codec.py +++ b/Lib/encodings/base64_codec.py @@ -1,7 +1,6 @@ """Python 'base64_codec' Codec - base64 content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/hex_codec.py b/Lib/encodings/hex_codec.py --- a/Lib/encodings/hex_codec.py +++ b/Lib/encodings/hex_codec.py @@ -1,7 +1,6 @@ """Python 'hex_codec' Codec - 2-digit hex content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/quopri_codec.py b/Lib/encodings/quopri_codec.py --- a/Lib/encodings/quopri_codec.py +++ b/Lib/encodings/quopri_codec.py @@ -1,7 +1,6 @@ """Codec for quoted-printable encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. """ import codecs diff --git a/Lib/encodings/rot_13.py b/Lib/encodings/rot_13.py --- a/Lib/encodings/rot_13.py +++ b/Lib/encodings/rot_13.py @@ -1,8 +1,7 @@ #!/usr/bin/env python """ Python Character Mapping Codec for ROT13. -This codec de/encodes from str to str and is therefore usable with -str.transform() and str.untransform(). +This codec de/encodes from str to str. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/uu_codec.py b/Lib/encodings/uu_codec.py --- a/Lib/encodings/uu_codec.py +++ b/Lib/encodings/uu_codec.py @@ -1,7 +1,6 @@ """Python 'uu_codec' Codec - UU content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). Some details were adapted from uu.py which was written by Lance Ellinghouse and diff --git a/Lib/encodings/zlib_codec.py b/Lib/encodings/zlib_codec.py --- a/Lib/encodings/zlib_codec.py +++ b/Lib/encodings/zlib_codec.py @@ -1,7 +1,6 @@ """Python 'zlib_codec' Codec - zlib compression encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 03:17:48 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 03:17:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_small_tweaks_t?= =?utf-8?q?o_codecs_entry=2E?= Message-ID: <3flVtN2JDPz7LjN@mail.python.org> http://hg.python.org/cpython/rev/6ddd2c2108f2 changeset: 89649:6ddd2c2108f2 user: R David Murray date: Thu Mar 13 20:56:27 2014 -0400 summary: whatsnew: small tweaks to codecs entry. files: Doc/whatsnew/3.4.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -289,10 +289,10 @@ but were previously only discoverable through runtime introspection. Unlike the convenience methods on :class:`str`, :class:`bytes` and -:class:`bytearray`, these convenience functions support arbitrary codecs -in both Python 2 and Python 3, rather than being limited to Unicode text -encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions -(in Python 2). +:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary +codecs in both Python 2 and Python 3, rather than being limited to Unicode text +encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in +Python 2). In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these @@ -315,7 +315,7 @@ In a related change, whenever it is feasible without breaking backwards compatibility, exceptions raised during encoding and decoding operations -will be wrapped in a chained exception of the same type that mentions the +are wrapped in a chained exception of the same type that mentions the name of the codec responsible for producing the error:: >>> import codecs @@ -361,7 +361,7 @@ The binary and text transforms provided in the standard library are detailed in :ref:`binary-transforms` and :ref:`text-transforms`. -(Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, +(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, :issue:`17828` and :issue:`19619`) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 03:17:49 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 03:17:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_difflib=2Eisbj?= =?utf-8?q?unk_=26c_were_removed=2C_not_deprecated=2E?= Message-ID: <3flVtP3ndtz7LjN@mail.python.org> http://hg.python.org/cpython/rev/46d2c9021ee9 changeset: 89650:46d2c9021ee9 user: R David Murray date: Thu Mar 13 21:34:54 2014 -0400 summary: whatsnew: difflib.isbjunk &c were removed, not deprecated. Also move NEWS item to correct position (it was in 3.3). files: Doc/whatsnew/3.4.rst | 10 ++++++---- Misc/NEWS | 5 +++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2051,10 +2051,6 @@ Deprecations in the Python API ------------------------------ -* :meth:`difflib.SequenceMatcher.isbjunk` and - :meth:`difflib.SequenceMatcher.isbpopular` were removed: use ``x in sm.bjunk`` and - ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object. - * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` methods and functions are deprecated: :meth:`importlib.find_loader` is replaced by :func:`importlib.util.find_spec`; @@ -2194,6 +2190,12 @@ that do not have a __format__ method that handles it. See :issue:`7994` for background. +* :meth:`difflib.SequenceMatcher.isbjunk` and + :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have + now been removed: use ``x in sm.bjunk`` and + ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object + (:issue:`13248`). + Code Cleanups ------------- diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3077,6 +3077,9 @@ - Issue #17467: add readline and readlines support to mock_open in unittest.mock. +- Issue #13248: removed deprecated and undocumented difflib.isbjunk, + isbpopular. + - Issue #17192: Update the ctypes module's libffi to v3.0.13. This specifically addresses a stack misalignment issue on x86 and issues on some more recent platforms. @@ -4512,8 +4515,6 @@ Library ------- -- Issue #13248: removed deprecated and undocumented difflib.isbjunk, isbpopular. - - Issue #13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 03:17:50 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 03:17:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_link_from_modu?= =?utf-8?q?le=5Fspec_section_to_Deprecation=2C_for_replacements=2E?= Message-ID: <3flVtQ5YCSz7LjX@mail.python.org> http://hg.python.org/cpython/rev/f2c257820a73 changeset: 89651:f2c257820a73 user: R David Murray date: Thu Mar 13 22:17:28 2014 -0400 summary: whatsnew: link from module_spec section to Deprecation, for replacements. files: Doc/whatsnew/3.4.rst | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -382,7 +382,9 @@ Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing -importers should be updated to implement the new methods. +importers should be updated to implement the new methods. See the +:ref:`deprecated-3.4` section for a list of methods that should be replaced and +their replacements. Other Language Changes @@ -2038,6 +2040,8 @@ multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) +.. _deprecated-3.4: + Deprecated ========== -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Mar 14 09:54:35 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 14 Mar 2014 09:54:35 +0100 Subject: [Python-checkins] Daily reference leaks (f2c257820a73): sum=8 Message-ID: results for f2c257820a73 on branch "default" -------------------------------------------- test_asyncio leaked [4, 0, 0] memory blocks, sum=4 test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 0, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogcZQGDQ', '-x'] From python-checkins at python.org Fri Mar 14 14:42:31 2014 From: python-checkins at python.org (vinay.sajip) Date: Fri, 14 Mar 2014 14:42:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2320918=3A_Added_h?= =?utf-8?q?andling_for_exceptions_during_fallback_output_of_logging?= Message-ID: <3flp4R3f8Rz7LjM@mail.python.org> http://hg.python.org/cpython/rev/73c2a70e4b35 changeset: 89652:73c2a70e4b35 user: Vinay Sajip date: Fri Mar 14 13:42:19 2014 +0000 summary: Closes #20918: Added handling for exceptions during fallback output of logging exceptions. files: Lib/logging/__init__.py | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -18,7 +18,7 @@ Logging package for Python. Based on PEP 282 and comments thereto in comp.lang.python. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ @@ -42,6 +42,7 @@ __author__ = "Vinay Sajip " __status__ = "production" +# The following module attributes are no longer updated. __version__ = "0.5.1.2" __date__ = "07 February 2010" @@ -902,8 +903,15 @@ sys.stderr.write('Logged from file %s, line %s\n' % ( record.filename, record.lineno)) # Issue 18671: output logging message and arguments - sys.stderr.write('Message: %r\n' - 'Arguments: %s\n' % (record.msg, record.args)) + try: + sys.stderr.write('Message: %r\n' + 'Arguments: %s\n' % (record.msg, + record.args)) + except Exception: + sys.stderr.write('Unable to print the message and arguments' + ' - possible formatting error.\nUse the' + ' traceback above to help find the error.\n' + ) except OSError: #pragma: no cover pass # see issue 5971 finally: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 14:46:42 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 14 Mar 2014 14:46:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320884=3A_whatsnew?= =?utf-8?q?=3A_Frozen_modules_don=27t_set_=5F=5Ffile=5F=5F_anymore=2E?= Message-ID: <3flp9G5Rk1z7LjN@mail.python.org> http://hg.python.org/cpython/rev/b626f4978a28 changeset: 89653:b626f4978a28 user: Brett Cannon date: Fri Mar 14 09:46:28 2014 -0400 summary: Issue #20884: whatsnew: Frozen modules don't set __file__ anymore. files: Doc/whatsnew/3.4.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2275,6 +2275,14 @@ to determine if a module is a package or not is to use``hasattr(module, '__path__')`` (:issue:`18065`). +* Frozen modules no longer define a ``__file__`` attribute. It's semantically + incorrect for frozen modules to set the attribute as they are not loaded from + any explicit location. If you must know that a module comes from frozen code + then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, + check if the loader is a subclass of + :class:`importlib.machinery.FrozenImporter`, + or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. + * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path it would write to is a symlink or a non-regular file. This is to act as a warning that import will overwrite those files with a regular file regardless -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 14 15:20:26 2014 From: python-checkins at python.org (vinay.sajip) Date: Fri, 14 Mar 2014 15:20:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Added_Doc/tools/_subdirs_t?= =?utf-8?q?o_=2Ehgignore=2E?= Message-ID: <3flpwB0Njyz7LjX@mail.python.org> http://hg.python.org/cpython/rev/edb12dad7bf6 changeset: 89654:edb12dad7bf6 user: Vinay Sajip date: Fri Mar 14 14:20:09 2014 +0000 summary: Added Doc/tools/ subdirs to .hgignore. files: .hgignore | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -24,6 +24,11 @@ reflog.txt$ tags$ Lib/plat-mac/errors.rsrc.df.rsrc +Doc/tools/sphinx/ +Doc/tools/docutils/ +Doc/tools/jinja/ +Doc/tools/jinja2/ +Doc/tools/pygments/ Misc/python.pc Misc/python-config.sh$ Modules/Setup$ -- Repository URL: http://hg.python.org/cpython From benjamin at python.org Fri Mar 14 16:10:24 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 14 Mar 2014 08:10:24 -0700 Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <3flpwB0Njyz7LjX@mail.python.org> References: <3flpwB0Njyz7LjX@mail.python.org> Message-ID: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > http://hg.python.org/cpython/rev/edb12dad7bf6 > changeset: 89654:edb12dad7bf6 > user: Vinay Sajip > date: Fri Mar 14 14:20:09 2014 +0000 > summary: > Added Doc/tools/ subdirs to .hgignore. > > files: > .hgignore | 5 +++++ > 1 files changed, 5 insertions(+), 0 deletions(-) > > > diff --git a/.hgignore b/.hgignore > --- a/.hgignore > +++ b/.hgignore > @@ -24,6 +24,11 @@ > reflog.txt$ > tags$ > Lib/plat-mac/errors.rsrc.df.rsrc > +Doc/tools/sphinx/ > +Doc/tools/docutils/ > +Doc/tools/jinja/ > +Doc/tools/jinja2/ > +Doc/tools/pygments/ Why are you readding these when they were apparently purposely removed in eef7899ea7ab? From zachary.ware+pydev at gmail.com Fri Mar 14 16:52:28 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 14 Mar 2014 10:52:28 -0500 Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson wrote: > On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: >> diff --git a/.hgignore b/.hgignore >> --- a/.hgignore >> +++ b/.hgignore >> @@ -24,6 +24,11 @@ >> reflog.txt$ >> tags$ >> Lib/plat-mac/errors.rsrc.df.rsrc >> +Doc/tools/sphinx/ >> +Doc/tools/docutils/ >> +Doc/tools/jinja/ >> +Doc/tools/jinja2/ >> +Doc/tools/pygments/ > > Why are you readding these when they were apparently purposely removed > in eef7899ea7ab? It is rather ugly to build the docs in a 3.3 checkout, then update to default and run 'hg status' without those entries in .hgignore. I can understand not wanting to be bombarded by all of the untracked files, but it's also pretty easy to enable the purge extension and do "hg purge Doc/tools". -- Zach From benjamin at python.org Fri Mar 14 16:55:30 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 14 Mar 2014 08:55:30 -0700 Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: > On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson > wrote: > > On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > >> diff --git a/.hgignore b/.hgignore > >> --- a/.hgignore > >> +++ b/.hgignore > >> @@ -24,6 +24,11 @@ > >> reflog.txt$ > >> tags$ > >> Lib/plat-mac/errors.rsrc.df.rsrc > >> +Doc/tools/sphinx/ > >> +Doc/tools/docutils/ > >> +Doc/tools/jinja/ > >> +Doc/tools/jinja2/ > >> +Doc/tools/pygments/ > > > > Why are you readding these when they were apparently purposely removed > > in eef7899ea7ab? > > It is rather ugly to build the docs in a 3.3 checkout, then update to > default and run 'hg status' without those entries in .hgignore. I can > understand not wanting to be bombarded by all of the untracked files, > but it's also pretty easy to enable the purge extension and do "hg > purge Doc/tools". This is why everyone should use the share extension and have separate working copies for every branch. :) From python-checkins at python.org Fri Mar 14 17:40:49 2014 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 14 Mar 2014 17:40:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Assign_PEP_=23465_to_=22Dedic?= =?utf-8?q?ated_infix_operators_for_matrix_multiplication_and?= Message-ID: <3flt294MxPz7Ljp@mail.python.org> http://hg.python.org/peps/rev/733c83581fbe changeset: 5406:733c83581fbe user: Guido van Rossum date: Fri Mar 14 09:40:42 2014 -0700 summary: Assign PEP #465 to "Dedicated infix operators for matrix multiplication and matrix power" by Nathaniel J. Smith. files: pep-0465.txt | 1223 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 1223 insertions(+), 0 deletions(-) diff --git a/pep-0465.txt b/pep-0465.txt new file mode 100644 --- /dev/null +++ b/pep-0465.txt @@ -0,0 +1,1223 @@ +PEP: 465 +Title: Dedicated infix operators for matrix multiplication and matrix power +Version: $Revision$ +Last-Modified: $Date$ +Author: Nathaniel J. Smith +Status: Draft +Type: Standards Track +Python-Version: 3.5 +Content-Type: text/x-rst +Created: 20-Feb-2014 +Post-History: 13-Mar-2014 + +Abstract +======== + +This PEP proposes two new binary operators dedicated to matrix +multiplication and matrix power, spelled ``@`` and ``@@`` +respectively. (Mnemonic: ``@`` is ``*`` for mATrices.) + + +Specification +============= + +Two new binary operators are added to the Python language, together +with corresponding in-place versions: + +======= ========================= =============================== + Op Precedence/associativity Methods +======= ========================= =============================== +``@`` Same as ``*`` ``__matmul__``, ``__rmatmul__`` +``@@`` Same as ``**`` ``__matpow__``, ``__rmatpow__`` +``@=`` n/a ``__imatmul__`` +``@@=`` n/a ``__imatpow__`` +======= ========================= =============================== + +No implementations of these methods are added to the builtin or +standard library types. However, a number of projects have reached +consensus on the recommended semantics for these operations; see +`Intended usage details`_ below. + + +Motivation +========== + +Executive summary +----------------- + +In numerical code, there are two important operations which compete +for use of Python's ``*`` operator: elementwise multiplication, and +matrix multiplication. In the nearly twenty years since the Numeric +library was first proposed, there have been many attempts to resolve +this tension [#hugunin]_; none have been really satisfactory. +Currently, most numerical Python code uses ``*`` for elementwise +multiplication, and function/method syntax for matrix multiplication; +however, this leads to ugly and unreadable code in common +circumstances. The problem is bad enough that significant amounts of +code continue to use the opposite convention (which has the virtue of +producing ugly and unreadable code in *different* circumstances), and +this API fragmentation across codebases then creates yet more +problems. There does not seem to be any *good* solution to the +problem of designing a numerical API within current Python syntax -- +only a landscape of options that are bad in different ways. The +minimal change to Python syntax which is sufficient to resolve these +problems is the addition of a single new infix operator for matrix +multiplication. + +Matrix multiplication has a singular combination of features which +distinguish it from other binary operations, which together provide a +uniquely compelling case for the addition of a dedicated infix +operator: + +* Just as for the existing numerical operators, there exists a vast + body of prior art supporting the use of infix notation for matrix + multiplication across all fields of mathematics, science, and + engineering; ``@`` harmoniously fills a hole in Python's existing + operator system. + +* ``@`` greatly clarifies real-world code. + +* ``@`` provides a smoother onramp for less experienced users, who are + particularly harmed by hard-to-read code and API fragmentation. + +* ``@`` benefits a substantial and growing portion of the Python user + community. + +* ``@`` will be used frequently -- in fact, evidence suggests it may + be used more frequently than ``//`` or the bitwise operators. + +* ``@`` allows the Python numerical community to reduce fragmentation, + and finally standardize on a single consensus duck type for all + numerical array objects. + +And, given the existence of ``@``, it makes more sense than not to +have ``@@``, ``@=``, and ``@@=``, so they are added as well. + + +Background: What's wrong with the status quo? +--------------------------------------------- + +When we crunch numbers on a computer, we usually have lots and lots of +numbers to deal with. Trying to deal with them one at a time is +cumbersome and slow -- especially when using an interpreted language. +Instead, we want the ability to write down simple operations that +apply to large collections of numbers all at once. The *n-dimensional +array* is the basic object that all popular numeric computing +environments use to make this possible. Python has several libraries +that provide such arrays, with numpy being at present the most +prominent. + +When working with n-dimensional arrays, there are two different ways +we might want to define multiplication. One is elementwise +multiplication:: + + [[1, 2], [[11, 12], [[1 * 11, 2 * 12], + [3, 4]] x [13, 14]] = [3 * 13, 4 * 14]] + +and the other is `matrix multiplication`_: + +.. _matrix multiplication: https://en.wikipedia.org/wiki/Matrix_multiplication + +:: + + [[1, 2], [[11, 12], [[1 * 11 + 2 * 13, 1 * 12 + 2 * 14], + [3, 4]] x [13, 14]] = [3 * 11 + 4 * 13, 3 * 12 + 4 * 14]] + +Elementwise multiplication is useful because it lets us easily and +quickly perform many multiplications on a large collection of values, +without writing a slow and cumbersome ``for`` loop. And this works as +part of a very general schema: when using the array objects provided +by numpy or other numerical libraries, all Python operators work +elementwise on arrays of all dimensionalities. The result is that one +can write functions using straightforward code like ``a * b + c / d``, +treating the variables as if they were simple values, but then +immediately use this function to efficiently perform this calculation +on large collections of values, while keeping them organized using +whatever arbitrarily complex array layout works best for the problem +at hand. + +Matrix multiplication is more of a special case. It's only defined on +2d arrays (also known as "matrices"), and multiplication is the only +operation that has a meaningful "matrix" version -- "matrix addition" +is the same as elementwise addition; there is no such thing as "matrix +bitwise-or" or "matrix floordiv"; "matrix division" can be defined but +is not very useful, etc. However, matrix multiplication is still used +very heavily across all numerical application areas; mathematically, +it's one of the most fundamental operations there is. + +Because Python syntax currently allows for only a single +multiplication operator ``*``, libraries providing array-like objects +must decide: either use ``*`` for elementwise multiplication, or use +``*`` for matrix multiplication. And, unfortunately, it turns out +that when doing general-purpose number crunching, both operations are +used frequently, and there are major advantages to using infix rather +than function call syntax in both cases. Thus it is not at all clear +which convention is optimal, or even acceptable; often it varies on a +case-by-case basis. + +Nonetheless, network effects mean that it is very important that we +pick *just one* convention. In numpy, for example, it is technically +possible to switch between the conventions, because numpy provides two +different types with different ``__mul__`` methods. For +``numpy.ndarray`` objects, ``*`` performs elementwise multiplication, +and matrix multiplication must use a function call (``numpy.dot``). +For ``numpy.matrix`` objects, ``*`` performs matrix multiplication, +and elementwise multiplication requires function syntax. Writing code +using ``numpy.ndarray`` works fine. Writing code using +``numpy.matrix`` also works fine. But trouble begins as soon as we +try to integrate these two pieces of code together. Code that expects +an ``ndarray`` and gets a ``matrix``, or vice-versa, may crash or +return incorrect results. Keeping track of which functions expect +which types as inputs, and return which types as outputs, and then +converting back and forth all the time, is incredibly cumbersome and +impossible to get right at any scale. Functions that defensively try +to handle both types as input and DTRT, find themselves floundering +into a swamp of ``isinstance`` and ``if`` statements. + +PEP 238 split ``/`` into two operators: ``/`` and ``//``. Imagine the +chaos that would have resulted if it had instead split ``int`` into +two types: ``classic_int``, whose ``__div__`` implemented floor +division, and ``new_int``, whose ``__div__`` implemented true +division. This, in a more limited way, is the situation that Python +number-crunchers currently find themselves in. + +In practice, the vast majority of projects have settled on the +convention of using ``*`` for elementwise multiplication, and function +call syntax for matrix multiplication (e.g., using ``numpy.ndarray`` +instead of ``numpy.matrix``). This reduces the problems caused by API +fragmentation, but it doesn't eliminate them. The strong desire to +use infix notation for matrix multiplication has caused a number of +specialized array libraries to continue to use the opposing convention +(e.g., scipy.sparse, pyoperators, pyviennacl) despite the problems +this causes, and ``numpy.matrix`` itself still gets used in +introductory programming courses, often appears in StackOverflow +answers, and so forth. Well-written libraries thus must continue to +be prepared to deal with both types of objects, and, of course, are +also stuck using unpleasant funcall syntax for matrix multiplication. +After nearly two decades of trying, the numerical community has still +not found any way to resolve these problems within the constraints of +current Python syntax (see `Rejected alternatives to adding a new +operator`_ below). + +This PEP proposes the minimum effective change to Python syntax that +will allow us to drain this swamp. It splits ``*`` into two +operators, just as was done for ``/``: ``*`` for elementwise +multiplication, and ``@`` for matrix multiplication. (Why not the +reverse? Because this way is compatible with the existing consensus, +and because it gives us a consistent rule that all the built-in +numeric operators also apply in an elementwise manner to arrays; the +reverse convention would lead to more special cases.) + +So that's why matrix multiplication doesn't and can't just use ``*``. +Now, in the the rest of this section, we'll explain why it nonetheless +meets the high bar for adding a new operator. + + +Why should matrix multiplication be infix? +------------------------------------------ + +Right now, most numerical code in Python uses syntax like +``numpy.dot(a, b)`` or ``a.dot(b)`` to perform matrix multiplication. +This obviously works, so why do people make such a fuss about it, even +to the point of creating API fragmentation and compatibility swamps? + +Matrix multiplication shares two features with ordinary arithmetic +operations like addition and multiplication on numbers: (a) it is used +very heavily in numerical programs -- often multiple times per line of +code -- and (b) it has an ancient and universally adopted tradition of +being written using infix syntax. This is because, for typical +formulas, this notation is dramatically more readable than any +function call syntax. Here's an example to demonstrate: + +One of the most useful tools for testing a statistical hypothesis is +the linear hypothesis test for OLS regression models. It doesn't +really matter what all those words I just said mean; if we find +ourselves having to implement this thing, what we'll do is look up +some textbook or paper on it, and encounter many mathematical formulas +that look like: + +.. math:: + + S = (H \beta - r)^T (H V H^T)^{-1} (H \beta - r) + +Here the various variables are all vectors or matrices (details for +the curious: [#lht]_). + +Now we need to write code to perform this calculation. In current +numpy, matrix multiplication can be performed using either the +function or method call syntax. Neither provides a particularly +readable translation of the formula:: + + import numpy as np + from numpy.linalg import inv, solve + + # Using dot function: + S = np.dot((np.dot(H, beta) - r).T, + np.dot(inv(np.dot(np.dot(H, V), H.T)), np.dot(H, beta) - r)) + + # Using dot method: + S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r) + +With the ``@`` operator, the direct translation of the above formula +becomes:: + + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + +Notice that there is now a transparent, 1-to-1 mapping between the +symbols in the original formula and the code that implements it. + +Of course, an experienced programmer will probably notice that this is +not the best way to compute this expression. The repeated computation +of :math:`H \beta - r` should perhaps be factored out; and, +expressions of the form ``dot(inv(A), B)`` should almost always be +replaced by the more numerically stable ``solve(A, B)``. When using +``@``, performing these two refactorings gives us:: + + # Version 1 (as above) + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + + # Version 2 + trans_coef = H @ beta - r + S = trans_coef.T @ inv(H @ V @ H.T) @ trans_coef + + # Version 3 + S = trans_coef.T @ solve(H @ V @ H.T, trans_coef) + +Notice that when comparing between each pair of steps, it's very easy +to see exactly what was changed. If we apply the equivalent +transformations to the code using the .dot method, then the changes +are much harder to read out or verify for correctness:: + + # Version 1 (as above) + S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r) + + # Version 2 + trans_coef = H.dot(beta) - r + S = trans_coef.T.dot(inv(H.dot(V).dot(H.T))).dot(trans_coef) + + # Version 3 + S = trans_coef.T.dot(solve(H.dot(V).dot(H.T)), trans_coef) + +Readability counts! The statements using ``@`` are shorter, contain +more whitespace, can be directly and easily compared both to each +other and to the textbook formula, and contain only meaningful +parentheses. This last point is particularly important for +readability: when using function-call syntax, the required parentheses +on every operation create visual clutter that makes it very difficult +to parse out the overall structure of the formula by eye, even for a +relatively simple formula like this one. Eyes are terrible at parsing +non-regular languages. I made and caught many errors while trying to +write out the 'dot' formulas above. I know they still contain at +least one error, maybe more. (Exercise: find it. Or them.) The +``@`` examples, by contrast, are not only correct, they're obviously +correct at a glance. + +If we are even more sophisticated programmers, and writing code that +we expect to be reused, then considerations of speed or numerical +accuracy might lead us to prefer some particular order of evaluation. +Because ``@`` makes it possible to omit irrelevant parentheses, we can +be certain that if we *do* write something like ``(H @ V) @ H.T``, +then our readers will know that the parentheses must have been added +intentionally to accomplish some meaningful purpose. In the ``dot`` +examples, it's impossible to know which nesting decisions are +important, and which are arbitrary. + +Infix ``@`` dramatically improves matrix code usability at all stages +of programmer interaction. + + +Transparent syntax is especially crucial for non-expert programmers +------------------------------------------------------------------- + +A large proportion of scientific code is written by people who are +experts in their domain, but are not experts in programming. And +there are many university courses run each year with titles like "Data +analysis for social scientists" which assume no programming +background, and teach some combination of mathematical techniques, +introduction to programming, and the use of programming to implement +these mathematical techniques, all within a 10-15 week period. These +courses are more and more often being taught in Python rather than +special-purpose languages like R or Matlab. + +For these kinds of users, whose programming knowledge is fragile, the +existence of a transparent mapping between formulas and code often +means the difference between succeeding and failing to write that code +at all. This is so important that such classes often use the +``numpy.matrix`` type which defines ``*`` to mean matrix +multiplication, even though this type is buggy and heavily +disrecommended by the rest of the numpy community for the +fragmentation that it causes. This pedagogical use case is, in fact, +the *only* reason ``numpy.matrix`` remains a supported part of numpy. +Adding ``@`` will benefit both beginning and advanced users with +better syntax; and furthermore, it will allow both groups to +standardize on the same notation from the start, providing a smoother +on-ramp to expertise. + + +But isn't matrix multiplication a pretty niche requirement? +----------------------------------------------------------- + +The world is full of continuous data, and computers are increasingly +called upon to work with it in sophisticated ways. Arrays are the +lingua franca of finance, machine learning, 3d graphics, computer +vision, robotics, operations research, econometrics, meteorology, +computational linguistics, recommendation systems, neuroscience, +astronomy, bioinformatics (including genetics, cancer research, drug +discovery, etc.), physics engines, quantum mechanics, geophysics, +network analysis, and many other application areas. In most or all of +these areas, Python is rapidly becoming a dominant player, in large +part because of its ability to elegantly mix traditional discrete data +structures (hash tables, strings, etc.) on an equal footing with +modern numerical data types and algorithms. + +We all live in our own little sub-communities, so some Python users +may be surprised to realize the sheer extent to which Python is used +for number crunching -- especially since much of this particular +sub-community's activity occurs outside of traditional Python/FOSS +channels. So, to give some rough idea of just how many numerical +Python programmers are actually out there, here are two numbers: In +2013, there were 7 international conferences organized specifically on +numerical Python [#scipy-conf]_ [#pydata-conf]_. At PyCon 2014, ~20% +of the tutorials appear to involve the use of matrices +[#pycon-tutorials]_. + +To quantify this further, we used Github's "search" function to look +at what modules are actually imported across a wide range of +real-world code (i.e., all the code on Github). We checked for +imports of several popular stdlib modules, a variety of numerically +oriented modules, and various other extremely high-profile modules +like django and lxml (the latter of which is the #1 most downloaded +package on PyPI). Starred lines indicate packages which export array- +or matrix-like objects which will adopt ``@`` if this PEP is +approved:: + + Count of Python source files on Github matching given search terms + (as of 2014-04-10, ~21:00 UTC) + ================ ========== =============== ======= =========== + module "import X" "from X import" total total/numpy + ================ ========== =============== ======= =========== + sys 2374638 63301 2437939 5.85 + os 1971515 37571 2009086 4.82 + re 1294651 8358 1303009 3.12 + numpy ************** 337916 ********** 79065 * 416981 ******* 1.00 + warnings 298195 73150 371345 0.89 + subprocess 281290 63644 344934 0.83 + django 62795 219302 282097 0.68 + math 200084 81903 281987 0.68 + threading 212302 45423 257725 0.62 + pickle+cPickle 215349 22672 238021 0.57 + matplotlib 119054 27859 146913 0.35 + sqlalchemy 29842 82850 112692 0.27 + pylab *************** 36754 ********** 41063 ** 77817 ******* 0.19 + scipy *************** 40829 ********** 28263 ** 69092 ******* 0.17 + lxml 19026 38061 57087 0.14 + zlib 40486 6623 47109 0.11 + multiprocessing 25247 19850 45097 0.11 + requests 30896 560 31456 0.08 + jinja2 8057 24047 32104 0.08 + twisted 13858 6404 20262 0.05 + gevent 11309 8529 19838 0.05 + pandas ************** 14923 *********** 4005 ** 18928 ******* 0.05 + sympy 2779 9537 12316 0.03 + theano *************** 3654 *********** 1828 *** 5482 ******* 0.01 + ================ ========== =============== ======= =========== + +These numbers should be taken with several grains of salt (see +footnote for discussion: [#github-details]_), but, to the extent they +can be trusted, they suggest that ``numpy`` might be the single +most-imported non-stdlib module in the entire Pythonverse; it's even +more-imported than such stdlib stalwarts as ``subprocess``, ``math``, +``pickle``, and ``threading``. And numpy users represent only a +subset of the broader numerical community that will benefit from the +``@`` operator. Matrices may once have been a niche data type +restricted to Fortran programs running in university labs and military +clusters, but those days are long gone. Number crunching is a +mainstream part of modern Python usage. + +In addition, there is some precedence for adding an infix operator to +handle a more-specialized arithmetic operation: the floor division +operator ``//``, like the bitwise operators, is very useful under +certain circumstances when performing exact calculations on discrete +values. But it seems likely that there are many Python programmers +who have never had reason to use ``//`` (or, for that matter, the +bitwise operators). ``@`` is no more niche than ``//``. + + +So ``@`` is good for matrix formulas, but how common are those really? +---------------------------------------------------------------------- + +We've seen that ``@`` makes matrix formulas dramatically easier to +work with for both experts and non-experts, that matrix formulas +appear in many important applications, and that numerical libraries +like numpy are used by a substantial proportion of Python's user base. +But numerical libraries aren't just about matrix formulas, and being +important doesn't necessarily mean taking up a lot of code: if matrix +formulas only occured in one or two places in the average +numerically-oriented project, then it still wouldn't be worth adding a +new operator. So how common is matrix multiplication, really? + +When the going gets tough, the tough get empirical. To get a rough +estimate of how useful the ``@`` operator will be, the table below +shows the rate at which different Python operators are actually used +in the stdlib, and also in two high-profile numerical packages -- the +scikit-learn machine learning library, and the nipy neuroimaging +library -- normalized by source lines of code (SLOC). Rows are sorted +by the 'combined' column, which pools all three code bases together. +The combined column is thus strongly weighted towards the stdlib, +which is much larger than both projects put together (stdlib: 411575 +SLOC, scikit-learn: 50924 SLOC, nipy: 37078 SLOC). [#sloc-details]_ + +The ``dot`` row (marked ``******``) counts how common matrix multiply +operations are in each codebase. + +:: + + ==== ====== ============ ==== ======== + op stdlib scikit-learn nipy combined + ==== ====== ============ ==== ======== + = 2969 5536 4932 3376 / 10,000 SLOC + - 218 444 496 261 + + 224 201 348 231 + == 177 248 334 196 + * 156 284 465 192 + % 121 114 107 119 + ** 59 111 118 68 + != 40 56 74 44 + / 18 121 183 41 + > 29 70 110 39 + += 34 61 67 39 + < 32 62 76 38 + >= 19 17 17 18 + <= 18 27 12 18 + dot ***** 0 ********** 99 ** 74 ****** 16 + | 18 1 2 15 + & 14 0 6 12 + << 10 1 1 8 + // 9 9 1 8 + -= 5 21 14 8 + *= 2 19 22 5 + /= 0 23 16 4 + >> 4 0 0 3 + ^ 3 0 0 3 + ~ 2 4 5 2 + |= 3 0 0 2 + &= 1 0 0 1 + //= 1 0 0 1 + ^= 1 0 0 0 + **= 0 2 0 0 + %= 0 0 0 0 + <<= 0 0 0 0 + >>= 0 0 0 0 + ==== ====== ============ ==== ======== + +These two numerical packages alone contain ~780 uses of matrix +multiplication. Within these packages, matrix multiplication is used +more heavily than most comparison operators (``<`` ``!=`` ``<=`` +``>=``). Even when we dilute these counts by including the stdlib +into our comparisons, matrix multiplication is still used more often +in total than any of the bitwise operators, and 2x as often as ``//``. +This is true even though the stdlib, which contains a fair amount of +integer arithmetic and no matrix operations, makes up more than 80% of +the combined code base. + +By coincidence, the numeric libraries make up approximately the same +proportion of the 'combined' codebase as numeric tutorials make up of +PyCon 2014's tutorial schedule, which suggests that the 'combined' +column may not be *wildly* unrepresentative of new Python code in +general. While it's impossible to know for certain, from this data it +seems entirely possible that across all Python code currently being +written, matrix multiplication is already used more often than ``//`` +and the bitwise operations. + + +But isn't it weird to add an operator with no stdlib uses? +---------------------------------------------------------- + +It's certainly unusual (though ``Ellipsis`` was also added without any +stdlib uses). But the important thing is whether a change will +benefit users, not where the software is being downloaded from. It's +clear from the above that ``@`` will be used, and used heavily. And +this PEP provides the critical piece that will allow the Python +numerical community to finally reach consensus on a standard duck type +for all array-like objects, which is a necessary precondition to ever +adding a numerical array type to the stdlib. + + +Matrix power and in-place operators +----------------------------------- + +The primary motivation for this PEP is ``@``; the other proposed +operators don't have nearly as much impact. The matrix power operator +``@@`` is useful and well-defined, but not really necessary. It is +still included, though, for consistency: if we have an ``@`` that is +analogous to ``*``, then it would be weird and surprising to *not* +have an ``@@`` that is analogous to ``**``. Similarly, the in-place +operators ``@=`` and ``@@=`` provide limited value -- it's more common +to write ``a = (b @ a)`` than it is to write ``a = (a @ b)``, and +in-place matrix operations still generally have to allocate +substantial temporary storage -- but they are included for +completeness and symmetry. + + +Compatibility considerations +============================ + +Currently, the only legal use of the ``@`` token in Python code is at +statement beginning in decorators. The new operators are all infix; +the one place they can never occur is at statement beginning. +Therefore, no existing code will be broken by the addition of these +operators, and there is no possible parsing ambiguity between +decorator-@ and the new operators. + +Another important kind of compatibility is the mental cost paid by +users to update their understanding of the Python language after this +change, particularly for users who do not work with matrices and thus +do not benefit. Here again, ``@`` has minimal impact: even +comprehensive tutorials and references will only need to add a +sentence or two to fully document this PEP's changes for a +non-numerical audience. + + +Intended usage details +====================== + +This section is informative, rather than normative -- it documents the +consensus of a number of libraries that provide array- or matrix-like +objects on how the ``@`` and ``@@`` operators will be implemented. + +This section uses the numpy terminology for describing arbitrary +multidimensional arrays of data, because it is a superset of all other +commonly used models. In this model, the *shape* of any array is +represented by a tuple of integers. Because matrices are +two-dimensional, they have len(shape) == 2, while 1d vectors have +len(shape) == 1, and scalars have shape == (), i.e., they are "0 +dimensional". Any array contains prod(shape) total entries. Notice +that `prod(()) == 1`_ (for the same reason that sum(()) == 0); scalars +are just an ordinary kind of array, not a special case. Notice also +that we distinguish between a single scalar value (shape == (), +analogous to ``1``), a vector containing only a single entry (shape == +(1,), analogous to ``[1]``), a matrix containing only a single entry +(shape == (1, 1), analogous to ``[[1]]``), etc., so the dimensionality +of any array is always well-defined. Other libraries with more +restricted representations (e.g., those that support 2d arrays only) +might implement only a subset of the functionality described here. + +.. _prod(()) == 1: https://en.wikipedia.org/wiki/Empty_product + +Semantics +--------- + +The recommended semantics for ``@`` for different inputs are: + +* 2d inputs are conventional matrices, and so the semantics are + obvious: we apply conventional matrix multiplication. If we write + ``arr(2, 3)`` to represent an arbitrary 2x3 array, then ``arr(3, 4) + @ arr(4, 5)`` returns an array with shape (3, 5). + +* 1d vector inputs are promoted to 2d by prepending or appending a '1' + to the shape, the operation is performed, and then the added + dimension is removed from the output. The 1 is always added on the + "outside" of the shape: prepended for left arguments, and appended + for right arguments. The result is that matrix @ vector and vector + @ matrix are both legal (assuming compatible shapes), and both + return 1d vectors; vector @ vector returns a scalar. This is + clearer with examples. + + * ``arr(2, 3) @ arr(3, 1)`` is a regular matrix product, and returns + an array with shape (2, 1), i.e., a column vector. + + * ``arr(2, 3) @ arr(3)`` performs the same computation as the + previous (i.e., treats the 1d vector as a matrix containing a + single *column*, shape = (3, 1)), but returns the result with + shape (2,), i.e., a 1d vector. + + * ``arr(1, 3) @ arr(3, 2)`` is a regular matrix product, and returns + an array with shape (1, 2), i.e., a row vector. + + * ``arr(3) @ arr(3, 2)`` performs the same computation as the + previous (i.e., treats the 1d vector as a matrix containing a + single *row*, shape = (1, 3)), but returns the result with shape + (2,), i.e., a 1d vector. + + * ``arr(1, 3) @ arr(3, 1)`` is a regular matrix product, and returns + an array with shape (1, 1), i.e., a single value in matrix form. + + * ``arr(3) @ arr(3)`` performs the same computation as the + previous, but returns the result with shape (), i.e., a single + scalar value, not in matrix form. So this is the standard inner + product on vectors. + + An infelicity of this definition for 1d vectors is that it makes + ``@`` non-associative in some cases (``(Mat1 @ vec) @ Mat2`` != + ``Mat1 @ (vec @ Mat2)``). But this seems to be a case where + practicality beats purity: non-associativity only arises for strange + expressions that would never be written in practice; if they are + written anyway then there is a consistent rule for understanding + what will happen (``Mat1 @ vec @ Mat2`` is parsed as ``(Mat1 @ vec) + @ Mat2``, just like ``a - b - c``); and, not supporting 1d vectors + would rule out many important use cases that do arise very commonly + in practice. No-one wants to explain to new users why to solve the + simplest linear system in the obvious way, they have to type + ``(inv(A) @ b[:, np.newaxis]).flatten()`` instead of ``inv(A) @ b``, + or perform an ordinary least-squares regression by typing + ``solve(X.T @ X, X @ y[:, np.newaxis]).flatten()`` instead of + ``solve(X.T @ X, X @ y)``. No-one wants to type ``(a[np.newaxis, :] + @ b[:, np.newaxis])[0, 0]`` instead of ``a @ b`` every time they + compute an inner product, or ``(a[np.newaxis, :] @ Mat @ b[:, + np.newaxis])[0, 0]`` for general quadratic forms instead of ``a @ + Mat @ b``. In addition, sage and sympy (see below) use these + non-associative semantics with an infix matrix multiplication + operator (they use ``*``), and they report that they haven't + experienced any problems caused by it. + +* For inputs with more than 2 dimensions, we treat the last two + dimensions as being the dimensions of the matrices to multiply, and + 'broadcast' across the other dimensions. This provides a convenient + way to quickly compute many matrix products in a single operation. + For example, ``arr(10, 2, 3) @ arr(10, 3, 4)`` performs 10 separate + matrix multiplies, each of which multiplies a 2x3 and a 3x4 matrix + to produce a 2x4 matrix, and then returns the 10 resulting matrices + together in an array with shape (10, 2, 4). The intuition here is + that we treat these 3d arrays of numbers as if they were 1d arrays + *of matrices*, and then apply matrix multiplication in an + elementwise manner, where now each 'element' is a whole matrix. + Note that broadcasting is not limited to perfectly aligned arrays; + in more complicated cases, it allows several simple but powerful + tricks for controlling how arrays are aligned with each other; see + [#broadcasting]_ for details. (In particular, it turns out that + when broadcasting is taken into account, the standard scalar * + matrix product is a special case of the elementwise multiplication + operator ``*``.) + + If one operand is >2d, and another operand is 1d, then the above + rules apply unchanged, with 1d->2d promotion performed before + broadcasting. E.g., ``arr(10, 2, 3) @ arr(3)`` first promotes to + ``arr(10, 2, 3) @ arr(3, 1)``, then broadcasts the right argument to + create the aligned operation ``arr(10, 2, 3) @ arr(10, 3, 1)``, + multiplies to get an array with shape (10, 2, 1), and finally + removes the added dimension, returning an array with shape (10, 2). + Similarly, ``arr(2) @ arr(10, 2, 3)`` produces an intermediate array + with shape (10, 1, 3), and a final array with shape (10, 3). + +* 0d (scalar) inputs raise an error. Scalar * matrix multiplication + is a mathematically and algorithmically distinct operation from + matrix @ matrix multiplication, and is already covered by the + elementwise ``*`` operator. Allowing scalar @ matrix would thus + both require an unnecessary special case, and violate TOOWTDI. + +The recommended semantics for ``@@`` are:: + + def __matpow__(self, n): + if not isinstance(n, numbers.Integral): + raise TypeError("@@ not implemented for fractional powers") + if n == 0: + return identity_matrix_with_shape(self.shape) + elif n < 0: + return inverse(self) @ (self @@ (n + 1)) + else: + return self @ (self @@ (n - 1)) + +(Of course we expect that much more efficient implementations will be +used in practice.) Notice that if given an appropriate definition of +``identity_matrix_with_shape``, then this definition will +automatically handle >2d arrays appropriately. Notice also that with +this definition, ``vector @@ 2`` gives the squared Euclidean length of +the vector, a commonly used value. Also, while it is rarely useful to +explicitly compute inverses or other negative powers in standard +immediate-mode dense matrix code, these computations are natural when +doing symbolic or deferred-mode computations (as in e.g. sympy, +theano, numba, numexpr); therefore, negative powers are fully +supported. Fractional powers, though, bring in variety of +`mathematical complications`_, so we leave it to individual projects +to decide whether they want to try to define some reasonable semantics +for fractional inputs. + +.. _`mathematical complications`: +https://en.wikipedia.org/wiki/Square_root_of_a_matrix + + +Adoption +-------- + +We group existing Python projects which provide array- or matrix-like +types based on what API they currently use for elementwise and matrix +multiplication. + +**Projects which currently use * for *elementwise* multiplication, and +function/method calls for *matrix* multiplication:** + +The developers of the following projects have expressed an intention +to implement ``@`` and ``@@`` on their array-like types using the +above semantics: + +* numpy +* pandas +* blaze +* theano + +The following projects have been alerted to the existence of the PEP, +but it's not yet known what they plan to do if it's accepted. We +don't anticipate that they'll have any objections, though, since +everything proposed here is consistent with how they already do +things: + +* pycuda +* panda3d + +**Projects which currently use * for *matrix* multiplication, and +function/method calls for *elementwise* multiplication:** + +The following projects have expressed an intention, if this PEP is +accepted, to migrate from their current API to the elementwise-``*``, +matmul-``@`` convention (i.e., this is a list of projects whose API +fragmentation will probably be eliminated if this PEP is accepted): + +* numpy (``numpy.matrix``) +* scipy.sparse +* pyoperators +* pyviennacl + +The following projects have been alerted to the existence of the PEP, +but it's not known what they plan to do if it's accepted (i.e., this +is a list of projects whose API fragmentation may or may not be +eliminated if this PEP is accepted): + +* cvxopt + +**Projects which currently use * for *matrix* multiplication, and +which do not implement elementwise multiplication at all:** + +There are several projects which implement matrix types, but from a +very different perspective than the numerical libraries discussed +above. These projects focus on computational methods for analyzing +matrices in the sense of abstract mathematical objects (i.e., linear +maps over free modules over rings), rather than as big bags full of +numbers that need crunching. And it turns out that from the abstract +math point of view, there isn't much use for elementwise operations in +the first place; as discussed in the Background section above, +elementwise operations are motivated by the bag-of-numbers approach. +The different goals of these projects mean that they don't encounter +the basic problem that this PEP exists to address, making it mostly +irrelevant to them; while they appear superficially similar, they're +actually doing something quite different. They use ``*`` for matrix +multiplication (and for group actions, and so forth), and if this PEP +is accepted, their expressed intention is to continue doing so, while +perhaps adding ``@`` and ``@@`` on matrices as aliases for ``*`` and +``**``: + +* sympy +* sage + +If you know of any actively maintained Python libraries which provide +an interface for working with numerical arrays or matrices, and which +are not listed above, then please let the PEP author know: +njs at pobox.com + + +Rationale for specification details +=================================== + +Choice of operator +------------------ + +Why ``@`` instead of some other punctuation symbol? It doesn't matter +much, and there isn't any consensus across other programming languages +about how this operator should be named [#matmul-other-langs]_, but +``@`` has a few advantages: + +* ``@`` is a friendly character that Pythoneers are already used to + typing in decorators, and its use in email addresses means it is + more likely to be easily accessible across keyboard layouts than + some other characters (e.g. ``$`` or non-ASCII characters). + +* The mATrices mnemonic is cute. + +* It's round like ``*`` and :math:`\cdot`. + +* The use of a single-character token makes ``@@`` possible, which is + a nice bonus. + +* The swirly shape is reminiscent of the simultaneous sweeps over rows + and columns that define matrix multiplication; its asymmetry is + evocative of its non-commutative nature. + + +(Non)-Definitions for built-in types +------------------------------------ + +No ``__matmul__`` or ``__matpow__`` are defined for builtin numeric +types (``float``, ``int``, etc.) or for the ``numbers.Number`` +hierarchy, because these types represent scalars, and the consensus +semantics for ``@`` are that it should raise an error on scalars. + +We do not -- for now -- define a ``__matmul__`` method on the standard +``memoryview`` or ``array.array`` objects, for several reasons. +First, there is currently no way to create multidimensional memoryview +objects using only the stdlib, and array objects cannot represent +multidimensional data at all, which makes ``__matmul__`` much less +useful. Second, providing a quality implementation of matrix +multiplication is highly non-trivial. Naive nested loop +implementations are very slow and providing one in CPython would just +create a trap for users. But the alternative -- providing a modern, +competitive matrix multiply -- would require that CPython link to a +BLAS library, which brings a set of new complications. In particular, +several popular BLAS libraries (including the one that ships by +default on OS X) currently break the use of ``multiprocessing`` +[#blas-fork]_. And finally, we'd have to add quite a bit beyond +``__matmul__`` before ``memoryview`` or ``array.array`` would be +useful for numeric work -- like elementwise versions of the other +arithmetic operators, just to start. Put together, these +considerations mean that the cost/benefit of adding ``__matmul__`` to +these types just isn't there, so for now we'll continue to delegate +these problems to numpy and friends, and defer a more systematic +solution to a future proposal. + +There are also non-numeric Python builtins which define ``__mul__`` +(``str``, ``list``, ...). We do not define ``__matmul__`` for these +types either, because why would we even do that. + + +Unresolved issues +----------------- + +Associativity of ``@`` +'''''''''''''''''''''' + +It's been suggested that ``@`` should be right-associative, on the +grounds that for expressions like ``Mat @ Mat @ vec``, the two +different evaluation orders produce the same result, but the +right-associative order ``Mat @ (Mat @ vec)`` will be faster and use +less memory than the left-associative order ``(Mat @ Mat) @ vec``. +(Matrix-vector multiplication is much cheaper than matrix-matrix +multiplication). It would be a shame if users found themselves +required to use an overabundance of parentheses to achieve acceptable +speed/memory usage in common situations, but, it's not currently clear +whether such cases actually are common enough to override Python's +general rule of left-associativity, or even whether they're more +common than the symmetric cases where left-associativity would be +faster (though this does seem intuitively plausible). The only way to +answer this is probably to do an audit of some real-world uses and +check how often the associativity matters in practice; if this PEP is +accepted in principle, then we should probably do this check before +finalizing it. + + +Rejected alternatives to adding a new operator +============================================== + +Over the past few decades, the Python numeric community has explored a +variety of ways to resolve the tension between matrix and elementwise +multiplication operations. PEP 211 and PEP 225, both proposed in 2000 +and last seriously discussed in 2008 [#threads-2008]_, were early +attempts to add new operators to solve this problem, but suffered from +serious flaws; in particular, at that time the Python numerical +community had not yet reached consensus on the proper API for array +objects, or on what operators might be needed or useful (e.g., PEP 225 +proposes 6 new operators with unspecified semantics). Experience +since then has now led to consensus that the best solution, for both +numeric Python and core Python, is to add a single infix operator for +matrix multiply (together with the other new operators this implies +like ``@=``). + +We review some of the rejected alternatives here. + +**Use a second type that defines __mul__ as matrix multiplication:** +As discussed above (`Background: What's wrong with the status quo?`_), +this has been tried this for many years via the ``numpy.matrix`` type +(and its predecessors in Numeric and numarray). The result is a +strong consensus among both numpy developers and developers of +downstream packages that ``numpy.matrix`` should essentially never be +used, because of the problems caused by having conflicting duck types +for arrays. (Of course one could then argue we should *only* define +``__mul__`` to be matrix multiplication, but then we'd have the same +problem with elementwise multiplication.) There have been several +pushes to remove ``numpy.matrix`` entirely; the only counter-arguments +have come from educators who find that its problems are outweighed by +the need to provide a simple and clear mapping between mathematical +notation and code for novices (see `Transparent syntax is especially +crucial for non-expert programmers`_). But, of course, starting out +newbies with a dispreferred syntax and then expecting them to +transition later causes its own problems. The two-type solution is +worse than the disease. + +**Add lots of new operators, or add a new generic syntax for defining +infix operators:** In addition to being generally un-Pythonic and +repeatedly rejected by BDFL fiat, this would be using a sledgehammer +to smash a fly. The scientific python community has consensus that +adding one operator for matrix multiplication is enough to fix the one +otherwise unfixable pain point. (In retrospect, we all think PEP 225 +was a bad idea too -- or at least far more complex than it needed to +be.) + +**Add a new @ (or whatever) operator that has some other meaning in +general Python, and then overload it in numeric code:** This was the +approach taken by PEP 211, which proposed defining ``@`` to be the +equivalent of ``itertools.product``. The problem with this is that +when taken on its own terms, adding an infix operator for +``itertools.product`` is just silly. (During discussions of this PEP, +a similar suggestion was made to define ``@`` as a general purpose +function composition operator, and this suffers from the same problem; +``functools.compose`` isn't even useful enough to exist.) Matrix +multiplication has a uniquely strong rationale for inclusion as an +infix operator. There almost certainly don't exist any other binary +operations that will ever justify adding any other infix operators to +Python. + +**Add a .dot method to array types so as to allow "pseudo-infix" +A.dot(B) syntax:** This has been in numpy for some years, and in many +cases it's better than dot(A, B). But it's still much less readable +than real infix notation, and in particular still suffers from an +extreme overabundance of parentheses. See `Why should matrix +multiplication be infix?`_ above. + +**Use a 'with' block to toggle the meaning of * within a single code +block**: E.g., numpy could define a special context object so that +we'd have:: + + c = a * b # element-wise multiplication + with numpy.mul_as_dot: + c = a * b # matrix multiplication + +However, this has two serious problems: first, it requires that every +array-like type's ``__mul__`` method know how to check some global +state (``numpy.mul_is_currently_dot`` or whatever). This is fine if +``a`` and ``b`` are numpy objects, but the world contains many +non-numpy array-like objects. So this either requires non-local +coupling -- every numpy competitor library has to import numpy and +then check ``numpy.mul_is_currently_dot`` on every operation -- or +else it breaks duck-typing, with the above code doing radically +different things depending on whether ``a`` and ``b`` are numpy +objects or some other sort of object. Second, and worse, ``with`` +blocks are dynamically scoped, not lexically scoped; i.e., any +function that gets called inside the ``with`` block will suddenly find +itself executing inside the mul_as_dot world, and crash and burn +horribly -- if you're lucky. So this is a construct that could only +be used safely in rather limited cases (no function calls), and which +would make it very easy to shoot yourself in the foot without warning. + +**Use a language preprocessor that adds extra numerically-oriented +operators and perhaps other syntax:** (As per recent BDFL suggestion: +[#preprocessor]_) This suggestion seems based on the idea that +numerical code needs a wide variety of syntax additions. In fact, +given ``@``, most numerical users don't need any other operators or +syntax; it solves the one really painful problem that cannot be solved +by other means, and that causes painful reverberations through the +larger ecosystem. Defining a new language (presumably with its own +parser which would have to be kept in sync with Python's, etc.), just +to support a single binary operator, is neither practical nor +desireable. In the numerical context, Python's competition is +special-purpose numerical languages (Matlab, R, IDL, etc.). Compared +to these, Python's killer feature is exactly that one can mix +specialized numerical code with code for XML parsing, web page +generation, database access, network programming, GUI libraries, and +so forth, and we also gain major benefits from the huge variety of +tutorials, reference material, introductory classes, etc., which use +Python. Fragmenting "numerical Python" from "real Python" would be a +major source of confusion. A major motivation for this PEP is to +*reduce* fragmentation. Having to set up a preprocessor would be an +especially prohibitive complication for unsophisticated users. And we +use Python because we like Python! We don't want +almost-but-not-quite-Python. + +**Use overloading hacks to define a "new infix operator" like *dot*, +as in a well-known Python recipe:** (See: [#infix-hack]_) Beautiful is +better than ugly. This is... not beautiful. And not Pythonic. And +especially unfriendly to beginners, who are just trying to wrap their +heads around the idea that there's a coherent underlying system behind +these magic incantations that they're learning, when along comes an +evil hack like this that violates that system, creates bizarre error +messages when accidentally misused, and whose underlying mechanisms +can't be understood without deep knowledge of how object oriented +systems work. We've considered promoting this as a general solution, +and perhaps if the PEP is rejected we'll revisit this option, but so +far the numeric community has mostly elected to leave this one on the +shelf. + + +References +========== + +.. [#preprocessor] From a comment by GvR on a G+ post by GvR; the + comment itself does not seem to be directly linkable: +https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u +.. [#infix-hack] http://code.activestate.com/recipes/384122-infix-operators/ + http://www.sagemath.org/doc/reference/misc/sage/misc/decorators.html#sage.misc.decorators.infix_operator +.. [#scipy-conf] http://conference.scipy.org/past.html +.. [#pydata-conf] http://pydata.org/events/ +.. [#lht] In this formula, :math:`\beta` is a vector or matrix of + regression coefficients, :math:`V` is the estimated + variance/covariance matrix for these coefficients, and we want to + test the null hypothesis that :math:`H\beta = r`; a large :math:`S` + then indicates that this hypothesis is unlikely to be true. For + example, in an analysis of human height, the vector :math:`\beta` + might contain one value which was the the average height of the + measured men, and another value which was the average height of the + measured women, and then setting :math:`H = [1, -1], r = 0` would + let us test whether men and women are the same height on + average. Compare to eq. 2.139 in + http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/tutorials/xegbohtmlnode17.html + + Example code is adapted from +https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb7/rerpy/incremental_ls.py#L202 + +.. [#pycon-tutorials] Out of the 36 tutorials scheduled for PyCon 2014 + (https://us.pycon.org/2014/schedule/tutorials/), we guess that the + 8 below will almost certainly deal with matrices: + + * Dynamics and control with Python + + * Exploring machine learning with Scikit-learn + + * How to formulate a (science) problem and analyze it using Python + code + + * Diving deeper into Machine Learning with Scikit-learn + + * Data Wrangling for Kaggle Data Science Competitions ? An etude + + * Hands-on with Pydata: how to build a minimal recommendation + engine. + + * Python for Social Scientists + + * Bayesian statistics made simple + + In addition, the following tutorials could easily involve matrices: + + * Introduction to game programming + + * mrjob: Snakes on a Hadoop *("We'll introduce some data science + concepts, such as user-user similarity, and show how to calculate + these metrics...")* + + * Mining Social Web APIs with IPython Notebook + + * Beyond Defaults: Creating Polished Visualizations Using Matplotlib + + This gives an estimated range of 8 to 12 / 36 = 22% to 33% of + tutorials dealing with matrices; saying ~20% then gives us some + wiggle room in case our estimates are high. + +.. [#sloc-details] SLOCs were defined as physical lines which contain + at least one token that is not a COMMENT, NEWLINE, ENCODING, + INDENT, or DEDENT. Counts were made by using ``tokenize`` module + from Python 3.2.3 to examine the tokens in all files ending ``.py`` + underneath some directory. Only tokens which occur at least once + in the source trees are included in the table. The counting script + will be available as an auxiliary file once this PEP is submitted; + until then, it can be found here: + https://gist.github.com/njsmith/9157645 + + Matrix multiply counts were estimated by counting how often certain + tokens which are used as matrix multiply function names occurred in + each package. In principle this could create false positives, but + as far as I know the counts are exact; it's unlikely that anyone is + using ``dot`` as a variable name when it's also the name of one of + the most widely-used numpy functions. + + All counts were made using the latest development version of each + project as of 21 Feb 2014. + + 'stdlib' is the contents of the Lib/ directory in commit + d6aa3fa646e2 to the cpython hg repository, and treats the following + tokens as indicating matrix multiply: n/a. + + 'scikit-learn' is the contents of the sklearn/ directory in commit + 69b71623273ccfc1181ea83d8fb9e05ae96f57c7 to the scikit-learn + repository (https://github.com/scikit-learn/scikit-learn), and + treats the following tokens as indicating matrix multiply: ``dot``, + ``fast_dot``, ``safe_sparse_dot``. + + 'nipy' is the contents of the nipy/ directory in commit + 5419911e99546401b5a13bd8ccc3ad97f0d31037 to the nipy repository + (https://github.com/nipy/nipy/), and treats the following tokens as + indicating matrix multiply: ``dot``. + +.. [#blas-fork] BLAS libraries have a habit of secretly spawning + threads, even when used from single-threaded programs. And threads + play very poorly with ``fork()``; the usual symptom is that + attempting to perform linear algebra in a child process causes an + immediate deadlock. + +.. [#threads-2008] http://fperez.org/py4science/numpy-pep225/numpy-pep225.html + +.. [#broadcasting] http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html + +.. [#matmul-other-langs] +http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html + +.. [#github-details] Counts were produced by manually entering the + string ``"import foo"`` or ``"from foo import"`` (with quotes) into + the Github code search page, e.g.: + https://github.com/search?q=%22import+numpy%22&ref=simplesearch&type=Code + on 2014-04-10 at ~21:00 UTC. The reported values are the numbers + given in the "Languages" box on the lower-left corner, next to + "Python". This also causes some undercounting (e.g., leaving out + Cython code, and possibly one should also count HTML docs and so + forth), but these effects are negligible (e.g., only ~1% of numpy + usage appears to occur in Cython code, and probably even less for + the other modules listed). The use of this box is crucial, + however, because these counts appear to be stable, while the + "overall" counts listed at the top of the page ("We've found ___ + code results") are highly variable even for a single search -- + simply reloading the page can cause this number to vary by a factor + of 2 (!!). (They do seem to settle down if one reloads the page + repeatedly, but nonetheless this is spooky enough that it seemed + better to avoid these numbers.) + + These numbers should of course be taken with multiple grains of + salt; it's not clear how representative Github is of Python code in + general, and limitations of the search tool make it impossible to + get precise counts. AFAIK this is the best data set currently + available, but it'd be nice if it were better. In particular: + + * Lines like ``import sys, os`` will only be counted in the ``sys`` + row. + + * A file containing both ``import X`` and ``from X import`` will be + counted twice + + * Imports of the form ``from X.foo import ...`` are missed. We + could catch these by instead searching for "from X", but this is + a common phrase in English prose, so we'd end up with false + positives from comments, strings, etc. For many of the modules + considered this shouldn't matter too much -- for example, the + stdlib modules have flat namespaces -- but it might especially + lead to undercounting of django, scipy, and twisted. + + Also, it's possible there exist other non-stdlib modules we didn't + think to test that are even more-imported than numpy -- though we + tried quite a few of the obvious suspects. If you find one, let us + know! The modules tested here were chosen based on a combination + of intuition and the top-100 list at pypi-ranking.info. + + Fortunately, it doesn't really matter if it turns out that numpy + is, say, merely the *third* most-imported non-stdlib module, since + the point is just that numeric programming is a common and + mainstream activity. + + Finally, we should point out the obvious: whether a package is + import**ed** is rather different from whether it's import**ant**. + No-one's claiming numpy is "the most important package" or anything + like that. Certainly more packages depend on distutils, e.g., then + depend on numpy -- and far fewer source files import distutils than + import numpy. But this is fine for our present purposes. Most + source files don't import distutils because most source files don't + care how they're distributed, so long as they are; these source + files thus don't care about details of how distutils' API works. + This PEP is in some sense about changing how numpy's and related + packages' APIs work, so the relevant metric is to look at source + files that are choosing to directly interact with that API, which + is sort of like what we get by looking at import statements. + +.. [#hugunin] The first such proposal occurs in Jim Hugunin's very + first email to the matrix SIG in 1995, which lays out the first + draft of what became Numeric. He suggests using ``*`` for + elementwise multiplication, and ``%`` for matrix multiplication: + https://mail.python.org/pipermail/matrix-sig/1995-August/000002.html + + +Copyright +========= + +This document has been placed in the public domain. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Mar 14 17:43:14 2014 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 14 Mar 2014 17:43:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_mark-up_errors_I_introduc?= =?utf-8?q?ed_by_copy/pasting=2E?= Message-ID: <3flt4y5bCYz7Ljg@mail.python.org> http://hg.python.org/peps/rev/8053a66b0bab changeset: 5407:8053a66b0bab user: Guido van Rossum date: Fri Mar 14 09:43:12 2014 -0700 summary: Fix mark-up errors I introduced by copy/pasting. files: pep-0465.txt | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0465.txt b/pep-0465.txt --- a/pep-0465.txt +++ b/pep-0465.txt @@ -733,7 +733,7 @@ for fractional inputs. .. _`mathematical complications`: -https://en.wikipedia.org/wiki/Square_root_of_a_matrix + https://en.wikipedia.org/wiki/Square_root_of_a_matrix Adoption @@ -1039,7 +1039,7 @@ .. [#preprocessor] From a comment by GvR on a G+ post by GvR; the comment itself does not seem to be directly linkable: -https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u + https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u .. [#infix-hack] http://code.activestate.com/recipes/384122-infix-operators/ http://www.sagemath.org/doc/reference/misc/sage/misc/decorators.html#sage.misc.decorators.infix_operator .. [#scipy-conf] http://conference.scipy.org/past.html @@ -1058,7 +1058,7 @@ http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/tutorials/xegbohtmlnode17.html Example code is adapted from -https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb7/rerpy/incremental_ls.py#L202 + https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb7/rerpy/incremental_ls.py#L202 .. [#pycon-tutorials] Out of the 36 tutorials scheduled for PyCon 2014 (https://us.pycon.org/2014/schedule/tutorials/), we guess that the @@ -1144,7 +1144,7 @@ .. [#broadcasting] http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html .. [#matmul-other-langs] -http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html + http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html .. [#github-details] Counts were produced by manually entering the string ``"import foo"`` or ``"from foo import"`` (with quotes) into -- Repository URL: http://hg.python.org/peps From zachary.ware+pydev at gmail.com Fri Mar 14 18:08:22 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 14 Mar 2014 12:08:22 -0500 Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> Message-ID: On Fri, Mar 14, 2014 at 10:55 AM, Benjamin Peterson wrote: > On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: >> On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson >> wrote: >> > Why are you readding these when they were apparently purposely removed >> > in eef7899ea7ab? >> >> It is rather ugly to build the docs in a 3.3 checkout, then update to >> default and run 'hg status' without those entries in .hgignore. I can >> understand not wanting to be bombarded by all of the untracked files, >> but it's also pretty easy to enable the purge extension and do "hg >> purge Doc/tools". > > This is why everyone should use the share extension and have separate > working copies for every branch. :) I've been using that setup for a few months now (and cursing myself for not switching sooner), but I've still found myself annoyed by this issue a time or two since the doc build chain change. I personally don't mind whether those entries are in .hgignore or not, but I don't see the harm in them sticking around (at least until 3.3 is out of maintenance; updating from 2.7 to default is a great source of problems even aside from this issue). -- Zach From solipsis at pitrou.net Fri Mar 14 18:12:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Mar 2014 18:12:30 +0100 Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> Message-ID: <20140314181230.063e4f6a@fsol> On Fri, 14 Mar 2014 12:08:22 -0500 Zachary Ware wrote: > On Fri, Mar 14, 2014 at 10:55 AM, Benjamin Peterson wrote: > > On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: > >> On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson > >> wrote: > >> > Why are you readding these when they were apparently purposely removed > >> > in eef7899ea7ab? > >> > >> It is rather ugly to build the docs in a 3.3 checkout, then update to > >> default and run 'hg status' without those entries in .hgignore. I can > >> understand not wanting to be bombarded by all of the untracked files, > >> but it's also pretty easy to enable the purge extension and do "hg > >> purge Doc/tools". > > > > This is why everyone should use the share extension and have separate > > working copies for every branch. :) > > I've been using that setup for a few months now (and cursing myself > for not switching sooner), but I've still found myself annoyed by this > issue a time or two since the doc build chain change. I personally > don't mind whether those entries are in .hgignore or not, but I don't > see the harm in them sticking around (at least until 3.3 is out of > maintenance; updating from 2.7 to default is a great source of > problems even aside from this issue). Indeed, I think it's ok to keep them. Regards Antoine. From python-checkins at python.org Fri Mar 14 18:46:00 2014 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 14 Mar 2014 18:46:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_465=3A_Updated_section_?= =?utf-8?q?=22Choice_of_operator=22=2E?= Message-ID: <3flvTN2mmXz7Ljg@mail.python.org> http://hg.python.org/peps/rev/790f47e5419b changeset: 5408:790f47e5419b user: Guido van Rossum date: Fri Mar 14 10:45:59 2014 -0700 summary: PEP 465: Updated section "Choice of operator". files: pep-0465.txt | 105 +++++++++++++++++++++++++++++++++----- 1 files changed, 89 insertions(+), 16 deletions(-) diff --git a/pep-0465.txt b/pep-0465.txt --- a/pep-0465.txt +++ b/pep-0465.txt @@ -820,26 +820,99 @@ Choice of operator ------------------ -Why ``@`` instead of some other punctuation symbol? It doesn't matter -much, and there isn't any consensus across other programming languages -about how this operator should be named [#matmul-other-langs]_, but -``@`` has a few advantages: +Why ``@`` instead of some other spelling? There isn't any consensus +across other programming languages about how this operator should be +named [#matmul-other-langs]_; here we discuss the various options. -* ``@`` is a friendly character that Pythoneers are already used to - typing in decorators, and its use in email addresses means it is - more likely to be easily accessible across keyboard layouts than - some other characters (e.g. ``$`` or non-ASCII characters). +Restricting ourselves only to symbols present on US English keyboards, +the punctuation characters that don't already have a meaning in Python +expression context are: ``@``, backtick, ``$``, ``!``, and ``?``. Of +these options, ``@`` is clearly the best; ``!`` and ``?`` are already +heavily freighted with inapplicable meanings in the programming +context, backtick has been banned from Python by BDFL pronouncement +(see PEP 3099), and ``$`` is uglier, even more dissimilar to ``*`` and +:math:`\cdot`, and has Perl/PHP baggage. ``$`` is probably the +second-best option of these, though. + +Symbols which are not present on US English keyboards start at a +significant disadvantage (having to spend 5 minutes at the beginning +of every numeric Python tutorial just going over keyboard layouts is +not a hassle anyone really wants). Plus, even if we somehow overcame +the typing problem, it's not clear there are any that are actually +better than ``@``. Some options that have been suggested include: + +* U+00D7 MULTIPLICATION SIGN: ``A ? B`` +* U+22C5 DOT OPERATOR: ``A ? B`` +* U+2297 CIRCLED TIMES: ``A ? B`` +* U+00B0 DEGREE: ``A ? B`` + +What we need, though, is an operator that means "matrix +multiplication, as opposed to scalar/elementwise multiplication". +There is no conventional symbol for these in mathematics or +programming, where these operations are usually distinguished by +context. (And U+2297 CIRCLED TIMES is actually used conventionally to +mean exactly the opposite: elementwise multiplication -- the "Hadamard +product" -- as opposed to matrix multiplication). ``@`` at least has +the virtue that it *looks* like a funny non-commutative operator; a +naive user who knows maths but not programming couldn't look at ``A * +B`` versus ``A ? B``, or ``A * B`` versus ``A ? B``, or ``A * B`` +versus ``A ? B`` and guess which one is the usual multiplication, and +which one is the special case. + +Finally, there is the option of using multi-character tokens. Some +options: + +* Matlab uses a ``.*`` operator. Aside from being visually confusable + with ``*``, this would be a terrible choice for us because in + Matlab, ``*`` means matrix multiplication and ``.*`` means + elementwise multiplication, so using ``.*`` for matrix + multiplication would make us exactly backwards from what Matlab + users expect. + +* APL apparently used ``+.?``, which by combining a multi-character + token, confusing attribute-access-like . syntax, and a unicode + character, ranks somewhere below U+2603 SNOWMAN on our candidate + list. If we like the idea of combining addition and multiplication + operators as being evocative of how matrix multiplication actually + works, then something like ``+*`` could be used -- though this may + be too easy to confuse with ``*+``, which is just multiplication + combined with the unary ``+`` operator. + +* PEP 211 suggested ``~*`` and ``~**``. This has the downside that it + sort of suggests that there is a unary ``*`` operator that is being + combined with unary ``~``, but it could work. + +* R uses ``%*%`` for matrix multiplication. In R this forms part of a + general extensible infix system in which all tokens of the form + ``%foo%`` are user-defined binary operators. We could steal the + token without stealing the system. + +* Some other plausible candidates that have been suggested: ``><`` (= + ascii drawing of the multiplication sign ?); the footnote operators + ``[*]`` and ``[**]`` or ``|*|`` and ``|**|`` (but when used in + context, the use of vertical grouping symbols tends to recreate the + nested parentheses visual clutter that was noted as one of the major + downsides of the function syntax we're trying to get away from); + ``^*`` and ``^^``. + +So, it doesn't matter much, but ``@`` seems as good or better than any +of the alternatives: + +* It's a friendly character that Pythoneers are already used to typing + in decorators, but the decorator usage and the math expression + usage are sufficiently dissimilar that it would be hard to confuse + them in practice. + +* It's widely accessible across keyboard layouts (and thanks to its + use in email addresses, this is true even of weird keyboards like + those in phones). + +* It's round like ``*`` and :math:`\cdot`. * The mATrices mnemonic is cute. -* It's round like ``*`` and :math:`\cdot`. - -* The use of a single-character token makes ``@@`` possible, which is - a nice bonus. - -* The swirly shape is reminiscent of the simultaneous sweeps over rows - and columns that define matrix multiplication; its asymmetry is - evocative of its non-commutative nature. +* The use of a single-character token reduces the line-noise effect, + and makes ``@@`` possible, which is a nice bonus. (Non)-Definitions for built-in types -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Mar 14 21:17:13 2014 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 14 Mar 2014 21:17:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_put_python-version_after_crea?= =?utf-8?q?ted?= Message-ID: <3flyqs4ZpKz7LjN@mail.python.org> http://hg.python.org/peps/rev/b7cdb9a7e146 changeset: 5409:b7cdb9a7e146 user: Benjamin Peterson date: Fri Mar 14 15:17:05 2014 -0500 summary: put python-version after created files: pep-0465.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0465.txt b/pep-0465.txt --- a/pep-0465.txt +++ b/pep-0465.txt @@ -5,9 +5,9 @@ Author: Nathaniel J. Smith Status: Draft Type: Standards Track -Python-Version: 3.5 Content-Type: text/x-rst Created: 20-Feb-2014 +Python-Version: 3.5 Post-History: 13-Mar-2014 Abstract -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Mar 14 23:31:23 2014 From: python-checkins at python.org (r.david.murray) Date: Fri, 14 Mar 2014 23:31:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_final_edit_pas?= =?utf-8?q?s=2E?= Message-ID: <3fm1pg6ddQz7LjX@mail.python.org> http://hg.python.org/cpython/rev/cc27535a7c00 changeset: 89655:cc27535a7c00 user: R David Murray date: Fri Mar 14 18:31:07 2014 -0400 summary: whatsnew: final edit pass. files: Doc/includes/email-alternative-new-api.py | 2 +- Doc/library/exceptions.rst | 1 - Doc/library/smtplib.rst | 3 + Doc/whatsnew/3.4.rst | 179 +++++---- 4 files changed, 94 insertions(+), 91 deletions(-) diff --git a/Doc/includes/email-alternative-new-api.py b/Doc/includes/email-alternative-new-api.py --- a/Doc/includes/email-alternative-new-api.py +++ b/Doc/includes/email-alternative-new-api.py @@ -19,7 +19,7 @@ [1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718 ---?ric +--Pep? """) # Add the html version. This converts the message into a multipart/alternative diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -265,7 +265,6 @@ :exc:`mmap.error` have been merged into :exc:`OSError`. .. versionchanged:: 3.4 - The :attr:`filename` attribute is now the original file name passed to the function, instead of the name encoded to or decoded from the filesystem encoding. Also, the :attr:`filename2` attribute was added. diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -117,6 +117,9 @@ Subclass of :exc:`OSError` that is the base exception class for all the other exceptions provided by this module. + .. versionchanged:: 3.4 + SMTPException became subclass of :exc:`OSError` + .. exception:: SMTPServerDisconnected diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -406,10 +406,10 @@ ``__main__.__file__`` when a script has been executed directly using a relative path (Contributed by Brett Cannon in :issue:`18416`). -* Now all the UTF-\* codecs (except UTF-7) reject surrogates during both +* All the UTF-\* codecs (except UTF-7) now reject surrogates during both encoding and decoding unless the ``surrogatepass`` error handler is used, - with the exception of the UTF-16 decoder that accepts valid surrogate pairs, - and the UTF-16 encoder that produces them while encoding non-BMP characters. + with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) + and the UTF-16 encoder (which produces them while encoding non-BMP characters). Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. @@ -563,7 +563,7 @@ The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to trace memory blocks allocated by Python. It provides the following information: -* Traceback where an object was allocated +* Trace where an object was allocated * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Compute the differences between two snapshots to detect memory leaks @@ -643,7 +643,7 @@ :func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to encode and decode binary data from and to ``Ascii85`` and the git/mercurial ``Base85`` formats, respectively. The ``a85`` functions have options that can -be sued to make them compatible with the variants of the ``Ascii85`` encoding, +be used to make them compatible with the variants of the ``Ascii85`` encoding, including the Adobe variant. (Contributed by Martin Morrison, the Mercurial project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) @@ -861,15 +861,15 @@ :pep:`443` -- Single-dispatch generic functions PEP written and implemented by ?ukasz Langa. +:func:`~functools.total_ordering` now supports a return value of +:const:`NotImplemented` from the underlying comparison function. (Contributed +by Katie Miller in :issue:`10042`.) + A pure-python version of the :func:`~functools.partial` function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in :issue:`12428`.) -:func:`~functools.total_ordering` now supports a return value of -:const:`NotImplemented` from the underlying comparison function. (Contributed -by Katie Miller in :issue:`10042`.) - gc -- @@ -1093,29 +1093,17 @@ mmap ---- -mmap objects can now be weakref'ed. (Contributed by Valerie Lambert in +mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in :issue:`4885`.) -mock ----- - -:mod:`~unittest.mock` objects now inspect their specification signatures when -matching calls, which means an argument can now be matched by either position -or name, instead of only by position. (Contributed by Antoine Pitrou in -:issue:`17015`.) - -:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` -methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) - - multiprocessing --------------- .. _whatsnew-multiprocessing-no-fork: -On Unix, two new :ref:`start methods ` -(``spawn`` and ``forkserver``) have been added for starting processes using +On Unix two new :ref:`start methods `, +(``spawn`` and ``forkserver``, have been added for starting processes using :mod:`multiprocessing`. These make the mixing of processes with threads more robust, and the ``spawn`` method matches the semantics that multiprocessing has always used on Windows. New function @@ -1141,7 +1129,7 @@ :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes when using the ``spawn`` or ``forkserver`` start methods. This resolves some -edge cases where combining multiprocessing, the ``-m`` command line switch +edge cases where combining multiprocessing, the ``-m`` command line switch, and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in :issue:`19946`) @@ -1149,29 +1137,29 @@ operator -------- -There is now a pure-python version of the :mod:`operator` module available for -reference and for use by alternate implementations of Python. (Contributed by -Zachary Ware in :issue:`16694`.) - New function :func:`~operator.length_hint` provides an implementation of the specification for how the :meth:`~object.__length_hint__` special method should be used, as part of the :pep:`424` formal specification of this language feature. (Contributed by Armin Ronacher in :issue:`16148`.) +There is now a pure-python version of the :mod:`operator` module available for +reference and for use by alternate implementations of Python. (Contributed by +Zachary Ware in :issue:`16694`.) + os -- -New functions to get and set the :ref:`inheritable flag ` of a file -descriptors or a Windows handle: - -* :func:`os.get_inheritable`, :func:`os.set_inheritable` -* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` - -The :mod:`os` module now provides a :func:`~os.cpu_count` function, analogous to -the :func:`multiprocessing.cpu_count` function (which is now implemented in -terms of the new :mod:`os` function). (Contributed by Trent Nelson, Yogesh -Chaudhari, Victor Stinner, and Charles-Fran?ois Natali in :issue:`17914`.) +There are new functions to get and set the :ref:`inheritable flag +` of a file descriptor (:func:`os.get_inheritable`, +:func:`os.set_inheritable`) or a Windows handle +(:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). + +New function :func:`~os.cpu_count` reports the number of CPUs available on the +platform on which Python is running (or ``None`` if the count can't be +determined). The :func:`multiprocessing.cpu_count` function is now implemented +in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, +Victor Stinner, and Charles-Fran?ois Natali in :issue:`17914`.) :func:`os.path.samestat` is now available on the Windows platform (and the :func:`os.path.samefile` implementation is now shared between Unix and @@ -1181,10 +1169,10 @@ root on Windows. (Contributed by Tim Golden in :issue:`9035`.) :func:`os.open` supports two new flags on platforms that provide them, -:data:`~os.O_TMPFILE` (unnamed temporary file, as of 3.4.0 release available -only on Linux systems with a kernel version of 3.11 or newer that have uapi -headers), and :data:`~os.O_PATH` (un-opened file descriptor). (Contributed by -Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) +:data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` +(unnamed temporary file; as of 3.4.0 release available only on Linux systems +with a kernel version of 3.11 or newer that have uapi headers). (Contributed +by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) pdb @@ -1214,7 +1202,7 @@ :mod:`pickle` now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large -strings and containers, or classes whose :meth:`__new__` method takes +strings and containers, and classes whose :meth:`__new__` method takes keyword-only arguments. It also provides some efficiency improvements. .. seealso:: @@ -1269,11 +1257,11 @@ pydoc ----- -The :mod:`pydoc` module is now based directly on the -:func:`inspect.signature` introspection API, allowing it to provide -signature information for a wider variety of callable objects. This change -also means that ``__wrapped__`` attributes are now taken into account when -display help information (Contributed by Larry Hastings in :issue:`19674`) +The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` +introspection API, allowing it to provide signature information for a wider +variety of callable objects. This change also means that ``__wrapped__`` +attributes are now taken into account when displaying help information +(Contributed by Larry Hastings in :issue:`19674`) The :mod:`pydoc` module no longer displays the ``self`` parameter for already bound methods. Instead, it aims to always display the exact current @@ -1362,7 +1350,7 @@ ----- The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now -accept a *map* keyword argument, which if specified is passed in to +accept a *map* keyword argument which, if specified, is passed in to :class:`asynchat.async_chat` as its *map* argument. This allows an application to avoid affecting the global socket map. (Contributed by Vinay Sajip in :issue:`11959`.) @@ -1374,7 +1362,7 @@ :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. -(:issue:`2118`). +(Contributed by Ned Jackson Lovely in :issue:`2118`). socket @@ -1400,10 +1388,10 @@ sqlite3 ------- -A new boolean parameter, *uri*, to the :func:`~sqlite3.connect` function can -be used to indicate that the *database* parameter is a ``uri`` (see -the `SQLite URI documentation `_). -(Contributed by poq in :issue:`13773`.) +A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be +used to indicate that the *database* parameter is a ``uri`` (see the `SQLite +URI documentation `_). (Contributed by poq in +:issue:`13773`.) ssl @@ -1418,6 +1406,18 @@ .. _whatsnew34-sslcontext: +New function :func:`~ssl.create_default_context` provides a standard way to +obtain an :class:`~ssl.SSLContext` whose settings are intended to be a +reasonable balance between compatibility and security. These settings are +more stringent than the defaults provided by the :class:`~ssl.SSLContext` +constructor, and may be adjusted in the future, without prior deprecation, if +best-practice security requirements change. The new recommended best +practice for using stdlib libraries that support SSL is to use +:func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` +object, modify it if needed, and then pass it as the *context* argument +of the appropriate stdlib API. (Contributed by Christian Heimes +in :issue:`19689`.) + :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` accepts a new optional argument *cadata*, which can be used to provide PEM or DER encoded certificates directly via strings or bytes, respectively. @@ -1435,7 +1435,7 @@ ``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists (``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that returns a list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in -and :issue:`18147`.) +:issue:`18147`.) If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the @@ -1454,14 +1454,6 @@ (``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian Heimes in :issue:`19292`.) -New function :func:`~ssl.create_default_context` provides a standard way to -obtain an :class:`~ssl.SSLContext` whose settings are intended to be a -reasonable balance between compatibility and security. These settings are -more stringent than the defaults provided by the :class:`~ssl.SSLContext` -constructor, and may be adjusted in the future, without prior deprecation, if -best-practice security requirements change. (Contributed by Christian Heimes -in :issue:`19689`.) - .. _whatsnew34-win-cert-store: Two new windows-only functions, :func:`~ssl.enum_certificates` and @@ -1485,19 +1477,20 @@ The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C implementation is required as most of the values aren't standardized and -platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) - -The module supports new file types: door, event port and whiteout. +are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) + +The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, +:attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by +Christian Hiemes in :issue:`11016`.) struct ------ -:mod:`struct` now supports the streamed unpacking of a buffer containing -repeated instances of a given format of data. Both a module level -:mod:`~struct.iter_unpack` function and a :meth:`struct.Struct.iter_unpack` -method on compiled formats have been added. (Contributed by Antoine Pitrou in -:issue:`17804`.) +New function :mod:`~struct.iter_unpack` and a new +:meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed +unpacking of a buffer containing repeated instances of a given format of data. +(Contributed by Antoine Pitrou in :issue:`17804`.) subprocess @@ -1508,8 +1501,8 @@ (Contributed by Zack Weinberg in :issue:`16624`.) :func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now -work on Windows. This change was actually made in 3.3.4. (Contributed -by Tim Golden in :issue:`10197`.) +work on Windows. This change was actually inadvertently made in 3.3.4. +(Contributed by Tim Golden in :issue:`10197`.) sunau @@ -1536,9 +1529,9 @@ --- New function :func:`sys.getallocatedblocks` returns the current number of -blocks allocated by the interpreter (in CPython with the default +blocks allocated by the interpreter. (In CPython with the default ``--with-pymalloc`` setting, this is allocations made through the -:c:func:`PyObject_Malloc` API). This can be useful for tracking memory leaks, +:c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in :issue:`13390`.) @@ -1692,6 +1685,14 @@ Test discovery now works with namespace packages (Contributed by Claudiu Popa in :issue:`17457`.) +:mod:`unittest.mock` objects now inspect their specification signatures when +matching calls, which means an argument can now be matched by either position +or name, instead of only by position. (Contributed by Antoine Pitrou in +:issue:`17015`.) + +:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` +methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) + venv ---- @@ -1844,7 +1845,7 @@ implemented in C. Some standard library extension modules have been converted to use Argument -Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` has been updated +Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated accordingly. It is expected that signature metadata for programmatic introspection will @@ -1879,7 +1880,7 @@ marked as accepting ``const char *`` rather than ``char *`` (Contributed by Serhiy Storchaka in :issue:`1772673`). -* New shell version of ``python-config``; can be used even when a python +* A new shell version of ``python-config`` can be used even when a python interpreter is not available (for example, in cross compilation scenarios). * :c:func:`PyUnicode_FromFormat` now supports width and precision @@ -2086,8 +2087,9 @@ * The :mod:`formatter` module is pending deprecation and is slated for removal in Python 3.6. -* MD5 as default digestmod for :mod:`hmac` is deprecated. Python 3.6 will - require an explicit digest name or constructor as *digestmod* argument. +* ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is + deprecated. Python 3.6 will require an explicit digest name or constructor as + *digestmod* argument. * The internal ``Netrc`` class in the :mod:`ftplib` module has been documented as deprecated in its docstring for quite some time. It now emits a @@ -2099,9 +2101,6 @@ * The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. -* The :func:`hmac.new` *digestmod* keyword having a default value (currently - ``MD5``) is deprecated. - * The :mod:`plistlib` :func:`~plistlib.readPlist`, :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the @@ -2117,7 +2116,7 @@ appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* argument. -* The *parser* argument of :func:`~xml.etree.ElementTree.iterparse` has +* The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has been deprecated, as has the *html* argument of :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the latter, all arguments to ``XMLParser`` should be passed by keyword. @@ -2308,7 +2307,7 @@ no ``__wrapped__`` attribute. * :func:`inspect.getfullargspec` has been reimplemented on top of - :func`inspect.signature` and hence handles a much wider variety of callable + :func:`inspect.signature` and hence handles a much wider variety of callable objects than it did in the past. It is expected that additional builtin and extension module callables will gain signature metadata over the course of the Python 3.4 series. Code that assumes that @@ -2321,8 +2320,10 @@ iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` will not find all keys. A module's ``__file__`` when imported in the current working directory will also now have an absolute path, including when using - ``-m`` with the interpreter (this does not influence when the path to a file - is specified on the command-line) (:issue:`18416`). + ``-m`` with the interpreter (except for ``__main__.__file__`` when a script + has been executed directly using a relative path) (Contributed by Brett + Cannon in :issue:`18416`). is specified on the command-line) + (:issue:`18416`). * The removal of the *strict* argument to :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` changes the meaning of the @@ -2383,7 +2384,7 @@ perpetually alive (for example, database connections kept in thread-local storage). (:issue:`17094`.) -* Parameter names in ``__annotations__`` dict are now mangled properly, +* Parameter names in ``__annotations__`` dicts are now mangled properly, similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in :issue:`20625`). @@ -2408,7 +2409,7 @@ * The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of - :class:``match`` objects now always return a *bytes* object when the string + ``match`` objects now always return a *bytes* object when the string to be matched is a :term:`bytes-like object`. Previously the return type matched the input type, so if your code was depending on the return value being, say, a ``bytearray``, you will need to change your code. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 00:13:10 2014 From: python-checkins at python.org (vinay.sajip) Date: Sat, 15 Mar 2014 00:13:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Reverted_change_introduced?= =?utf-8?q?_in_edb12dad7bf6=2E?= Message-ID: <3fm2kt0VwWz7LjP@mail.python.org> http://hg.python.org/cpython/rev/63d2d0c15b37 changeset: 89656:63d2d0c15b37 user: Vinay Sajip date: Fri Mar 14 23:13:00 2014 +0000 summary: Reverted change introduced in edb12dad7bf6. files: .hgignore | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -24,11 +24,6 @@ reflog.txt$ tags$ Lib/plat-mac/errors.rsrc.df.rsrc -Doc/tools/sphinx/ -Doc/tools/docutils/ -Doc/tools/jinja/ -Doc/tools/jinja2/ -Doc/tools/pygments/ Misc/python.pc Misc/python-config.sh$ Modules/Setup$ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 02:15:37 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 02:15:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_cast_negative_numbers_to_s?= =?utf-8?q?ize=5Ft_before_shifting_them_=28=2320929=29?= Message-ID: <3fm5S921mfz7LkP@mail.python.org> http://hg.python.org/cpython/rev/e68f156ea0e6 changeset: 89657:e68f156ea0e6 user: Benjamin Peterson date: Fri Mar 14 20:15:29 2014 -0500 summary: cast negative numbers to size_t before shifting them (#20929) files: Include/objimpl.h | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Include/objimpl.h b/Include/objimpl.h --- a/Include/objimpl.h +++ b/Include/objimpl.h @@ -265,7 +265,7 @@ #define _PyGCHead_REFS(g) ((g)->gc.gc_refs >> _PyGC_REFS_SHIFT) #define _PyGCHead_SET_REFS(g, v) do { \ (g)->gc.gc_refs = ((g)->gc.gc_refs & ~_PyGC_REFS_MASK) \ - | (v << _PyGC_REFS_SHIFT); \ + | (((size_t)(v)) << _PyGC_REFS_SHIFT); \ } while (0) #define _PyGCHead_DECREF(g) ((g)->gc.gc_refs -= 1 << _PyGC_REFS_SHIFT) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -8,6 +8,8 @@ Core and Builtins ----------------- +- Issue #20929: Add a type cast to avoid shifting a negative number. + - Issue #20731: Properly position in source code files even if they are opened in text mode. Patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 03:47:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 03:47:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogZG9uJ3QgZG8gcG9p?= =?utf-8?q?nter_arithmetic_with_signed_numbers?= Message-ID: <3fm7Vl329mz7LjP@mail.python.org> http://hg.python.org/cpython/rev/d1b392141123 changeset: 89658:d1b392141123 branch: 3.3 parent: 89647:d7950e916f20 user: Benjamin Peterson date: Fri Mar 14 21:47:23 2014 -0500 summary: don't do pointer arithmetic with signed numbers files: Objects/longobject.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -36,7 +36,8 @@ static PyObject * get_small_int(sdigit ival) { - PyObject *v = (PyObject*)(small_ints + ival + NSMALLNEGINTS); + assert(-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS); + PyObject *v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 03:48:00 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 03:48:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fm7Vm4kJWz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/061a9679ea2b changeset: 89659:061a9679ea2b parent: 89657:e68f156ea0e6 parent: 89658:d1b392141123 user: Benjamin Peterson date: Fri Mar 14 21:47:36 2014 -0500 summary: merge 3.3 files: Objects/longobject.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -37,7 +37,8 @@ static PyObject * get_small_int(sdigit ival) { - PyObject *v = (PyObject*)(small_ints + ival + NSMALLNEGINTS); + assert(-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS); + PyObject *v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 03:55:05 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 03:55:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_fix_c89_declar?= =?utf-8?q?ation_order?= Message-ID: <3fm7fx3W2Wz7LjR@mail.python.org> http://hg.python.org/cpython/rev/8877c6b39461 changeset: 89660:8877c6b39461 branch: 3.3 parent: 89658:d1b392141123 user: Benjamin Peterson date: Fri Mar 14 21:53:51 2014 -0500 summary: fix c89 declaration order files: Objects/longobject.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -36,8 +36,9 @@ static PyObject * get_small_int(sdigit ival) { + PyObject *v; assert(-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS); - PyObject *v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; + v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 03:55:06 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 03:55:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_avoid_referenc?= =?utf-8?q?ing_out-of-bounds_memory?= Message-ID: <3fm7fy5FWfz7LjR@mail.python.org> http://hg.python.org/cpython/rev/ea27f0ed741f changeset: 89661:ea27f0ed741f branch: 3.3 user: Benjamin Peterson date: Fri Mar 14 21:54:31 2014 -0500 summary: avoid referencing out-of-bounds memory files: Objects/listobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2036,7 +2036,7 @@ if (keys != NULL) { for (i = 0; i < saved_ob_size; i++) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 03:55:08 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 03:55:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fm7g001NTz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/7ce22d0899e4 changeset: 89662:7ce22d0899e4 parent: 89659:061a9679ea2b parent: 89661:ea27f0ed741f user: Benjamin Peterson date: Fri Mar 14 21:54:41 2014 -0500 summary: merge 3.3 files: Objects/listobject.c | 2 +- Objects/longobject.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2043,7 +2043,7 @@ if (keys != NULL) { for (i = 0; i < saved_ob_size; i++) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); } diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -37,8 +37,9 @@ static PyObject * get_small_int(sdigit ival) { + PyObject *v; assert(-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS); - PyObject *v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; + v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Mar 15 09:51:44 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 15 Mar 2014 09:51:44 +0100 Subject: [Python-checkins] Daily reference leaks (e68f156ea0e6): sum=0 Message-ID: results for e68f156ea0e6 on branch "default" -------------------------------------------- test_asyncio leaked [4, 0, 0] memory blocks, sum=4 test_site leaked [-2, 2, -2] references, sum=-2 test_site leaked [-2, 2, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogA6ySLy', '-x'] From python-checkins at python.org Sat Mar 15 17:03:25 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 15 Mar 2014 17:03:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzIwOTMzOiBBdCBs?= =?utf-8?q?east_one_place_maps_=27test=27_to_=27localhost=27=2E=2E=2Efix_t?= =?utf-8?q?est=2E?= Message-ID: <3fmT8Y29Mcz7LjM@mail.python.org> http://hg.python.org/cpython/rev/6a78bbd0fda4 changeset: 89663:6a78bbd0fda4 branch: 3.3 parent: 89661:ea27f0ed741f user: R David Murray date: Sat Mar 15 12:00:14 2014 -0400 summary: #20933: At least one place maps 'test' to 'localhost'...fix test. Discovery and patch by Wenzhu Man. University of Waterloo apparently maps the local name 'test' to localhost, which is in the bypass list, causing the test to fail. So change 'test' to a name unlikely to get mapped to localhost. files: Lib/test/test_urllib2.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1209,7 +1209,8 @@ self.assertTrue(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be True' % host) # Check hosts that should not trigger the proxy bypass - for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', 'test'): + for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', + 'notinbypass'): self.assertFalse(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be False' % host) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 17:03:26 2014 From: python-checkins at python.org (r.david.murray) Date: Sat, 15 Mar 2014 17:03:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2320933=3A_At_least_one_place_maps_=27test=27_?= =?utf-8?b?dG8gJ2xvY2FsaG9zdCcuLi5maXggdGVzdC4=?= Message-ID: <3fmT8Z5gnnz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/586de8d96817 changeset: 89664:586de8d96817 parent: 89662:7ce22d0899e4 parent: 89663:6a78bbd0fda4 user: R David Murray date: Sat Mar 15 12:03:02 2014 -0400 summary: Merge: #20933: At least one place maps 'test' to 'localhost'...fix test. files: Lib/test/test_urllib2.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1227,7 +1227,8 @@ self.assertTrue(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be True' % host) # Check hosts that should not trigger the proxy bypass - for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', 'test'): + for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', + 'notinbypass'): self.assertFalse(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be False' % host) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 17:50:25 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 17:50:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_unused_?= =?utf-8?q?zero_constants?= Message-ID: <3fmVBn3Krsz7LjP@mail.python.org> http://hg.python.org/cpython/rev/a3f80c22a441 changeset: 89665:a3f80c22a441 branch: 3.3 parent: 89663:6a78bbd0fda4 user: Benjamin Peterson date: Sat Mar 15 11:49:49 2014 -0500 summary: remove unused zero constants files: Modules/_decimal/_decimal.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -3928,9 +3928,6 @@ return ret; } -static mpd_uint_t data_zero[1] = {0}; -static const mpd_t zero = {MPD_STATIC|MPD_CONST_DATA, 0, 1, 1, 1, data_zero}; - static PyObject * nm_mpd_qpow(PyObject *base, PyObject *exp, PyObject *mod) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 17:50:26 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 17:50:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_only_define_ze?= =?utf-8?q?ro_constant_when_it_is_used?= Message-ID: <3fmVBp57K7z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/3f93907ab8e1 changeset: 89666:3f93907ab8e1 branch: 3.3 user: Benjamin Peterson date: Sat Mar 15 11:50:00 2014 -0500 summary: only define zero constant when it is used files: Modules/_math.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Modules/_math.c b/Modules/_math.c --- a/Modules/_math.c +++ b/Modules/_math.c @@ -22,7 +22,9 @@ static const double ln2 = 6.93147180559945286227E-01; static const double two_pow_m28 = 3.7252902984619141E-09; /* 2**-28 */ static const double two_pow_p28 = 268435456.0; /* 2**28 */ +#ifndef Py_NAN static const double zero = 0.0; +#endif /* acosh(x) * Method : -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 17:50:27 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 17:50:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fmVBq6xXtz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/85cb7c37503e changeset: 89667:85cb7c37503e parent: 89664:586de8d96817 parent: 89666:3f93907ab8e1 user: Benjamin Peterson date: Sat Mar 15 11:50:14 2014 -0500 summary: merge 3.3 files: Modules/_decimal/_decimal.c | 3 --- Modules/_math.c | 2 ++ 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -3928,9 +3928,6 @@ return ret; } -static mpd_uint_t data_zero[1] = {0}; -static const mpd_t zero = {MPD_STATIC|MPD_CONST_DATA, 0, 1, 1, 1, data_zero}; - static PyObject * nm_mpd_qpow(PyObject *base, PyObject *exp, PyObject *mod) { diff --git a/Modules/_math.c b/Modules/_math.c --- a/Modules/_math.c +++ b/Modules/_math.c @@ -22,7 +22,9 @@ static const double ln2 = 6.93147180559945286227E-01; static const double two_pow_m28 = 3.7252902984619141E-09; /* 2**-28 */ static const double two_pow_p28 = 268435456.0; /* 2**28 */ +#ifndef Py_NAN static const double zero = 0.0; +#endif /* acosh(x) * Method : -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 18:22:03 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 18:22:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_avoid_referenc?= =?utf-8?q?ing_past_the_bounds_of_an_array?= Message-ID: <3fmVvH5CTQz7LjW@mail.python.org> http://hg.python.org/cpython/rev/f5e9495df22f changeset: 89668:f5e9495df22f branch: 3.3 parent: 89666:3f93907ab8e1 user: Benjamin Peterson date: Sat Mar 15 12:21:28 2014 -0500 summary: avoid referencing past the bounds of an array files: Objects/listobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -1963,7 +1963,7 @@ if (keys[i] == NULL) { for (i=i-1 ; i>=0 ; i--) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); goto keyfunc_fail; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 18:22:05 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 18:22:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fmVvK0Mfxz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/5c1bacba828d changeset: 89669:5c1bacba828d parent: 89667:85cb7c37503e parent: 89668:f5e9495df22f user: Benjamin Peterson date: Sat Mar 15 12:21:47 2014 -0500 summary: merge 3.3 files: Objects/listobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -1970,7 +1970,7 @@ if (keys[i] == NULL) { for (i=i-1 ; i>=0 ; i--) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); goto keyfunc_fail; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 18:29:33 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 18:29:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_runtime?= =?utf-8?q?=5Flibrary=5Fdirs_for_=5Fsqlite=3B_it_isn=27t_needed?= Message-ID: <3fmW3x73nSz7LjW@mail.python.org> http://hg.python.org/cpython/rev/7cb8faf45539 changeset: 89670:7cb8faf45539 branch: 3.3 parent: 89668:f5e9495df22f user: Benjamin Peterson date: Sat Mar 15 12:29:04 2014 -0500 summary: remove runtime_library_dirs for _sqlite; it isn't needed files: setup.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1128,7 +1128,6 @@ define_macros=sqlite_defines, include_dirs=include_dirs, library_dirs=sqlite_libdir, - runtime_library_dirs=sqlite_libdir, extra_link_args=sqlite_extra_link_args, libraries=["sqlite3",])) else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 18:29:35 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 18:29:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_runtime?= =?utf-8?q?=5Flibrary=5Fdirs_for_=5Fsqlite=3B_it_isn=27t_needed?= Message-ID: <3fmW3z1nkPz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/71da05fa2d8a changeset: 89671:71da05fa2d8a branch: 2.7 parent: 89642:67ada6ab7fe2 user: Benjamin Peterson date: Sat Mar 15 12:29:04 2014 -0500 summary: remove runtime_library_dirs for _sqlite; it isn't needed files: setup.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1187,7 +1187,6 @@ include_dirs=["Modules/_sqlite", sqlite_incdir], library_dirs=sqlite_libdir, - runtime_library_dirs=sqlite_libdir, extra_link_args=sqlite_extra_link_args, libraries=["sqlite3",])) else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 18:29:36 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 18:29:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fmW403SDTz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/fe0ad06da28c changeset: 89672:fe0ad06da28c parent: 89669:5c1bacba828d parent: 89670:7cb8faf45539 user: Benjamin Peterson date: Sat Mar 15 12:29:21 2014 -0500 summary: merge 3.3 files: setup.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1138,7 +1138,6 @@ define_macros=sqlite_defines, include_dirs=include_dirs, library_dirs=sqlite_libdir, - runtime_library_dirs=sqlite_libdir, extra_link_args=sqlite_extra_link_args, libraries=["sqlite3",])) else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:19:53 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 15 Mar 2014 21:19:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_Fix_test=5Fgeturl_failure_in_test=5Furllibnet_due_to?= Message-ID: <3fmZrT6cCHz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/9b02f23bc129 changeset: 89673:9b02f23bc129 branch: 2.7 parent: 89671:71da05fa2d8a user: Ned Deily date: Sat Mar 15 13:14:15 2014 -0700 summary: Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. files: Lib/test/test_urllibnet.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -92,7 +92,7 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "http://www.python.org/" + URL = "https://www.python.org/" open_url = self.urlopen(URL) try: gotten_url = open_url.geturl() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -316,6 +316,9 @@ - Issue #20605: Make test_socket getaddrinfo OS X segfault test more robust. +- Issue #20939: Fix test_geturl failure in test_urllibnet due to + new redirect of http://www.python.org/ to https://www.python.org. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:19:55 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 15 Mar 2014 21:19:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_Fix_test=5Fgeturl_failure_in_test=5Furllibnet_due_to?= Message-ID: <3fmZrW24vQz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/030efedd70ca changeset: 89674:030efedd70ca branch: 3.3 parent: 89670:7cb8faf45539 user: Ned Deily date: Sat Mar 15 13:15:31 2014 -0700 summary: Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. files: Lib/test/test_urllibnet.py | 2 +- Misc/NEWS | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -80,7 +80,7 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "http://www.python.org/" + URL = "https://www.python.org/" with self.urlopen(URL) as open_url: gotten_url = open_url.geturl() self.assertEqual(gotten_url, URL) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,12 @@ as documented. The pattern and source keyword parameters are left as deprecated aliases. +Tests +----- + +- Issue #20939: Fix test_geturl failure in test_urllibnet due to + new redirect of http://www.python.org/ to https://www.python.org. + What's New in Python 3.3.5? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:19:56 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 15 Mar 2014 21:19:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320939=3A_merge_from_3=2E3?= Message-ID: <3fmZrX4Vz7z7Lk0@mail.python.org> http://hg.python.org/cpython/rev/9358a41b245b changeset: 89675:9358a41b245b parent: 89672:fe0ad06da28c parent: 89674:030efedd70ca user: Ned Deily date: Sat Mar 15 13:19:20 2014 -0700 summary: Issue #20939: merge from 3.3 files: Lib/test/test_urllibnet.py | 2 +- Misc/NEWS | 4 ++++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -80,7 +80,7 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "http://www.python.org/" + URL = "https://www.python.org/" with self.urlopen(URL) as open_url: gotten_url = open_url.geturl() self.assertEqual(gotten_url, URL) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -72,6 +72,10 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #20939: Fix test_geturl failure in test_urllibnet due to + new redirect of http://www.python.org/ to https://www.python.org. + + Build ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:24:30 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 21:24:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_send_people_to?= =?utf-8?q?_the_right_editors_page_=28=2320938=29?= Message-ID: <3fmZxp4kSRz7LjX@mail.python.org> http://hg.python.org/cpython/rev/772404a89b88 changeset: 89676:772404a89b88 branch: 2.7 parent: 89673:9b02f23bc129 user: Benjamin Peterson date: Sat Mar 15 15:23:32 2014 -0500 summary: send people to the right editors page (#20938) files: Doc/faq/general.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -471,7 +471,8 @@ Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult -http://www.python.org/editors/ for a full list of Python editing environments. +`the Python wiki `_ for a full list +of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:24:32 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 21:24:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_send_people_to?= =?utf-8?q?_the_right_editors_page_=28=2320938=29?= Message-ID: <3fmZxr086bz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/014c2a62c168 changeset: 89677:014c2a62c168 branch: 3.3 parent: 89674:030efedd70ca user: Benjamin Peterson date: Sat Mar 15 15:23:32 2014 -0500 summary: send people to the right editors page (#20938) files: Doc/faq/general.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -471,7 +471,8 @@ Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult -http://www.python.org/editors/ for a full list of Python editing environments. +`the Python wiki `_ for a full list +of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 15 21:24:33 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 15 Mar 2014 21:24:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3fmZxs2fC8z7Ljd@mail.python.org> http://hg.python.org/cpython/rev/f4c15e978483 changeset: 89678:f4c15e978483 parent: 89675:9358a41b245b parent: 89677:014c2a62c168 user: Benjamin Peterson date: Sat Mar 15 15:24:10 2014 -0500 summary: merge 3.3 files: Doc/faq/general.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -471,7 +471,8 @@ Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult -http://www.python.org/editors/ for a full list of Python editing environments. +`the Python wiki `_ for a full list +of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 02:15:36 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 16 Mar 2014 02:15:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_Clarify_headin?= =?utf-8?q?g=2E?= Message-ID: <3fmjPh6xg4z7LjP@mail.python.org> http://hg.python.org/cpython/rev/21ecc3d52806 changeset: 89679:21ecc3d52806 user: R David Murray date: Sat Mar 15 21:15:18 2014 -0400 summary: whatsnew: Clarify heading. files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -84,7 +84,7 @@ * No new syntax features were added in Python 3.4. -New expected features for Python implementations: +New Features: * :ref:`pip should always be "available" ` (:pep:`453`). * :ref:`Newly created file descriptors are non-inheritable ` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 02:16:56 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 16 Mar 2014 02:16:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQ2xvc2UgIzE2NjY1?= =?utf-8?q?=3A_improve_documentation_for_hex=28=29=2E__Patch_by_Jessica_Mc?= =?utf-8?q?Kellar=2E?= Message-ID: <3fmjRD06Tvz7LjP@mail.python.org> http://hg.python.org/cpython/rev/464c22bf9fe1 changeset: 89680:464c22bf9fe1 branch: 3.3 parent: 89677:014c2a62c168 user: Antoine Pitrou date: Sun Mar 16 02:12:20 2014 +0100 summary: Close #16665: improve documentation for hex(). Patch by Jessica McKellar. files: Doc/library/functions.rst | 16 +++++++++++++--- 1 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -608,9 +608,19 @@ .. function:: hex(x) - Convert an integer number to a hexadecimal string. The result is a valid Python - expression. If *x* is not a Python :class:`int` object, it has to define an - :meth:`__index__` method that returns an integer. + Convert an integer number to a lowercase hexadecimal string + prefixed with "0x", for example: + + >>> hex(255) + '0xff' + >>> hex(-42) + '-0x2a' + + If x is not a Python :class:`int` object, it has to define an __index__() + method that returns an integer. + + See also :func:`int` for converting a hexadecimal string to an + integer using a base of 16. .. note:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 02:16:57 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 16 Mar 2014 02:16:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2UgIzE2NjY1?= =?utf-8?q?=3A_improve_documentation_for_hex=28=29=2E__Patch_by_Jessica_Mc?= =?utf-8?q?Kellar=2E?= Message-ID: <3fmjRF23Bmz7LjP@mail.python.org> http://hg.python.org/cpython/rev/d14beaf03f55 changeset: 89681:d14beaf03f55 branch: 2.7 parent: 89676:772404a89b88 user: Antoine Pitrou date: Sun Mar 16 02:12:20 2014 +0100 summary: Close #16665: improve documentation for hex(). Patch by Jessica McKellar. files: Doc/library/functions.rst | 17 +++++++++++++++-- 1 files changed, 15 insertions(+), 2 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -596,8 +596,21 @@ .. function:: hex(x) - Convert an integer number (of any size) to a hexadecimal string. The result is a - valid Python expression. + Convert an integer number (of any size) to a lowercase hexadecimal string + prefixed with "0x", for example: + + >>> hex(255) + '0xff' + >>> hex(-42) + '-0x2a' + >>> hex(1L) + '0x1L' + + If x is not a Python :class:`int` or :class:`long` object, it has to + define an __index__() method that returns an integer. + + See also :func:`int` for converting a hexadecimal string to an + integer using a base of 16. .. note:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 02:16:58 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 16 Mar 2014 02:16:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Close_=2316665=3A_improve_documentation_for_hex=28=29=2E?= =?utf-8?q?__Patch_by_Jessica_McKellar=2E?= Message-ID: <3fmjRG40Jrz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/c267f4eb8173 changeset: 89682:c267f4eb8173 parent: 89679:21ecc3d52806 parent: 89680:464c22bf9fe1 user: Antoine Pitrou date: Sun Mar 16 02:13:07 2014 +0100 summary: Close #16665: improve documentation for hex(). Patch by Jessica McKellar. files: Doc/library/functions.rst | 16 +++++++++++++--- 1 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -617,9 +617,19 @@ .. function:: hex(x) - Convert an integer number to a hexadecimal string. The result is a valid Python - expression. If *x* is not a Python :class:`int` object, it has to define an - :meth:`__index__` method that returns an integer. + Convert an integer number to a lowercase hexadecimal string + prefixed with "0x", for example: + + >>> hex(255) + '0xff' + >>> hex(-42) + '-0x2a' + + If x is not a Python :class:`int` object, it has to define an __index__() + method that returns an integer. + + See also :func:`int` for converting a hexadecimal string to an + integer using a base of 16. .. note:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 02:41:51 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 16 Mar 2014 02:41:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Issue_=2320926=3A_Enhance?= =?utf-8?q?_Quick_Start_portion_of_instructions_for_Windows=2E__Patch?= Message-ID: <3fmjzz1tVRz7LjN@mail.python.org> http://hg.python.org/devguide/rev/a775fe9e7f86 changeset: 676:a775fe9e7f86 user: Antoine Pitrou date: Sun Mar 16 02:41:37 2014 +0100 summary: Issue #20926: Enhance Quick Start portion of instructions for Windows. Patch by Kathleen Weaver. files: index.rst | 3 ++- setup.rst | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -27,7 +27,8 @@ and :menuselection:`Build --> Build Solution`. Run :file:`Tools\\buildbot\\external.bat` or :file:`Tools\\buildbot\\external-amd64.bat` to download and compile 3rd - party libraries. + party libraries. Note: Visual Studio may throw errors, but Python will + be built. 3. :doc:`Run the tests `:: diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -217,6 +217,9 @@ Windows ''''''' +The readme included in the solution has more details, especially on the +software needed to resolve the below mentioned build errors. + **Python 3.3** and later use Microsoft Visual Studio 2010. You can download Microsoft Visual C++ 2010 Express `from Microsoft's site `_. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sun Mar 16 04:16:01 2014 From: python-checkins at python.org (larry.hastings) Date: Sun, 16 Mar 2014 04:16:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320931=3A_Fix_othe?= =?utf-8?q?r_confusingly-worded_heading=2E?= Message-ID: <3fmm4d2jq8z7LjM@mail.python.org> http://hg.python.org/cpython/rev/3488a9fbe09e changeset: 89683:3488a9fbe09e user: Larry Hastings date: Sat Mar 15 20:15:55 2014 -0700 summary: Issue #20931: Fix other confusingly-worded heading. files: Doc/whatsnew/3.4.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -174,8 +174,8 @@ -New Expected Features for Python Implementations -================================================ +New Features +============ .. _whatsnew-pep-453: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 04:31:26 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 16 Mar 2014 04:31:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_Fix_Features_s?= =?utf-8?q?ection_title=2C_use_title/sentence_case_consistently=2E?= Message-ID: <3fmmQQ2Y4yz7LjT@mail.python.org> http://hg.python.org/cpython/rev/7e78e602ec73 changeset: 89684:7e78e602ec73 user: R David Murray date: Sat Mar 15 23:27:30 2014 -0400 summary: whatsnew: Fix Features section title, use title/sentence case consistently. closes #20931 files: Doc/whatsnew/3.4.rst | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -84,9 +84,9 @@ * No new syntax features were added in Python 3.4. -New Features: - -* :ref:`pip should always be "available" ` (:pep:`453`). +Other new features: + +* :ref:`pip should always be available ` (:pep:`453`). * :ref:`Newly created file descriptors are non-inheritable ` (:pep:`446`). * command line option for :ref:`isolated mode ` @@ -116,7 +116,7 @@ * :mod:`tracemalloc`: :ref:`Trace Python memory allocations ` (:pep:`454`). -Significantly Improved Library Modules: +Significantly improved library modules: * :ref:`Single-dispatch generic functions ` in :mod:`functools` (:pep:`443`). @@ -182,7 +182,7 @@ PEP 453: Explicit Bootstrapping of PIP in Python Installations -------------------------------------------------------------- -Bootstrapping pip by default +Bootstrapping pip By Default ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard @@ -214,7 +214,7 @@ __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors -Documentation changes +Documentation Changes ~~~~~~~~~~~~~~~~~~~~~ As part of this change, the :ref:`installing-index` and @@ -2223,7 +2223,7 @@ that may require changes to your code. -Changes in 'python' command behavior +Changes in 'python' Command Behavior ------------------------------------ * In a posix shell, setting the :envvar:`PATH` environment variable to -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 04:59:09 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 16 Mar 2014 04:59:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Bring_PEP_453_What=27s_New?= =?utf-8?q?_entry_up_to_date?= Message-ID: <3fmn2P5GdBz7LjT@mail.python.org> http://hg.python.org/cpython/rev/74144b0e7858 changeset: 89685:74144b0e7858 user: Nick Coghlan date: Sun Mar 16 13:55:19 2014 +1000 summary: Bring PEP 453 What's New entry up to date We changed a few details of the pip integration over the course of the Python 3.4 beta and release candidate cycle, but I hadn't updated the What's New entry appropriately. The Py2/Py3 split on POSIX and the generally challenges of dealing with parallel installs means we haven't attained complete consistency across all environments, but hopefully this updated text is clearer about the end result. Relevant to issue #20909. files: Doc/whatsnew/3.4.rst | 55 +++++++++++++++++++------------ 1 files changed, 34 insertions(+), 21 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -187,16 +187,23 @@ The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard cross-platform mechanism to bootstrap the pip installer into Python -installations and virtual environments. - -By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where -X.Y stands for the version of the Python installation), along with the -``pip`` Python package and its dependencies. +installations and virtual environments. The version of ``pip`` included +with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases +will update the bundled version to the latest version of ``pip`` that is +available at the time of creating the release candidate. + +By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all +platforms (where X.Y stands for the version of the Python installation), +along with the ``pip`` Python package and its dependencies. On Windows and +in virtual environments on all platforms, the unversioned ``pip`` command +will also be installed. On other platforms, the system wide unversioned +``pip`` command typically refers to the separately installed Python 2 +version. The :ref:`pyvenv ` command line utility and the :mod:`venv` module make use of the :mod:`ensurepip` module to make ``pip`` readily -available in virtual environments. When using the command line utility, ``pip`` -is installed by default, while when using the :mod:`venv` module +available in virtual environments. When using the command line utility, +``pip`` is installed by default, while when using the :mod:`venv` module :ref:`venv-api` installation of ``pip`` must be requested explicitly. For CPython :ref:`source builds on POSIX systems `, @@ -204,15 +211,31 @@ default. This behaviour can be controlled through configure options, and overridden through Makefile options. -On Windows and Mac OS X, the CPython installers now offer the option to -install ``pip`` along with CPython itself. +On Windows and Mac OS X, the CPython installers now default to installing +``pip`` along with CPython itself (users may opt out of installing it +during the installation process). Window users will need to opt in to the +automatic ``PATH`` modifications to have ``pip`` available from the command +line by default, otherwise it can still be accessed through the Python +launcher for Windows as ``py -m pip``. As `discussed in the PEP`__, platform packagers may choose not to install -``pip`` by default, as long as the command ``pip``, when invoked, provides -clear and simple directions on how to install ``pip`` on the platform. +these commands by default, as long as, when invoked, they provide clear and +simple directions on how to install them on that platform (usually using +the system package manager). __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors +.. note:: + + To avoid conflicts between parallel Python 2 and Python 3 installations, + only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by + default when ``ensurepip`` is invoked directly - the ``--default-pip`` + option is needed to also request the unversioned ``pip`` command. + ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` + command is made available in those environments, and ``pip`` can always be + invoked via the ``-m`` switch rather than directly to avoid ambiguity on + systems with multiple Python installations. + Documentation Changes ~~~~~~~~~~~~~~~~~~~~~ @@ -229,16 +252,6 @@ versions of those guides remaining available as :ref:`install-index` and :ref:`distutils-index`. -.. note:: - - To avoid conflicts between parallel Python 2 and Python 3 installations, - only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by - default when ``ensurepip`` is invoked directly (including by the CPython - installers). ``pyvenv`` ensures that the unqualified ``pip`` command is - made available in virtual environments, and ``pip`` can always be - invoked via the ``-m`` switch rather than directly to avoid ambiguity on - systems with multiple Python installations. - .. seealso:: :pep:`453` -- Explicit bootstrapping of pip in Python installations -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 05:43:21 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 16 Mar 2014 05:43:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_statistics_module_=26_new_has?= =?utf-8?q?h_are_in_3=2E4?= Message-ID: <3fmp1P0K4Cz7LjW@mail.python.org> http://hg.python.org/peps/rev/37a8fdcf4940 changeset: 5410:37a8fdcf4940 user: Nick Coghlan date: Sun Mar 16 14:43:06 2014 +1000 summary: statistics module & new hash are in 3.4 files: pep-0450.txt | 2 +- pep-0456.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0450.txt b/pep-0450.txt --- a/pep-0450.txt +++ b/pep-0450.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Steven D'Aprano -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/plain Created: 01-Aug-2013 diff --git a/pep-0456.txt b/pep-0456.txt --- a/pep-0456.txt +++ b/pep-0456.txt @@ -4,7 +4,7 @@ Last-Modified: $Date$ Author: Christian Heimes BDFL-Delegate: Nick Coghlan -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 27-Sep-2013 -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Sun Mar 16 09:54:52 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 16 Mar 2014 09:54:52 +0100 Subject: [Python-checkins] Daily reference leaks (c267f4eb8173): sum=0 Message-ID: results for c267f4eb8173 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 4] memory blocks, sum=4 test_site leaked [0, -2, 0] references, sum=-2 test_site leaked [0, -2, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogIA27Yk', '-x'] From vinay_sajip at yahoo.co.uk Sat Mar 15 00:06:50 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 14 Mar 2014 23:06:50 +0000 (GMT) Subject: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: <1394838410.68259.YahooMailNeo@web172402.mail.ir2.yahoo.com> Sorry, my mistake - I should have checked more carefully. Thank you for pointing me to the commit where they were removed - I'll install a venv with Sphinx, revert the change and remove those files from my repo. Regards, Vinay Sajip ________________________________ From: Benjamin Peterson To: python-dev at python.org; python-checkins at python.org; vinay_sajip at yahoo.co.uk Sent: Friday, 14 March 2014, 15:10 Subject: Re: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > http://hg.python.org/cpython/rev/edb12dad7bf6 > changeset:? 89654:edb12dad7bf6 > user:? ? ? ? Vinay Sajip > date:? ? ? ? Fri Mar 14 14:20:09 2014 +0000 > summary: >? Added Doc/tools/ subdirs to .hgignore. > > files: >? .hgignore |? 5 +++++ >? 1 files changed, 5 insertions(+), 0 deletions(-) > > > diff --git a/.hgignore b/.hgignore > --- a/.hgignore > +++ b/.hgignore > @@ -24,6 +24,11 @@ >? reflog.txt$ >? tags$ >? Lib/plat-mac/errors.rsrc.df.rsrc > +Doc/tools/sphinx/ > +Doc/tools/docutils/ > +Doc/tools/jinja/ > +Doc/tools/jinja2/ > +Doc/tools/pygments/ Why are you readding these when they were apparently purposely removed in eef7899ea7ab? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Sun Mar 16 17:31:12 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 16 Mar 2014 17:31:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_fix_ctypes_tes?= =?utf-8?q?t_alignment_assumptions_=28closes_=2320946=29?= Message-ID: <3fn5k82jkHz7LjM@mail.python.org> http://hg.python.org/cpython/rev/2150053d77ca changeset: 89686:2150053d77ca branch: 3.3 parent: 89680:464c22bf9fe1 user: Benjamin Peterson date: Sun Mar 16 10:07:26 2014 +0100 summary: fix ctypes test alignment assumptions (closes #20946) Patch by Andreas Schwab. files: Lib/ctypes/test/test_bitfields.py | 2 +- Lib/ctypes/test/test_structures.py | 18 +++++++++--------- Misc/NEWS | 2 ++ Modules/_ctypes/cfield.c | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/Lib/ctypes/test/test_bitfields.py b/Lib/ctypes/test/test_bitfields.py --- a/Lib/ctypes/test/test_bitfields.py +++ b/Lib/ctypes/test/test_bitfields.py @@ -207,7 +207,7 @@ class X(Structure): _fields_ = [("a", c_byte, 4), ("b", c_int, 32)] - self.assertEqual(sizeof(X), sizeof(c_int)*2) + self.assertEqual(sizeof(X), alignment(c_int)+sizeof(c_int)) def test_mixed_3(self): class X(Structure): diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -83,7 +83,7 @@ class Y(Structure): _fields_ = [("x", c_char * 3), ("y", c_int)] - self.assertEqual(alignment(Y), calcsize("i")) + self.assertEqual(alignment(Y), alignment(c_int)) self.assertEqual(sizeof(Y), calcsize("3si")) class SI(Structure): @@ -175,13 +175,6 @@ self.assertEqual(sizeof(X), 10) self.assertEqual(X.b.offset, 2) - class X(Structure): - _fields_ = [("a", c_byte), - ("b", c_longlong)] - _pack_ = 4 - self.assertEqual(sizeof(X), 12) - self.assertEqual(X.b.offset, 4) - import struct longlong_size = struct.calcsize("q") longlong_align = struct.calcsize("bq") - longlong_size @@ -189,9 +182,16 @@ class X(Structure): _fields_ = [("a", c_byte), ("b", c_longlong)] + _pack_ = 4 + self.assertEqual(sizeof(X), min(4, longlong_align) + longlong_size) + self.assertEqual(X.b.offset, min(4, longlong_align)) + + class X(Structure): + _fields_ = [("a", c_byte), + ("b", c_longlong)] _pack_ = 8 - self.assertEqual(sizeof(X), longlong_align + longlong_size) + self.assertEqual(sizeof(X), min(8, longlong_align) + longlong_size) self.assertEqual(X.b.offset, min(8, longlong_align)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -30,6 +30,8 @@ Tests ----- +- Issue #20946: Correct alignment assumptions of some ctypes tests. + - Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1640,9 +1640,9 @@ /* #define CHAR_ALIGN (sizeof(s_char) - sizeof(char)) #define SHORT_ALIGN (sizeof(s_short) - sizeof(short)) -#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define LONG_ALIGN (sizeof(s_long) - sizeof(long)) */ +#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define FLOAT_ALIGN (sizeof(s_float) - sizeof(float)) #define DOUBLE_ALIGN (sizeof(s_double) - sizeof(double)) #define LONGDOUBLE_ALIGN (sizeof(s_long_double) - sizeof(long double)) @@ -1684,8 +1684,8 @@ ffi_type ffi_type_uint16 = { 2, 2, FFI_TYPE_UINT16 }; ffi_type ffi_type_sint16 = { 2, 2, FFI_TYPE_SINT16 }; -ffi_type ffi_type_uint32 = { 4, 4, FFI_TYPE_UINT32 }; -ffi_type ffi_type_sint32 = { 4, 4, FFI_TYPE_SINT32 }; +ffi_type ffi_type_uint32 = { 4, INT_ALIGN, FFI_TYPE_UINT32 }; +ffi_type ffi_type_sint32 = { 4, INT_ALIGN, FFI_TYPE_SINT32 }; ffi_type ffi_type_uint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_UINT64 }; ffi_type ffi_type_sint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_SINT64 }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 17:31:13 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 16 Mar 2014 17:31:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_ctypes_tes?= =?utf-8?q?t_alignment_assumptions_=28closes_=2320946=29?= Message-ID: <3fn5k96ZfMz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/e5a09b09bb51 changeset: 89687:e5a09b09bb51 branch: 2.7 parent: 89681:d14beaf03f55 user: Benjamin Peterson date: Sun Mar 16 10:07:26 2014 +0100 summary: fix ctypes test alignment assumptions (closes #20946) Patch by Andreas Schwab. files: Lib/ctypes/test/test_bitfields.py | 2 +- Lib/ctypes/test/test_structures.py | 18 +++++++++--------- Misc/NEWS | 2 ++ Modules/_ctypes/cfield.c | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/Lib/ctypes/test/test_bitfields.py b/Lib/ctypes/test/test_bitfields.py --- a/Lib/ctypes/test/test_bitfields.py +++ b/Lib/ctypes/test/test_bitfields.py @@ -207,7 +207,7 @@ class X(Structure): _fields_ = [("a", c_byte, 4), ("b", c_int, 32)] - self.assertEqual(sizeof(X), sizeof(c_int)*2) + self.assertEqual(sizeof(X), alignment(c_int)+sizeof(c_int)) def test_mixed_3(self): class X(Structure): diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -83,7 +83,7 @@ class Y(Structure): _fields_ = [("x", c_char * 3), ("y", c_int)] - self.assertEqual(alignment(Y), calcsize("i")) + self.assertEqual(alignment(Y), alignment(c_int)) self.assertEqual(sizeof(Y), calcsize("3si")) class SI(Structure): @@ -175,13 +175,6 @@ self.assertEqual(sizeof(X), 10) self.assertEqual(X.b.offset, 2) - class X(Structure): - _fields_ = [("a", c_byte), - ("b", c_longlong)] - _pack_ = 4 - self.assertEqual(sizeof(X), 12) - self.assertEqual(X.b.offset, 4) - import struct longlong_size = struct.calcsize("q") longlong_align = struct.calcsize("bq") - longlong_size @@ -189,9 +182,16 @@ class X(Structure): _fields_ = [("a", c_byte), ("b", c_longlong)] + _pack_ = 4 + self.assertEqual(sizeof(X), min(4, longlong_align) + longlong_size) + self.assertEqual(X.b.offset, min(4, longlong_align)) + + class X(Structure): + _fields_ = [("a", c_byte), + ("b", c_longlong)] _pack_ = 8 - self.assertEqual(sizeof(X), longlong_align + longlong_size) + self.assertEqual(sizeof(X), min(8, longlong_align) + longlong_size) self.assertEqual(X.b.offset, min(8, longlong_align)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -271,6 +271,8 @@ Tests ----- +- Issue #20946: Correct alignment assumptions of some ctypes tests. + - Issue #20743: Fix a reference leak in test_tcl. - Issue #20510: Rewrote test_exit in test_sys to match existing comments, diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1687,9 +1687,9 @@ /* #define CHAR_ALIGN (sizeof(s_char) - sizeof(char)) #define SHORT_ALIGN (sizeof(s_short) - sizeof(short)) -#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define LONG_ALIGN (sizeof(s_long) - sizeof(long)) */ +#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define FLOAT_ALIGN (sizeof(s_float) - sizeof(float)) #define DOUBLE_ALIGN (sizeof(s_double) - sizeof(double)) #define LONGDOUBLE_ALIGN (sizeof(s_long_double) - sizeof(long double)) @@ -1731,8 +1731,8 @@ ffi_type ffi_type_uint16 = { 2, 2, FFI_TYPE_UINT16 }; ffi_type ffi_type_sint16 = { 2, 2, FFI_TYPE_SINT16 }; -ffi_type ffi_type_uint32 = { 4, 4, FFI_TYPE_UINT32 }; -ffi_type ffi_type_sint32 = { 4, 4, FFI_TYPE_SINT32 }; +ffi_type ffi_type_uint32 = { 4, INT_ALIGN, FFI_TYPE_UINT32 }; +ffi_type ffi_type_sint32 = { 4, INT_ALIGN, FFI_TYPE_SINT32 }; ffi_type ffi_type_uint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_UINT64 }; ffi_type ffi_type_sint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_SINT64 }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 17:31:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 16 Mar 2014 17:31:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMjA5NDYp?= Message-ID: <3fn5kC2VCXz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/3736bf94535c changeset: 89688:3736bf94535c parent: 89685:74144b0e7858 parent: 89686:2150053d77ca user: Benjamin Peterson date: Sun Mar 16 11:30:54 2014 -0500 summary: merge 3.3 (#20946) files: Lib/ctypes/test/test_bitfields.py | 2 +- Lib/ctypes/test/test_structures.py | 18 +++++++++--------- Misc/NEWS | 2 ++ Modules/_ctypes/cfield.c | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/Lib/ctypes/test/test_bitfields.py b/Lib/ctypes/test/test_bitfields.py --- a/Lib/ctypes/test/test_bitfields.py +++ b/Lib/ctypes/test/test_bitfields.py @@ -207,7 +207,7 @@ class X(Structure): _fields_ = [("a", c_byte, 4), ("b", c_int, 32)] - self.assertEqual(sizeof(X), sizeof(c_int)*2) + self.assertEqual(sizeof(X), alignment(c_int)+sizeof(c_int)) def test_mixed_3(self): class X(Structure): diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -83,7 +83,7 @@ class Y(Structure): _fields_ = [("x", c_char * 3), ("y", c_int)] - self.assertEqual(alignment(Y), calcsize("i")) + self.assertEqual(alignment(Y), alignment(c_int)) self.assertEqual(sizeof(Y), calcsize("3si")) class SI(Structure): @@ -175,13 +175,6 @@ self.assertEqual(sizeof(X), 10) self.assertEqual(X.b.offset, 2) - class X(Structure): - _fields_ = [("a", c_byte), - ("b", c_longlong)] - _pack_ = 4 - self.assertEqual(sizeof(X), 12) - self.assertEqual(X.b.offset, 4) - import struct longlong_size = struct.calcsize("q") longlong_align = struct.calcsize("bq") - longlong_size @@ -189,9 +182,16 @@ class X(Structure): _fields_ = [("a", c_byte), ("b", c_longlong)] + _pack_ = 4 + self.assertEqual(sizeof(X), min(4, longlong_align) + longlong_size) + self.assertEqual(X.b.offset, min(4, longlong_align)) + + class X(Structure): + _fields_ = [("a", c_byte), + ("b", c_longlong)] _pack_ = 8 - self.assertEqual(sizeof(X), longlong_align + longlong_size) + self.assertEqual(sizeof(X), min(8, longlong_align) + longlong_size) self.assertEqual(X.b.offset, min(8, longlong_align)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -72,6 +72,8 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #20946: Correct alignment assumptions of some ctypes tests. + - Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1640,9 +1640,9 @@ /* #define CHAR_ALIGN (sizeof(s_char) - sizeof(char)) #define SHORT_ALIGN (sizeof(s_short) - sizeof(short)) -#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define LONG_ALIGN (sizeof(s_long) - sizeof(long)) */ +#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define FLOAT_ALIGN (sizeof(s_float) - sizeof(float)) #define DOUBLE_ALIGN (sizeof(s_double) - sizeof(double)) #define LONGDOUBLE_ALIGN (sizeof(s_long_double) - sizeof(long double)) @@ -1684,8 +1684,8 @@ ffi_type ffi_type_uint16 = { 2, 2, FFI_TYPE_UINT16 }; ffi_type ffi_type_sint16 = { 2, 2, FFI_TYPE_SINT16 }; -ffi_type ffi_type_uint32 = { 4, 4, FFI_TYPE_UINT32 }; -ffi_type ffi_type_sint32 = { 4, 4, FFI_TYPE_SINT32 }; +ffi_type ffi_type_uint32 = { 4, INT_ALIGN, FFI_TYPE_UINT32 }; +ffi_type ffi_type_sint32 = { 4, INT_ALIGN, FFI_TYPE_SINT32 }; ffi_type ffi_type_uint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_UINT64 }; ffi_type ffi_type_sint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_SINT64 }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 21:29:53 2014 From: python-checkins at python.org (victor.stinner) Date: Sun, 16 Mar 2014 21:29:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320950=3A_Fix_typo?= =?utf-8?q?_asyncio_doc=2C_wait=28=29_has_no_self_parameter?= Message-ID: <3fnC1Y5X1Gz7LjS@mail.python.org> http://hg.python.org/cpython/rev/1009cf8cb304 changeset: 89689:1009cf8cb304 user: Victor Stinner date: Sun Mar 16 21:29:31 2014 +0100 summary: Issue #20950: Fix typo asyncio doc, wait() has no self parameter self parameter is implicit. Mention also that communicate() and wait() are coroutines. files: Doc/library/asyncio-subprocess.rst | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -107,6 +107,8 @@ The data read is buffered in memory, so do not use this method if the data size is large or unlimited. + This method is a :ref:`coroutine `. + .. method:: kill() Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to @@ -129,11 +131,13 @@ to the child. On Windows the Win32 API function :c:func:`TerminateProcess` is called to stop the child. - .. method:: wait(self): + .. method:: wait(): Wait for child process to terminate. Set and return :attr:`returncode` attribute. + This method is a :ref:`coroutine `. + Example ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 21:56:36 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 16 Mar 2014 21:56:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogaW1wcm92ZSBzZXRf?= =?utf-8?q?tunnel_docs_=28closes_=2311448=29?= Message-ID: <3fnCcN22PYz7LjW@mail.python.org> http://hg.python.org/cpython/rev/68a257ca6be6 changeset: 89690:68a257ca6be6 branch: 3.3 parent: 89686:2150053d77ca user: Benjamin Peterson date: Sun Mar 16 15:55:39 2014 -0500 summary: improve set_tunnel docs (closes #11448) Patch by Ryan Kelly, karl, and Nikolaus Rath. files: Doc/library/http.client.rst | 22 ++++++++++++++++++---- 1 files changed, 18 insertions(+), 4 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -451,11 +451,25 @@ .. method:: HTTPConnection.set_tunnel(host, port=None, headers=None) - Set the host and the port for HTTP Connect Tunnelling. Normally used when it - is required to a HTTPS Connection through a proxy server. + Set the host and the port for HTTP Connect Tunnelling. This allows running + the connection through a proxy server. - The headers argument should be a mapping of extra HTTP headers to send - with the CONNECT request. + The host and port arguments specify the endpoint of the tunneled connection + (i.e. the address included in the CONNECT request, *not* the address of the + proxy server). + + The headers argument should be a mapping of extra HTTP headers to send with + the CONNECT request. + + For example, to tunnel through a HTTPS proxy server running locally on port + 8080, we would pass the address of the proxy to the :class:`HTTPSConnection` + constructor, and the address of the host that we eventually want to reach to + the :meth:`~HTTPConnection.set_tunnel` method:: + + >>> import http.client + >>> conn = http.client.HTTPSConnection("localhost", 8080) + >>> conn.set_tunnel("www.python.org") + >>> conn.request("HEAD","/index.html") .. versionadded:: 3.2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 16 21:56:37 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 16 Mar 2014 21:56:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTE0NDgp?= Message-ID: <3fnCcP3jFRz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/5cab0ada97b2 changeset: 89691:5cab0ada97b2 parent: 89689:1009cf8cb304 parent: 89690:68a257ca6be6 user: Benjamin Peterson date: Sun Mar 16 15:56:24 2014 -0500 summary: merge 3.3 (#11448) files: Doc/library/http.client.rst | 22 ++++++++++++++++++---- 1 files changed, 18 insertions(+), 4 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -451,11 +451,25 @@ .. method:: HTTPConnection.set_tunnel(host, port=None, headers=None) - Set the host and the port for HTTP Connect Tunnelling. Normally used when it - is required to a HTTPS Connection through a proxy server. + Set the host and the port for HTTP Connect Tunnelling. This allows running + the connection through a proxy server. - The headers argument should be a mapping of extra HTTP headers to send - with the CONNECT request. + The host and port arguments specify the endpoint of the tunneled connection + (i.e. the address included in the CONNECT request, *not* the address of the + proxy server). + + The headers argument should be a mapping of extra HTTP headers to send with + the CONNECT request. + + For example, to tunnel through a HTTPS proxy server running locally on port + 8080, we would pass the address of the proxy to the :class:`HTTPSConnection` + constructor, and the address of the host that we eventually want to reach to + the :meth:`~HTTPConnection.set_tunnel` method:: + + >>> import http.client + >>> conn = http.client.HTTPSConnection("localhost", 8080) + >>> conn.set_tunnel("www.python.org") + >>> conn.request("HEAD","/index.html") .. versionadded:: 3.2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:23:56 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:23:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_allow-branch_hook_update_?= =?utf-8?q?instructions=2E?= Message-ID: <3fnSC02Schz7LjW@mail.python.org> http://hg.python.org/peps/rev/9b9d06e19586 changeset: 5411:9b9d06e19586 user: Georg Brandl date: Mon Mar 17 07:23:38 2014 +0100 summary: Add allow-branch hook update instructions. files: pep-0101.txt | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -38,6 +38,9 @@ * Access to ``dinsdale.python.org``, the Python web host. You'll be uploading files directly here. + * Shell access to ``hg.python.org``, the Python Mercurial host. You'll + have to adapt repository configuration there. + * Write access to the website SVN repository Instructions here: @@ -266,6 +269,12 @@ $ hg update deadbeef # revision ID noted down before $ hg branch 3.2 + ___ When you want to push back your new branch to the main CPython + repository, the new branch name must be added to the "allow-branches" + hook configuration, which protects against stray named branches being + pushed. Login to hg.python.org and edit (as the "hg" user) + ``/data/hg/repos/cpython/.hg/hgrc`` to that effect. + ___ Push your commits to the remote release clone. $ hg push ssh://hg.python.org/releasing/... -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 07:31:09 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:31:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_version_switcher_update_i?= =?utf-8?q?nstructions=2E?= Message-ID: <3fnSMK08w9z7Lk3@mail.python.org> http://hg.python.org/peps/rev/a62ae7ce5267 changeset: 5412:a62ae7ce5267 user: Georg Brandl date: Mon Mar 17 07:30:58 2014 +0100 summary: Add version switcher update instructions. files: pep-0101.txt | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -275,6 +275,10 @@ pushed. Login to hg.python.org and edit (as the "hg" user) ``/data/hg/repos/cpython/.hg/hgrc`` to that effect. + ___ For a final major release, Doc/tools/sphinxext/static/version_switch.js + must be updated in all maintained branches, so that the new maintenance + branch is not "dev" anymore and there is a new "dev" version. + ___ Push your commits to the remote release clone. $ hg push ssh://hg.python.org/releasing/... -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 07:31:15 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Created_releas?= =?utf-8?q?e_branch_for_3=2E4=2E?= Message-ID: <3fnSMS01ZNz7LjY@mail.python.org> http://hg.python.org/cpython/rev/c9aced9cca53 changeset: 89692:c9aced9cca53 branch: 3.4 parent: 89136:e64ae8b82672 user: Larry Hastings date: Thu Feb 20 23:38:01 2014 -0800 summary: Created release branch for 3.4. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:17 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_Use_even_shorter_sleep_in_test=5Ftimeout=5Frounding=28=29_t?= =?utf-8?q?o_make_the?= Message-ID: <3fnSMT1gRfz7LjY@mail.python.org> http://hg.python.org/cpython/rev/4f1df287392c changeset: 89693:4f1df287392c branch: 3.4 user: Victor Stinner date: Mon Feb 10 19:17:46 2014 +0100 summary: Issue #20505: Use even shorter sleep in test_timeout_rounding() to make the test more reliable (= fail more often on Windows with HPET enabled). files: Lib/test/test_asyncio/test_events.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1176,13 +1176,15 @@ loop = self.loop yield from asyncio.sleep(1e-2, loop=loop) yield from asyncio.sleep(1e-4, loop=loop) + yield from asyncio.sleep(1e-6, loop=loop) + yield from asyncio.sleep(1e-8, loop=loop) self.loop.run_until_complete(wait()) - # The ideal number of call is 6, but on some platforms, the selector + # The ideal number of call is 10, but on some platforms, the selector # may sleep at little bit less than timeout depending on the resolution - # of the clock used by the kernel. Tolerate 2 useless calls on these + # of the clock used by the kernel. Tolerate 5 useless calls on these # platforms. - self.assertLessEqual(self.loop._run_once_counter, 8, + self.assertLessEqual(self.loop._run_once_counter, 15, {'time_info': time.get_clock_info('time'), 'monotonic_info': time.get_clock_info('monotonic'), 'selector': self.loop._selector.__class__.__name__}) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:18 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_merge_with_3?= =?utf-8?q?=2E3?= Message-ID: <3fnSMV4PMxz7LjV@mail.python.org> http://hg.python.org/cpython/rev/bce5fc7c081f changeset: 89694:bce5fc7c081f branch: 3.4 user: Georg Brandl date: Mon Feb 10 22:11:21 2014 +0100 summary: merge with 3.3 files: .hgtags | 2 + Doc/tools/sphinxext/susp-ignored.csv | 1 + Lib/idlelib/NEWS.txt | 1 - Lib/smtplib.py | 5 ++- Lib/test/mock_socket.py | 9 +++- Lib/test/test_smtplib.py | 30 +++++++++++++++- 6 files changed, 43 insertions(+), 5 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -118,6 +118,8 @@ fd53c500f8b80f54f3ecedec9da2e8c7e52a6888 v3.3.3rc1 d32442c0e60dfbd71234e807d3d1dedd227495a9 v3.3.3rc2 c3896275c0f61b2510a6c7e6c458a750359a91b8 v3.3.3 +fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 +7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 46535f65e7f3bcdcf176f36d34bc1fed719ffd2b v3.4.0a1 9265a2168e2cb2a84785d8717792acc661e6b692 v3.4.0a2 dd9cdf90a5073510877e9dd5112f8e6cf20d5e89 v3.4.0a3 diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -282,3 +282,4 @@ whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" whatsnew/changelog,,:password,user:password +whatsnew/changelog,,:gz,w:gz diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -875,4 +875,3 @@ -------------------------------------------------------------------- Refer to HISTORY.txt for additional information on earlier releases. -------------------------------------------------------------------- - diff --git a/Lib/smtplib.py b/Lib/smtplib.py --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -62,6 +62,7 @@ SMTP_SSL_PORT = 465 CRLF = "\r\n" bCRLF = b"\r\n" +_MAXLINE = 8192 # more than 8 times larger than RFC 821, 4.5.3 OLDSTYLE_AUTH = re.compile(r"auth=(.*)", re.I) @@ -365,7 +366,7 @@ self.file = self.sock.makefile('rb') while 1: try: - line = self.file.readline() + line = self.file.readline(_MAXLINE + 1) except OSError as e: self.close() raise SMTPServerDisconnected("Connection unexpectedly closed: " @@ -375,6 +376,8 @@ raise SMTPServerDisconnected("Connection unexpectedly closed") if self.debuglevel > 0: print('reply:', repr(line), file=stderr) + if len(line) > _MAXLINE: + raise SMTPResponseException(500, "Line too long.") resp.append(line[4:].strip(b' \t\r\n')) code = line[:3] # Check that the error code is syntactically correct. diff --git a/Lib/test/mock_socket.py b/Lib/test/mock_socket.py --- a/Lib/test/mock_socket.py +++ b/Lib/test/mock_socket.py @@ -21,8 +21,13 @@ """ def __init__(self, lines): self.lines = lines - def readline(self): - return self.lines.pop(0) + b'\r\n' + def readline(self, limit=-1): + result = self.lines.pop(0) + b'\r\n' + if limit >= 0: + # Re-insert the line, removing the \r\n we added. + self.lines.insert(0, result[limit:-2]) + result = result[:limit] + return result def close(self): pass diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -563,6 +563,33 @@ HOST, self.port, 'localhost', 3) + at unittest.skipUnless(threading, 'Threading required for this test.') +class TooLongLineTests(unittest.TestCase): + respdata = b'250 OK' + (b'.' * smtplib._MAXLINE * 2) + b'\n' + + def setUp(self): + self.old_stdout = sys.stdout + self.output = io.StringIO() + sys.stdout = self.output + + self.evt = threading.Event() + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.settimeout(15) + self.port = support.bind_port(self.sock) + servargs = (self.evt, self.respdata, self.sock) + threading.Thread(target=server, args=servargs).start() + self.evt.wait() + self.evt.clear() + + def tearDown(self): + self.evt.wait() + sys.stdout = self.old_stdout + + def testLineTooLong(self): + self.assertRaises(smtplib.SMTPResponseException, smtplib.SMTP, + HOST, self.port, 'localhost', 3) + + sim_users = {'Mr.A at somewhere.com':'John A', 'Ms.B at xn--fo-fka.com':'Sally B', 'Mrs.C at somewhereesle.com':'Ruth C', @@ -888,7 +915,8 @@ def test_main(verbose=None): support.run_unittest(GeneralTests, DebuggingServerTests, NonConnectingTests, - BadHELOServerTests, SMTPSimTests) + BadHELOServerTests, SMTPSimTests, + TooLongLineTests) if __name__ == '__main__': test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:19 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_BaseEventLoop_uses_again_the_resolution_of_the_clock_to_dec?= =?utf-8?q?ide_if?= Message-ID: <3fnSMW6tXWz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/6733d9dfffbb changeset: 89695:6733d9dfffbb branch: 3.4 user: Victor Stinner date: Mon Feb 10 23:42:32 2014 +0100 summary: Issue #20505: BaseEventLoop uses again the resolution of the clock to decide if scheduled tasks should be executed or not. files: Lib/asyncio/base_events.py | 5 +++-- Lib/test/test_asyncio/test_events.py | 12 ++++++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -96,6 +96,7 @@ self._default_executor = None self._internal_fds = 0 self._running = False + self._clock_resolution = time.get_clock_info('monotonic').resolution def _make_socket_transport(self, sock, protocol, waiter=None, *, extra=None, server=None): @@ -643,10 +644,10 @@ self._process_events(event_list) # Handle 'later' callbacks that are ready. - now = self.time() + end_time = self.time() + self._clock_resolution while self._scheduled: handle = self._scheduled[0] - if handle._when > now: + if handle._when >= end_time: break handle = heapq.heappop(self._scheduled) self._ready.append(handle) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1178,15 +1178,15 @@ yield from asyncio.sleep(1e-4, loop=loop) yield from asyncio.sleep(1e-6, loop=loop) yield from asyncio.sleep(1e-8, loop=loop) + yield from asyncio.sleep(1e-10, loop=loop) self.loop.run_until_complete(wait()) - # The ideal number of call is 10, but on some platforms, the selector + # The ideal number of call is 12, but on some platforms, the selector # may sleep at little bit less than timeout depending on the resolution - # of the clock used by the kernel. Tolerate 5 useless calls on these - # platforms. - self.assertLessEqual(self.loop._run_once_counter, 15, - {'time_info': time.get_clock_info('time'), - 'monotonic_info': time.get_clock_info('monotonic'), + # of the clock used by the kernel. Tolerate a few useless calls on + # these platforms. + self.assertLessEqual(self.loop._run_once_counter, 20, + {'clock_resolution': self.loop._clock_resolution, 'selector': self.loop._selector.__class__.__name__}) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:21 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_Fix_TestLoop=2C_set_the_clock_resolution?= Message-ID: <3fnSMY1l4Hz7LkW@mail.python.org> http://hg.python.org/cpython/rev/375e4f9c9732 changeset: 89696:375e4f9c9732 branch: 3.4 user: Victor Stinner date: Tue Feb 11 09:03:47 2014 +0100 summary: Issue #20505: Fix TestLoop, set the clock resolution files: Lib/asyncio/test_utils.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -191,6 +191,7 @@ self._gen = gen() next(self._gen) self._time = 0 + self._clock_resolution = 1e-9 self._timers = [] self._selector = TestSelector() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:22 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_Improve_debug_info_in_asyncio_event_loop?= Message-ID: <3fnSMZ3ZHkz7LkF@mail.python.org> http://hg.python.org/cpython/rev/e44ff3b7a497 changeset: 89697:e44ff3b7a497 branch: 3.4 user: Victor Stinner date: Tue Feb 11 10:08:08 2014 +0100 summary: Issue #20505: Improve debug info in asyncio event loop files: Lib/asyncio/base_events.py | 23 ++++++++++++++++++----- 1 files changed, 18 insertions(+), 5 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -634,12 +634,25 @@ else: logger.log(level, 'poll took %.3f seconds', t1-t0) else: - t0 = self.time() + t0_monotonic = time.monotonic() + t0 = time.perf_counter() event_list = self._selector.select(timeout) - dt = self.time() - t0 - if not event_list and timeout and dt < timeout: - print("asyncio: selector.select(%.3f ms) took %.3f ms" - % (timeout*1e3, dt*1e3), + dt = time.perf_counter() - t0 + dt_monotonic = time.monotonic() - t0_monotonic + if not event_list and timeout: # and dt < timeout: + selector = self._selector.__class__.__name__ + if (selector.startswith(("Poll", "Epoll", "Iocp")) + or timeout > 1e-3 or dt > 1e-3): + unit, factor = "ms", 1e3 + else: + unit, factor = "us", 1e6 + print("asyncio: %s.select(%.3f %s) took %.3f %s" + " (monotonic: %.3f %s, clock res: %.3f %s)" + % (self._selector.__class__.__name__, + timeout * factor, unit, + dt * factor, unit, + dt_monotonic * factor, unit, + self._clock_resolution * factor, unit), file=sys.__stderr__, flush=True) self._process_events(event_list) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:23 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_Oops=2C_only_print_debug_info_if_selector=2Eselect=28timeou?= =?utf-8?q?t=29_took_less?= Message-ID: <3fnSMb5Xpbz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/af840e781700 changeset: 89698:af840e781700 branch: 3.4 user: Victor Stinner date: Tue Feb 11 10:10:41 2014 +0100 summary: Issue #20505: Oops, only print debug info if selector.select(timeout) took less than timeout files: Lib/asyncio/base_events.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -639,7 +639,7 @@ event_list = self._selector.select(timeout) dt = time.perf_counter() - t0 dt_monotonic = time.monotonic() - t0_monotonic - if not event_list and timeout: # and dt < timeout: + if not event_list and timeout and dt < timeout: selector = self._selector.__class__.__name__ if (selector.startswith(("Poll", "Epoll", "Iocp")) or timeout > 1e-3 or dt > 1e-3): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:25 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_use_also_the_monotonic_time_to_decide_if_asyncio_debug_trac?= =?utf-8?q?es?= Message-ID: <3fnSMd0LcNz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/2faf4b7c52ed changeset: 89699:2faf4b7c52ed branch: 3.4 user: Victor Stinner date: Tue Feb 11 10:26:53 2014 +0100 summary: Issue #20505: use also the monotonic time to decide if asyncio debug traces should be printed files: Lib/asyncio/base_events.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -639,15 +639,16 @@ event_list = self._selector.select(timeout) dt = time.perf_counter() - t0 dt_monotonic = time.monotonic() - t0_monotonic - if not event_list and timeout and dt < timeout: + if (not event_list and timeout + and (dt < timeout or dt_monotonic < timeout)): selector = self._selector.__class__.__name__ if (selector.startswith(("Poll", "Epoll", "Iocp")) or timeout > 1e-3 or dt > 1e-3): unit, factor = "ms", 1e3 else: unit, factor = "us", 1e6 - print("asyncio: %s.select(%.3f %s) took %.3f %s" - " (monotonic: %.3f %s, clock res: %.3f %s)" + print("asyncio: %s.select(%.4f %s) took %.3f %s" + " (monotonic=%.3f %s, clock res=%.3f %s)" % (self._selector.__class__.__name__, timeout * factor, unit, dt * factor, unit, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:26 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_126=3A_call=5Fsoon=28=29=2C_call=5Fsoon=5Fthreadsafe?= =?utf-8?b?KCksIGNhbGxfbGF0ZXIoKSw=?= Message-ID: <3fnSMf2xv3z7LkY@mail.python.org> http://hg.python.org/cpython/rev/0055d537b25e changeset: 89700:0055d537b25e branch: 3.4 user: Victor Stinner date: Tue Feb 11 11:34:30 2014 +0100 summary: asyncio, Tulip issue 126: call_soon(), call_soon_threadsafe(), call_later(), call_at() and run_in_executor() now raise a TypeError if the callback is a coroutine function. files: Lib/asyncio/base_events.py | 6 +++ Lib/asyncio/test_utils.py | 5 ++- Lib/test/test_asyncio/test_base_events.py | 18 ++++++++++ Lib/test/test_asyncio/test_proactor_events.py | 2 +- Lib/test/test_asyncio/test_selector_events.py | 9 ++-- Lib/test/test_asyncio/test_tasks.py | 12 ++--- 6 files changed, 39 insertions(+), 13 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -227,6 +227,8 @@ def call_at(self, when, callback, *args): """Like call_later(), but uses an absolute time.""" + if tasks.iscoroutinefunction(callback): + raise TypeError("coroutines cannot be used with call_at()") timer = events.TimerHandle(when, callback, args) heapq.heappush(self._scheduled, timer) return timer @@ -241,6 +243,8 @@ Any positional arguments after the callback will be passed to the callback when it is called. """ + if tasks.iscoroutinefunction(callback): + raise TypeError("coroutines cannot be used with call_soon()") handle = events.Handle(callback, args) self._ready.append(handle) return handle @@ -252,6 +256,8 @@ return handle def run_in_executor(self, executor, callback, *args): + if tasks.iscoroutinefunction(callback): + raise TypeError("coroutines cannot be used with run_in_executor()") if isinstance(callback, events.Handle): assert not args assert not isinstance(callback, events.TimerHandle) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -135,7 +135,7 @@ if name.startswith('__') and name.endswith('__'): # skip magic names continue - dct[name] = unittest.mock.Mock(return_value=None) + dct[name] = MockCallback(return_value=None) return type('TestProtocol', (base,) + base.__bases__, dct)() @@ -274,3 +274,6 @@ def _write_to_self(self): pass + +def MockCallback(**kwargs): + return unittest.mock.Mock(spec=['__call__'], **kwargs) diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -567,6 +567,7 @@ m_socket.getaddrinfo.return_value = [ (2, 1, 6, '', ('127.0.0.1', 10100))] + m_socket.getaddrinfo._is_coroutine = False m_sock = m_socket.socket.return_value = unittest.mock.Mock() m_sock.bind.side_effect = Err @@ -577,6 +578,7 @@ @unittest.mock.patch('asyncio.base_events.socket') def test_create_datagram_endpoint_no_addrinfo(self, m_socket): m_socket.getaddrinfo.return_value = [] + m_socket.getaddrinfo._is_coroutine = False coro = self.loop.create_datagram_endpoint( MyDatagramProto, local_addr=('localhost', 0)) @@ -681,6 +683,22 @@ unittest.mock.ANY, MyProto, sock, None, None) + def test_call_coroutine(self): + @asyncio.coroutine + def coroutine_function(): + pass + + with self.assertRaises(TypeError): + self.loop.call_soon(coroutine_function) + with self.assertRaises(TypeError): + self.loop.call_soon_threadsafe(coroutine_function) + with self.assertRaises(TypeError): + self.loop.call_later(60, coroutine_function) + with self.assertRaises(TypeError): + self.loop.call_at(self.loop.time() + 60, coroutine_function) + with self.assertRaises(TypeError): + self.loop.run_in_executor(None, coroutine_function) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -402,7 +402,7 @@ NotImplementedError, BaseProactorEventLoop, self.proactor) def test_make_socket_transport(self): - tr = self.loop._make_socket_transport(self.sock, unittest.mock.Mock()) + tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) self.assertIsInstance(tr, _ProactorSocketTransport) def test_loop_self_reading(self): diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -44,8 +44,8 @@ def test_make_socket_transport(self): m = unittest.mock.Mock() self.loop.add_reader = unittest.mock.Mock() - self.assertIsInstance( - self.loop._make_socket_transport(m, m), _SelectorSocketTransport) + transport = self.loop._make_socket_transport(m, asyncio.Protocol()) + self.assertIsInstance(transport, _SelectorSocketTransport) @unittest.skipIf(ssl is None, 'No ssl module') def test_make_ssl_transport(self): @@ -54,8 +54,9 @@ self.loop.add_writer = unittest.mock.Mock() self.loop.remove_reader = unittest.mock.Mock() self.loop.remove_writer = unittest.mock.Mock() - self.assertIsInstance( - self.loop._make_ssl_transport(m, m, m, m), _SelectorSslTransport) + waiter = asyncio.Future(loop=self.loop) + transport = self.loop._make_ssl_transport(m, asyncio.Protocol(), m, waiter) + self.assertIsInstance(transport, _SelectorSslTransport) @unittest.mock.patch('asyncio.selector_events.ssl', None) def test_make_ssl_transport_without_ssl_error(self): diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -2,8 +2,6 @@ import gc import unittest -import unittest.mock -from unittest.mock import Mock import asyncio from asyncio import test_utils @@ -1358,7 +1356,7 @@ def _check_success(self, **kwargs): a, b, c = [asyncio.Future(loop=self.one_loop) for i in range(3)] fut = asyncio.gather(*self.wrap_futures(a, b, c), **kwargs) - cb = Mock() + cb = test_utils.MockCallback() fut.add_done_callback(cb) b.set_result(1) a.set_result(2) @@ -1380,7 +1378,7 @@ def test_one_exception(self): a, b, c, d, e = [asyncio.Future(loop=self.one_loop) for i in range(5)] fut = asyncio.gather(*self.wrap_futures(a, b, c, d, e)) - cb = Mock() + cb = test_utils.MockCallback() fut.add_done_callback(cb) exc = ZeroDivisionError() a.set_result(1) @@ -1399,7 +1397,7 @@ a, b, c, d = [asyncio.Future(loop=self.one_loop) for i in range(4)] fut = asyncio.gather(*self.wrap_futures(a, b, c, d), return_exceptions=True) - cb = Mock() + cb = test_utils.MockCallback() fut.add_done_callback(cb) exc = ZeroDivisionError() exc2 = RuntimeError() @@ -1460,7 +1458,7 @@ def test_one_cancellation(self): a, b, c, d, e = [asyncio.Future(loop=self.one_loop) for i in range(5)] fut = asyncio.gather(a, b, c, d, e) - cb = Mock() + cb = test_utils.MockCallback() fut.add_done_callback(cb) a.set_result(1) b.cancel() @@ -1479,7 +1477,7 @@ a, b, c, d, e, f = [asyncio.Future(loop=self.one_loop) for i in range(6)] fut = asyncio.gather(a, b, c, d, e, f, return_exceptions=True) - cb = Mock() + cb = test_utils.MockCallback() fut.add_done_callback(cb) a.set_result(1) zde = ZeroDivisionError() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:27 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_130=3A_Add_more_checks_on_subprocess=5Fexec/subprocess?= =?utf-8?q?=5Fshell?= Message-ID: <3fnSMg5Nqlz7Lkb@mail.python.org> http://hg.python.org/cpython/rev/85db10a4a355 changeset: 89701:85db10a4a355 branch: 3.4 user: Victor Stinner date: Tue Feb 11 11:44:56 2014 +0100 summary: asyncio, Tulip issue 130: Add more checks on subprocess_exec/subprocess_shell parameters files: Lib/asyncio/base_events.py | 12 +- Lib/asyncio/subprocess.py | 5 +- Lib/test/test_asyncio/test_base_events.py | 54 ++++++++++- 3 files changed, 64 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -558,7 +558,7 @@ stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, **kwargs): - if not isinstance(cmd, str): + if not isinstance(cmd, (bytes, str)): raise ValueError("cmd must be a string") if universal_newlines: raise ValueError("universal_newlines must be False") @@ -572,7 +572,7 @@ return transport, protocol @tasks.coroutine - def subprocess_exec(self, protocol_factory, *args, stdin=subprocess.PIPE, + def subprocess_exec(self, protocol_factory, program, *args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, **kwargs): @@ -582,9 +582,15 @@ raise ValueError("shell must be False") if bufsize != 0: raise ValueError("bufsize must be 0") + popen_args = (program,) + args + for arg in popen_args: + if not isinstance(arg, (str, bytes)): + raise TypeError("program arguments must be " + "a bytes or text string, not %s" + % type(arg).__name__) protocol = protocol_factory() transport = yield from self._make_subprocess_transport( - protocol, args, False, stdin, stdout, stderr, bufsize, **kwargs) + protocol, popen_args, False, stdin, stdout, stderr, bufsize, **kwargs) return transport, protocol def _add_callback(self, handle): diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -180,7 +180,7 @@ return Process(transport, protocol, loop) @tasks.coroutine -def create_subprocess_exec(*args, stdin=None, stdout=None, stderr=None, +def create_subprocess_exec(program, *args, stdin=None, stdout=None, stderr=None, loop=None, limit=streams._DEFAULT_LIMIT, **kwds): if loop is None: loop = events.get_event_loop() @@ -188,7 +188,8 @@ loop=loop) transport, protocol = yield from loop.subprocess_exec( protocol_factory, - *args, stdin=stdin, stdout=stdout, + program, *args, + stdin=stdin, stdout=stdout, stderr=stderr, **kwds) yield from protocol.waiter return Process(transport, protocol, loop) diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -3,6 +3,7 @@ import errno import logging import socket +import sys import time import unittest import unittest.mock @@ -234,8 +235,57 @@ self.assertEqual([handle], list(self.loop._ready)) def test_run_until_complete_type_error(self): - self.assertRaises( - TypeError, self.loop.run_until_complete, 'blah') + self.assertRaises(TypeError, + self.loop.run_until_complete, 'blah') + + def test_subprocess_exec_invalid_args(self): + args = [sys.executable, '-c', 'pass'] + + # missing program parameter (empty args) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol) + + # exepected multiple arguments, not a list + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, args) + + # program arguments must be strings, not int + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, sys.executable, 123) + + # universal_newlines, shell, bufsize must not be set + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, universal_newlines=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, shell=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_exec, + asyncio.SubprocessProtocol, *args, bufsize=4096) + + def test_subprocess_shell_invalid_args(self): + # exepected a string, not an int or a list + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 123) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, [sys.executable, '-c', 'pass']) + + # universal_newlines, shell, bufsize must not be set + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', universal_newlines=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', shell=True) + self.assertRaises(TypeError, + self.loop.run_until_complete, self.loop.subprocess_shell, + asyncio.SubprocessProtocol, 'exit 0', bufsize=4096) class MyProto(asyncio.Protocol): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:29 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_131=3A_as=5Fcompleted=28=29_and_wait=28=29_now_raises_?= =?utf-8?q?a_TypeError_if?= Message-ID: <3fnSMj6yClz7LkL@mail.python.org> http://hg.python.org/cpython/rev/5f7a601f2245 changeset: 89702:5f7a601f2245 branch: 3.4 user: Victor Stinner date: Tue Feb 11 11:54:08 2014 +0100 summary: asyncio, Tulip issue 131: as_completed() and wait() now raises a TypeError if the list of futures is not a list but a Future, Task or coroutine object files: Lib/asyncio/tasks.py | 4 ++ Lib/test/test_asyncio/test_tasks.py | 26 +++++++++++++++++ 2 files changed, 30 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -358,6 +358,8 @@ Note: This does not raise TimeoutError! Futures that aren't done when the timeout occurs are returned in the second set. """ + if isinstance(fs, futures.Future) or iscoroutine(fs): + raise TypeError("expect a list of futures, not %s" % type(fs).__name__) if not fs: raise ValueError('Set of coroutines/Futures is empty.') @@ -474,6 +476,8 @@ Note: The futures 'f' are not necessarily members of fs. """ + if isinstance(fs, futures.Future) or iscoroutine(fs): + raise TypeError("expect a list of futures, not %s" % type(fs).__name__) loop = loop if loop is not None else events.get_event_loop() deadline = None if timeout is None else loop.time() + timeout todo = {async(f, loop=loop) for f in set(fs)} diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -7,6 +7,11 @@ from asyncio import test_utils + at asyncio.coroutine +def coroutine_function(): + pass + + class Dummy: def __repr__(self): @@ -1338,6 +1343,27 @@ child2.set_result(2) test_utils.run_briefly(self.loop) + def test_as_completed_invalid_args(self): + fut = asyncio.Future(loop=self.loop) + + # as_completed() expects a list of futures, not a future instance + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.as_completed(fut, loop=self.loop)) + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.as_completed(coroutine_function(), loop=self.loop)) + + def test_wait_invalid_args(self): + fut = asyncio.Future(loop=self.loop) + + # wait() expects a list of futures, not a future instance + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.wait(fut, loop=self.loop)) + self.assertRaises(TypeError, self.loop.run_until_complete, + asyncio.wait(coroutine_function(), loop=self.loop)) + + # wait() expects at least a future + self.assertRaises(ValueError, self.loop.run_until_complete, + asyncio.wait([], loop=self.loop)) class GatherTestsBase: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:31 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogbWVyZ2UgMy4zICgj?= =?utf-8?q?20594=29?= Message-ID: <3fnSMl1PSdz7LkD@mail.python.org> http://hg.python.org/cpython/rev/fd49c1d2fd6c changeset: 89703:fd49c1d2fd6c branch: 3.4 user: Benjamin Peterson date: Tue Feb 11 10:19:12 2014 -0500 summary: merge 3.3 (#20594) files: Misc/NEWS | 2 ++ Modules/posixmodule.c | 10 +++++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,8 @@ Library ------- +- Issue #20594: Avoid name clash with the libc function posix_close. + What's New in Python 3.4.0 release candidate 1? =============================================== diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -7763,8 +7763,12 @@ "close(fd)\n\n\ Close a file descriptor (for low level IO)."); -static PyObject * -posix_close(PyObject *self, PyObject *args) +/* +The underscore at end of function name avoids a name clash with the libc +function posix_close. +*/ +static PyObject * +posix_close_(PyObject *self, PyObject *args) { int fd, res; if (!PyArg_ParseTuple(args, "i:close", &fd)) @@ -11422,7 +11426,7 @@ {"open", (PyCFunction)posix_open,\ METH_VARARGS | METH_KEYWORDS, posix_open__doc__}, - {"close", posix_close, METH_VARARGS, posix_close__doc__}, + {"close", posix_close_, METH_VARARGS, posix_close__doc__}, {"closerange", posix_closerange, METH_VARARGS, posix_closerange__doc__}, {"device_encoding", device_encoding, METH_VARARGS, device_encoding__doc__}, {"dup", posix_dup, METH_VARARGS, posix_dup__doc__}, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:32 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTA1?= =?utf-8?q?=3A_Remove_debug_code?= Message-ID: <3fnSMm3FYbz7LkM@mail.python.org> http://hg.python.org/cpython/rev/702b20fa7af2 changeset: 89704:702b20fa7af2 branch: 3.4 user: Victor Stinner date: Tue Feb 11 17:53:47 2014 +0100 summary: Issue #20505: Remove debug code files: Lib/asyncio/base_events.py | 20 -------------------- 1 files changed, 0 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -646,27 +646,7 @@ else: logger.log(level, 'poll took %.3f seconds', t1-t0) else: - t0_monotonic = time.monotonic() - t0 = time.perf_counter() event_list = self._selector.select(timeout) - dt = time.perf_counter() - t0 - dt_monotonic = time.monotonic() - t0_monotonic - if (not event_list and timeout - and (dt < timeout or dt_monotonic < timeout)): - selector = self._selector.__class__.__name__ - if (selector.startswith(("Poll", "Epoll", "Iocp")) - or timeout > 1e-3 or dt > 1e-3): - unit, factor = "ms", 1e3 - else: - unit, factor = "us", 1e6 - print("asyncio: %s.select(%.4f %s) took %.3f %s" - " (monotonic=%.3f %s, clock res=%.3f %s)" - % (self._selector.__class__.__name__, - timeout * factor, unit, - dt * factor, unit, - dt_monotonic * factor, unit, - self._clock_resolution * factor, unit), - file=sys.__stderr__, flush=True) self._process_events(event_list) # Handle 'later' callbacks that are ready. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:33 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDk1?= =?utf-8?q?=3A_Skip_test=5Fread=5Fpty=5Foutput=28=29_of_test=5Fasyncio_on_?= =?utf-8?q?FreeBSD_older_than?= Message-ID: <3fnSMn4zn1z7Ljh@mail.python.org> http://hg.python.org/cpython/rev/2cf25865fc66 changeset: 89705:2cf25865fc66 branch: 3.4 user: Victor Stinner date: Tue Feb 11 18:40:56 2014 +0100 summary: Issue #20495: Skip test_read_pty_output() of test_asyncio on FreeBSD older than FreeBSD 8 files: Lib/test/test_asyncio/test_events.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -956,6 +956,8 @@ # select, poll and kqueue don't support character devices (PTY) on Mac OS X # older than 10.6 (Snow Leopard) @support.requires_mac_ver(10, 6) + # Issue #20495: The test hangs on FreeBSD 7.2 but pass on FreeBSD 9 + @support.requires_freebsd_version(8) def test_read_pty_output(self): proto = None -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:34 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5ldmVu?= =?utf-8?q?ts=3A_Use_=5F=5Fslots=5F=5F_in_Handle_and_TimerHandle?= Message-ID: <3fnSMp6Rbmz7LjY@mail.python.org> http://hg.python.org/cpython/rev/dcaa7a26eed3 changeset: 89706:dcaa7a26eed3 branch: 3.4 user: Yury Selivanov date: Wed Feb 12 17:01:52 2014 -0500 summary: asyncio.events: Use __slots__ in Handle and TimerHandle files: Lib/asyncio/events.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -19,6 +19,8 @@ class Handle: """Object returned by callback registration methods.""" + __slots__ = ['_callback', '_args', '_cancelled'] + def __init__(self, callback, args): assert not isinstance(callback, Handle), 'A Handle is not a callback' self._callback = callback @@ -46,6 +48,8 @@ class TimerHandle(Handle): """Object returned by timed callback registration methods.""" + __slots__ = ['_when'] + def __init__(self, when, callback, args): assert when is not None super().__init__(callback, args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:36 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQ2hh?= =?utf-8?q?nge_as=5Fcompleted=28=29_to_use_a_Queue=2C_to_avoid_O=28N**2=29?= =?utf-8?q?_behavior=2E_Fixes?= Message-ID: <3fnSMr1rNmz7Lkg@mail.python.org> http://hg.python.org/cpython/rev/b52113fb58a5 changeset: 89707:b52113fb58a5 branch: 3.4 user: Guido van Rossum date: Wed Feb 12 17:58:19 2014 -0800 summary: asyncio: Change as_completed() to use a Queue, to avoid O(N**2) behavior. Fixes issue #20566. files: Lib/asyncio/tasks.py | 53 ++++++++++------ Lib/test/test_asyncio/test_tasks.py | 23 +++++++- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -463,7 +463,11 @@ # This is *not* a @coroutine! It is just an iterator (yielding Futures). def as_completed(fs, *, loop=None, timeout=None): - """Return an iterator whose values, when waited for, are Futures. + """Return an iterator whose values are coroutines. + + When waiting for the yielded coroutines you'll get the results (or + exceptions!) of the original Futures (or coroutines), in the order + in which and as soon as they complete. This differs from PEP 3148; the proper way to use this is: @@ -471,8 +475,8 @@ result = yield from f # The 'yield from' may raise. # Use result. - Raises TimeoutError if the timeout occurs before all Futures are - done. + If a timeout is specified, the 'yield from' will raise + TimeoutError when the timeout occurs before all Futures are done. Note: The futures 'f' are not necessarily members of fs. """ @@ -481,27 +485,36 @@ loop = loop if loop is not None else events.get_event_loop() deadline = None if timeout is None else loop.time() + timeout todo = {async(f, loop=loop) for f in set(fs)} - completed = collections.deque() + from .queues import Queue # Import here to avoid circular import problem. + done = Queue(loop=loop) + timeout_handle = None + + def _on_timeout(): + for f in todo: + f.remove_done_callback(_on_completion) + done.put_nowait(None) # Queue a dummy value for _wait_for_one(). + todo.clear() # Can't do todo.remove(f) in the loop. + + def _on_completion(f): + if not todo: + return # _on_timeout() was here first. + todo.remove(f) + done.put_nowait(f) + if not todo and timeout_handle is not None: + timeout_handle.cancel() @coroutine def _wait_for_one(): - while not completed: - timeout = None - if deadline is not None: - timeout = deadline - loop.time() - if timeout < 0: - raise futures.TimeoutError() - done, pending = yield from _wait( - todo, timeout, FIRST_COMPLETED, loop) - # Multiple callers might be waiting for the same events - # and getting the same outcome. Dedupe by updating todo. - for f in done: - if f in todo: - todo.remove(f) - completed.append(f) - f = completed.popleft() - return f.result() # May raise. + f = yield from done.get() + if f is None: + # Dummy value from _on_timeout(). + raise futures.TimeoutError + return f.result() # May raise f.exception(). + for f in todo: + f.add_done_callback(_on_completion) + if todo and timeout is not None: + timeout_handle = loop.call_later(timeout, _on_timeout) for _ in range(len(todo)): yield _wait_for_one() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -779,7 +779,6 @@ yield 0 yield 0 yield 0.1 - yield 0.02 loop = test_utils.TestLoop(gen) self.addCleanup(loop.close) @@ -791,6 +790,8 @@ def foo(): values = [] for f in asyncio.as_completed([a, b], timeout=0.12, loop=loop): + if values: + loop.advance_time(0.02) try: v = yield from f values.append((1, v)) @@ -809,6 +810,26 @@ loop.advance_time(10) loop.run_until_complete(asyncio.wait([a, b], loop=loop)) + def test_as_completed_with_unused_timeout(self): + + def gen(): + yield + yield 0 + yield 0.01 + + loop = test_utils.TestLoop(gen) + self.addCleanup(loop.close) + + a = asyncio.sleep(0.01, 'a', loop=loop) + + @asyncio.coroutine + def foo(): + for f in asyncio.as_completed([a], timeout=1, loop=loop): + v = yield from f + self.assertEqual(v, 'a') + + res = loop.run_until_complete(asyncio.Task(foo(), loop=loop)) + def test_as_completed_reverse_wait(self): def gen(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:37 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXluY2lvLCBUdWxp?= =?utf-8?q?p_issue_129=3A_BaseEventLoop=2Esock=5Fconnect=28=29_now_raises_?= =?utf-8?q?an_error_if?= Message-ID: <3fnSMs4mRQz7LkL@mail.python.org> http://hg.python.org/cpython/rev/b4c75a9f1541 changeset: 89708:b4c75a9f1541 branch: 3.4 user: Victor Stinner date: Thu Feb 13 09:24:37 2014 +0100 summary: ayncio, Tulip issue 129: BaseEventLoop.sock_connect() now raises an error if the address is not resolved (hostname instead of an IP address) for AF_INET and AF_INET6 address families. files: Doc/library/asyncio-eventloop.rst | 6 +++ Lib/asyncio/base_events.py | 25 ++++++++++++++++ Lib/asyncio/proactor_events.py | 9 +++++- Lib/asyncio/selector_events.py | 20 +++++------- Lib/test/test_asyncio/test_events.py | 12 +++++++ 5 files changed, 59 insertions(+), 13 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -366,6 +366,12 @@ Connect to a remote socket at *address*. + The *address* must be already resolved to avoid the trap of hanging the + entire event loop when the address requires doing a DNS lookup. For + example, it must be an IP address, not an hostname, for + :py:data:`~socket.AF_INET` and :py:data:`~socket.AF_INET6` address families. + Use :meth:`getaddrinfo` to resolve the hostname asynchronously. + This method returns a :ref:`coroutine object `. .. seealso:: diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -41,6 +41,31 @@ """Raised to stop the event loop.""" +def _check_resolved_address(sock, address): + # Ensure that the address is already resolved to avoid the trap of hanging + # the entire event loop when the address requires doing a DNS lookup. + family = sock.family + if family not in (socket.AF_INET, socket.AF_INET6): + return + + host, port = address + type_mask = 0 + if hasattr(socket, 'SOCK_NONBLOCK'): + type_mask |= socket.SOCK_NONBLOCK + if hasattr(socket, 'SOCK_CLOEXEC'): + type_mask |= socket.SOCK_CLOEXEC + # Use getaddrinfo(AI_NUMERICHOST) to ensure that the address is + # already resolved. + try: + socket.getaddrinfo(host, port, + family=family, + type=(sock.type & ~type_mask), + proto=sock.proto, + flags=socket.AI_NUMERICHOST) + except socket.gaierror as err: + raise ValueError("address must be resolved (IP address), got %r: %s" + % (address, err)) + def _raise_stop_error(*args): raise _StopError diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -404,7 +404,14 @@ return self._proactor.send(sock, data) def sock_connect(self, sock, address): - return self._proactor.connect(sock, address) + try: + base_events._check_resolved_address(sock, address) + except ValueError as err: + fut = futures.Future(loop=self) + fut.set_exception(err) + return fut + else: + return self._proactor.connect(sock, address) def sock_accept(self, sock): return self._proactor.accept(sock) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -208,6 +208,8 @@ return fut def _sock_recv(self, fut, registered, sock, n): + # _sock_recv() can add itself as an I/O callback if the operation can't + # be done immediatly. Don't use it directly, call sock_recv(). fd = sock.fileno() if registered: # Remove the callback early. It should be rare that the @@ -260,22 +262,16 @@ def sock_connect(self, sock, address): """XXX""" - # That address better not require a lookup! We're not calling - # self.getaddrinfo() for you here. But verifying this is - # complicated; the socket module doesn't have a pattern for - # IPv6 addresses (there are too many forms, apparently). fut = futures.Future(loop=self) - self._sock_connect(fut, False, sock, address) + try: + base_events._check_resolved_address(sock, address) + except ValueError as err: + fut.set_exception(err) + else: + self._sock_connect(fut, False, sock, address) return fut def _sock_connect(self, fut, registered, sock, address): - # TODO: Use getaddrinfo() to look up the address, to avoid the - # trap of hanging the entire event loop when the address - # requires doing a DNS lookup. (OTOH, the caller should - # already have done this, so it would be nice if we could - # easily tell whether the address needs looking up or not. I - # know how to do this for IPv4, but IPv6 addresses have many - # syntaxes.) fd = sock.fileno() if registered: self.remove_writer(fd) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1191,6 +1191,18 @@ {'clock_resolution': self.loop._clock_resolution, 'selector': self.loop._selector.__class__.__name__}) + def test_sock_connect_address(self): + address = ('www.python.org', 80) + for family in (socket.AF_INET, socket.AF_INET6): + for sock_type in (socket.SOCK_STREAM, socket.SOCK_DGRAM): + sock = socket.socket(family, sock_type) + with sock: + connect = self.loop.sock_connect(sock, address) + with self.assertRaises(ValueError) as cm: + self.loop.run_until_complete(connect) + self.assertIn('address must be resolved', + str(cm.exception)) + class SubprocessTestsMixin: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:38 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogRml4IHRlc3RfYXN5?= =?utf-8?q?ncio/test=5Fevents=2Epy=3A_skip_IPv6_if_IPv6_is_disabled_on_the?= =?utf-8?q?_host?= Message-ID: <3fnSMt6Wtmz7LjV@mail.python.org> http://hg.python.org/cpython/rev/778c5dd644d1 changeset: 89709:778c5dd644d1 branch: 3.4 user: Victor Stinner date: Thu Feb 13 10:46:05 2014 +0100 summary: Fix test_asyncio/test_events.py: skip IPv6 if IPv6 is disabled on the host files: Lib/test/test_asyncio/test_events.py | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1192,8 +1192,12 @@ 'selector': self.loop._selector.__class__.__name__}) def test_sock_connect_address(self): + families = [socket.AF_INET] + if support.IPV6_ENABLED: + families.append(socket.AF_INET6) + address = ('www.python.org', 80) - for family in (socket.AF_INET, socket.AF_INET6): + for family in families: for sock_type in (socket.SOCK_STREAM, socket.SOCK_DGRAM): sock = socket.socket(family, sock_type) with sock: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:40 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTI2?= =?utf-8?q?=2C_=2319466=3A_Revert_changes_of_issue_=2319466_which_introduc?= =?utf-8?q?es_a?= Message-ID: <3fnSMw2QtZz7LkM@mail.python.org> http://hg.python.org/cpython/rev/9ce58a73b6b5 changeset: 89710:9ce58a73b6b5 branch: 3.4 user: Victor Stinner date: Thu Feb 13 12:48:54 2014 +0100 summary: Issue #20526, #19466: Revert changes of issue #19466 which introduces a regression: don't clear anymore the state of Python threads early during the Python shutdown. files: Lib/test/test_threading.py | 49 -------------------------- Misc/NEWS | 4 ++ Python/pythonrun.c | 20 ++------- 3 files changed, 9 insertions(+), 64 deletions(-) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -616,51 +616,6 @@ t.join() self.assertRaises(ValueError, bs.release) - def test_locals_at_exit(self): - # Issue #19466: thread locals must not be deleted before destructors - # are called - rc, out, err = assert_python_ok("-c", """if 1: - import threading - - class Atexit: - def __del__(self): - print("thread_dict.atexit = %r" % thread_dict.atexit) - - thread_dict = threading.local() - thread_dict.atexit = "atexit" - - atexit = Atexit() - """) - self.assertEqual(out.rstrip(), b"thread_dict.atexit = 'atexit'") - - def test_warnings_at_exit(self): - # Issue #19466: try to call most destructors at Python shutdown before - # destroying Python thread states - filename = __file__ - rc, out, err = assert_python_ok("-Wd", "-c", """if 1: - import time - import threading - - def open_sleep(): - # a warning will be emitted when the open file will be - # destroyed (without being explicitly closed) while the daemon - # thread is destroyed - fileobj = open(%a, 'rb') - start_event.set() - time.sleep(60.0) - - start_event = threading.Event() - - thread = threading.Thread(target=open_sleep) - thread.daemon = True - thread.start() - - # wait until the thread started - start_event.wait() - """ % filename) - self.assertRegex(err.rstrip(), - b"^sys:1: ResourceWarning: unclosed file ") - @cpython_only def test_frame_tstate_tracing(self): # Issue #14432: Crash when a generator is created in a C thread that is @@ -786,10 +741,6 @@ import sys import time import threading - import warnings - - # ignore "unclosed file ..." warnings - warnings.filterwarnings('ignore', '', ResourceWarning) thread_has_run = set() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #20526: Revert changes of issue #19466 which introduces a regression: + don't clear anymore the state of Python threads early during the Python + shutdown. + Library ------- diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -582,13 +582,11 @@ _Py_Finalizing = tstate; initialized = 0; - /* Destroy the state of all threads except of the current thread: in - practice, only daemon threads should still be alive. Clear frames of - other threads to call objects destructor. Destructors will be called in - the current Python thread. Since _Py_Finalizing has been set, no other - Python threads can lock the GIL at this point (if they try, they will - exit immediately). */ - _PyThreadState_DeleteExcept(tstate); + /* Flush stdout+stderr */ + flush_std_files(); + + /* Disable signal handling */ + PyOS_FiniInterrupts(); /* Collect garbage. This may call finalizers; it's nice to call these * before all modules are destroyed. @@ -603,7 +601,6 @@ * XXX I haven't seen a real-life report of either of these. */ PyGC_Collect(); - #ifdef COUNT_ALLOCS /* With COUNT_ALLOCS, it helps to run GC multiple times: each collection might release some types from the type @@ -611,13 +608,6 @@ while (PyGC_Collect() > 0) /* nothing */; #endif - - /* Flush stdout+stderr */ - flush_std_files(); - - /* Disable signal handling */ - PyOS_FiniInterrupts(); - /* Destroy all modules */ PyImport_Cleanup(); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:41 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_set_line_and_c?= =?utf-8?q?olumn_numbers_for_keyword-only_arg_nodes_=28closes_=2320619=29?= Message-ID: <3fnSMx4Njvz7Lkb@mail.python.org> http://hg.python.org/cpython/rev/3c01209ab697 changeset: 89711:3c01209ab697 branch: 3.4 user: Benjamin Peterson date: Thu Feb 13 19:22:14 2014 -0500 summary: set line and column numbers for keyword-only arg nodes (closes #20619) files: Lib/test/test_ast.py | 4 ++-- Misc/NEWS | 3 +++ Python/ast.c | 2 ++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -37,7 +37,7 @@ # FunctionDef with kwargs "def f(**kwargs): pass", # FunctionDef with all kind of args - "def f(a, b=1, c=None, d=[], e={}, *args, **kwargs): pass", + "def f(a, b=1, c=None, d=[], e={}, *args, f=42, **kwargs): pass", # ClassDef "class C:pass", # ClassDef, new style class @@ -973,7 +973,7 @@ ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None)], None, [], [], None, [('Num', (1, 8), 0)]), [('Pass', (1, 12))], [], None)]), ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], ('arg', (1, 7), 'args', None), [], [], None, []), [('Pass', (1, 14))], [], None)]), ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], ('arg', (1, 8), 'kwargs', None), []), [('Pass', (1, 17))], [], None)]), -('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [], [], ('arg', (1, 43), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 52))], [], None)]), + ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [('arg', (1, 41), 'f', None)], [('Num', (1, 43), 42)], ('arg', (1, 49), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 58))], [], None)]), ('Module', [('ClassDef', (1, 0), 'C', [], [], None, None, [('Pass', (1, 8))], [])]), ('Module', [('ClassDef', (1, 0), 'C', [('Name', (1, 8), 'object', ('Load',))], [], None, None, [('Pass', (1, 17))], [])]), ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Return', (1, 8), ('Num', (1, 15), 1))], [], None)]), diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #20619: Give the AST nodes of keyword-only arguments a column and line + number. + - Issue #20526: Revert changes of issue #19466 which introduces a regression: don't clear anymore the state of Python threads early during the Python shutdown. diff --git a/Python/ast.c b/Python/ast.c --- a/Python/ast.c +++ b/Python/ast.c @@ -1203,6 +1203,8 @@ arg = arg(argname, annotation, c->c_arena); if (!arg) goto error; + arg->lineno = LINENO(ch); + arg->col_offset = ch->n_col_offset; asdl_seq_SET(kwonlyargs, j++, arg); i += 2; /* the name and the comma */ break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:42 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5NzQ0?= =?utf-8?q?=3A_Handle_missing_SSL/TLS_in_ensurepip?= Message-ID: <3fnSMy62Mjz7LjX@mail.python.org> http://hg.python.org/cpython/rev/cd39d4cab680 changeset: 89712:cd39d4cab680 branch: 3.4 user: Nick Coghlan date: Sat Feb 15 09:14:54 2014 +1000 summary: Issue #19744: Handle missing SSL/TLS in ensurepip - now also allows POSIX installation with SSL/TLS missing - a goal for pip 1.6 is to allow local use without SSL/TLS files: Lib/ensurepip/__init__.py | 5 +++++ Lib/test/test_ensurepip.py | 8 ++++++++ Misc/NEWS | 4 ++++ 3 files changed, 17 insertions(+), 0 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -144,6 +144,11 @@ def _main(argv=None): + if ssl is None: + print("Ignoring ensurepip failure: {}".format(_MISSING_SSL_MESSAGE), + file=sys.stderr) + return + import argparse parser = argparse.ArgumentParser(prog="python -m ensurepip") parser.add_argument( diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -281,12 +281,20 @@ self.run_pip.assert_not_called() self.assertIn("PIP_THIS_SHOULD_STAY", self.os_environ) + def test_main_exits_early_with_warning(self): + with test.support.captured_stderr() as stderr: + ensurepip_no_ssl._main(["--version"]) + warning = stderr.getvalue().strip() + self.assertTrue(warning.endswith("requires SSL/TLS"), warning) + self.run_pip.assert_not_called() + # Basic testing of the main functions and their argument parsing EXPECTED_VERSION_OUTPUT = "pip " + ensurepip._PIP_VERSION class TestBootstrappingMainFunction(EnsurepipMixin, unittest.TestCase): + @requires_usable_pip def test_bootstrap_version(self): with test.support.captured_stdout() as stdout: with self.assertRaises(SystemExit): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,10 @@ Library ------- +- Issue #19744: the ensurepip installation step now just prints a warning to + stderr rather than failing outright if SSL/TLS is unavailable. This allows + local installation of POSIX builds without SSL/TLS support. + - Issue #20594: Avoid name clash with the libc function posix_close. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:44 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogbG9vayB1cCBfX2dl?= =?utf-8?q?tnewargs=5F=5F_and_=5F=5Fgetnewargs=5Fex=5F=5F_on_the_object_ty?= =?utf-8?b?cGUgKCMxNjI1MSk=?= Message-ID: <3fnSN00pMMz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/2514a577c7cb changeset: 89713:2514a577c7cb branch: 3.4 user: Benjamin Peterson date: Sun Feb 16 13:49:16 2014 -0500 summary: look up __getnewargs__ and __getnewargs_ex__ on the object type (#16251) files: Lib/test/test_descr.py | 14 ++++++++++++++ Misc/NEWS | 3 +++ Objects/typeobject.c | 18 ++++++------------ 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4701,6 +4701,20 @@ for proto in protocols: self._check_reduce(proto, obj, listitems=list(obj)) + def test_special_method_lookup(self): + protocols = range(pickle.HIGHEST_PROTOCOL + 1) + class Picky: + def __getstate__(self): + return {} + + def __getattr__(self, attr): + if attr in ("__getnewargs__", "__getnewargs_ex__"): + raise AssertionError(attr) + return None + for protocol in protocols: + state = {} if protocol >= 2 else None + self._check_reduce(protocol, Picky(), state=state) + def _assert_is_copy(self, obj, objcopy, msg=None): """Utility method to verify if two objects are copies of each others. """ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #20261: In pickle, lookup __getnewargs__ and __getnewargs_ex__ on the + type of the object. + - Issue #20619: Give the AST nodes of keyword-only arguments a column and line number. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3719,7 +3719,7 @@ /* We first attempt to fetch the arguments for __new__ by calling __getnewargs_ex__ on the object. */ - getnewargs_ex = _PyObject_GetAttrId(obj, &PyId___getnewargs_ex__); + getnewargs_ex = _PyObject_LookupSpecial(obj, &PyId___getnewargs_ex__); if (getnewargs_ex != NULL) { PyObject *newargs = PyObject_CallObject(getnewargs_ex, NULL); Py_DECREF(getnewargs_ex); @@ -3766,16 +3766,13 @@ return -1; } return 0; - } else { - if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { - return -1; - } - PyErr_Clear(); + } else if (PyErr_Occurred()) { + return -1; } /* The object does not have __getnewargs_ex__ so we fallback on using __getnewargs__ instead. */ - getnewargs = _PyObject_GetAttrId(obj, &PyId___getnewargs__); + getnewargs = _PyObject_LookupSpecial(obj, &PyId___getnewargs__); if (getnewargs != NULL) { *args = PyObject_CallObject(getnewargs, NULL); Py_DECREF(getnewargs); @@ -3791,11 +3788,8 @@ } *kwargs = NULL; return 0; - } else { - if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { - return -1; - } - PyErr_Clear(); + } else if (PyErr_Occurred()) { + return -1; } /* The object does not have __getnewargs_ex__ and __getnewargs__. This may -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:45 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_merge_backout_?= =?utf-8?q?for_=2320621?= Message-ID: <3fnSN14TVsz7LkB@mail.python.org> http://hg.python.org/cpython/rev/27b738209a60 changeset: 89714:27b738209a60 branch: 3.4 user: Benjamin Peterson date: Sun Feb 16 14:17:28 2014 -0500 summary: merge backout for #20621 files: Lib/test/test_zipimport.py | 104 +------- Modules/zipimport.c | 286 ++++-------------------- 2 files changed, 68 insertions(+), 322 deletions(-) diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -71,27 +71,6 @@ support.modules_cleanup(*self.modules_before) -def _write_zip_package(zipname, files, - data_to_prepend=b"", compression=ZIP_STORED): - z = ZipFile(zipname, "w") - try: - for name, (mtime, data) in files.items(): - zinfo = ZipInfo(name, time.localtime(mtime)) - zinfo.compress_type = compression - z.writestr(zinfo, data) - finally: - z.close() - - if data_to_prepend: - # Prepend data to the start of the zipfile - with open(zipname, "rb") as f: - zip_data = f.read() - - with open(zipname, "wb") as f: - f.write(data_to_prepend) - f.write(zip_data) - - class UncompressedZipImportTestCase(ImportHooksBaseTestCase): compression = ZIP_STORED @@ -104,9 +83,23 @@ ImportHooksBaseTestCase.setUp(self) def doTest(self, expected_ext, files, *modules, **kw): - _write_zip_package(TEMP_ZIP, files, data_to_prepend=kw.get("stuff"), - compression=self.compression) + z = ZipFile(TEMP_ZIP, "w") try: + for name, (mtime, data) in files.items(): + zinfo = ZipInfo(name, time.localtime(mtime)) + zinfo.compress_type = self.compression + z.writestr(zinfo, data) + z.close() + + stuff = kw.get("stuff", None) + if stuff is not None: + # Prepend 'stuff' to the start of the zipfile + with open(TEMP_ZIP, "rb") as f: + data = f.read() + with open(TEMP_ZIP, "wb") as f: + f.write(stuff) + f.write(data) + sys.path.insert(0, TEMP_ZIP) mod = __import__(".".join(modules), globals(), locals(), @@ -121,8 +114,7 @@ self.assertEqual(file, os.path.join(TEMP_ZIP, *modules) + expected_ext) finally: - while TEMP_ZIP in sys.path: - sys.path.remove(TEMP_ZIP) + z.close() os.remove(TEMP_ZIP) def testAFakeZlib(self): @@ -430,67 +422,10 @@ compression = ZIP_DEFLATED -class ZipFileModifiedAfterImportTestCase(ImportHooksBaseTestCase): - def setUp(self): - zipimport._zip_directory_cache.clear() - zipimport._zip_stat_cache.clear() - ImportHooksBaseTestCase.setUp(self) - - def tearDown(self): - ImportHooksBaseTestCase.tearDown(self) - if os.path.exists(TEMP_ZIP): - os.remove(TEMP_ZIP) - - def testZipFileChangesAfterFirstImport(self): - """Alter the zip file after caching its index and try an import.""" - packdir = TESTPACK + os.sep - files = {packdir + "__init__" + pyc_ext: (NOW, test_pyc), - packdir + TESTMOD + ".py": (NOW, "test_value = 38\n"), - "ziptest_a.py": (NOW, "test_value = 23\n"), - "ziptest_b.py": (NOW, "test_value = 42\n"), - "ziptest_c.py": (NOW, "test_value = 1337\n")} - zipfile_path = TEMP_ZIP - _write_zip_package(zipfile_path, files) - self.assertTrue(os.path.exists(zipfile_path)) - sys.path.insert(0, zipfile_path) - - # Import something out of the zipfile and confirm it is correct. - testmod = __import__(TESTPACK + "." + TESTMOD, - globals(), locals(), ["__dummy__"]) - self.assertEqual(testmod.test_value, 38) - # Import something else out of the zipfile and confirm it is correct. - ziptest_b = __import__("ziptest_b", globals(), locals(), ["test_value"]) - self.assertEqual(ziptest_b.test_value, 42) - - # Truncate and fill the zip file with non-zip garbage. - with open(zipfile_path, "rb") as orig_zip_file: - orig_zip_file_contents = orig_zip_file.read() - with open(zipfile_path, "wb") as byebye_valid_zip_file: - byebye_valid_zip_file.write(b"Tear down this wall!\n"*1987) - # Now that the zipfile has been replaced, import something else from it - # which should fail as the file contents are now garbage. - with self.assertRaises(ImportError): - ziptest_a = __import__("ziptest_a", globals(), locals(), - ["test_value"]) - self.assertEqual(ziptest_a.test_value, 23) - - # Now lets make it a valid zipfile that has some garbage at the start. - # This alters all of the offsets within the file - with open(zipfile_path, "wb") as new_zip_file: - new_zip_file.write(b"X"*1991) # The year Python was created. - new_zip_file.write(orig_zip_file_contents) - - # Now that the zip file has been "restored" to a valid but different - # zipfile the zipimporter should *successfully* re-read the new zip - # file's end of file central index and be able to import from it again. - ziptest_c = __import__("ziptest_c", globals(), locals(), ["test_value"]) - self.assertEqual(ziptest_c.test_value, 1337) - - class BadFileZipImportTestCase(unittest.TestCase): def assertZipFailure(self, filename): - with self.assertRaises(zipimport.ZipImportError): - zipimport.zipimporter(filename) + self.assertRaises(zipimport.ZipImportError, + zipimport.zipimporter, filename) def testNoFile(self): self.assertZipFailure('AdfjdkFJKDFJjdklfjs') @@ -564,7 +499,6 @@ UncompressedZipImportTestCase, CompressedZipImportTestCase, BadFileZipImportTestCase, - ZipFileModifiedAfterImportTestCase, ) finally: support.unlink(TESTMOD) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -49,16 +49,10 @@ static PyObject *ZipImportError; /* read_directory() cache */ static PyObject *zip_directory_cache = NULL; -static PyObject *zip_stat_cache = NULL; -/* posix.fstat or nt.fstat function. Used due to posixmodule.c's - * superior fstat implementation over libc's on Windows. */ -static PyObject *fstat_function = NULL; /* posix.fstat() or nt.fstat() */ /* forward decls */ -static FILE *fopen_rb_and_stat(PyObject *path, PyObject **py_stat_p); -static FILE *safely_reopen_archive(ZipImporter *self); -static PyObject *read_directory(FILE *fp, PyObject *archive); -static PyObject *get_data(FILE *fp, PyObject *archive, PyObject *toc_entry); +static PyObject *read_directory(PyObject *archive); +static PyObject *get_data(PyObject *archive, PyObject *toc_entry); static PyObject *get_module_code(ZipImporter *self, PyObject *fullname, int *p_ispackage, PyObject **p_modpath); @@ -137,39 +131,11 @@ files = PyDict_GetItem(zip_directory_cache, filename); if (files == NULL) { - PyObject *zip_stat = NULL; - FILE *fp = fopen_rb_and_stat(filename, &zip_stat); - if (fp == NULL) { - if (!PyErr_Occurred()) - PyErr_Format(ZipImportError, "can't open Zip file: %R", - filename); - - Py_XDECREF(zip_stat); + files = read_directory(filename); + if (files == NULL) goto error; - } - - if (Py_VerboseFlag) - PySys_FormatStderr("# zipimport: %U not cached, " - "reading TOC.\n", filename); - - files = read_directory(fp, filename); - fclose(fp); - if (files == NULL) { - Py_XDECREF(zip_stat); + if (PyDict_SetItem(zip_directory_cache, filename, files) != 0) goto error; - } - if (PyDict_SetItem(zip_directory_cache, filename, files) != 0) { - Py_DECREF(files); - Py_XDECREF(zip_stat); - goto error; - } - if (zip_stat && PyDict_SetItem(zip_stat_cache, filename, - zip_stat) != 0) { - Py_DECREF(files); - Py_DECREF(zip_stat); - goto error; - } - Py_XDECREF(zip_stat); } else Py_INCREF(files); @@ -594,8 +560,7 @@ { ZipImporter *self = (ZipImporter *)obj; PyObject *path, *key; - FILE *fp; - PyObject *toc_entry, *data; + PyObject *toc_entry; Py_ssize_t path_start, path_len, len; if (!PyArg_ParseTuple(args, "U:zipimporter.get_data", &path)) @@ -623,23 +588,15 @@ key = PyUnicode_Substring(path, path_start, path_len); if (key == NULL) goto error; - - fp = safely_reopen_archive(self); - if (fp == NULL) - goto error; - toc_entry = PyDict_GetItem(self->files, key); if (toc_entry == NULL) { PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, key); Py_DECREF(key); - fclose(fp); goto error; } Py_DECREF(key); Py_DECREF(path); - data = get_data(fp, self->archive, toc_entry); - fclose(fp); - return data; + return get_data(self->archive, toc_entry); error: Py_DECREF(path); return NULL; @@ -664,7 +621,6 @@ PyObject *toc_entry; PyObject *fullname, *subname, *path, *fullpath; enum zi_module_info mi; - FILE *fp; if (!PyArg_ParseTuple(args, "U:zipimporter.get_source", &fullname)) return NULL; @@ -694,18 +650,11 @@ if (fullpath == NULL) return NULL; - fp = safely_reopen_archive(self); - if (fp == NULL) { - Py_DECREF(fullpath); - return NULL; - } - toc_entry = PyDict_GetItem(self->files, fullpath); Py_DECREF(fullpath); if (toc_entry != NULL) { PyObject *res, *bytes; - bytes = get_data(fp, self->archive, toc_entry); - fclose(fp); + bytes = get_data(self->archive, toc_entry); if (bytes == NULL) return NULL; res = PyUnicode_FromStringAndSize(PyBytes_AS_STRING(bytes), @@ -713,10 +662,10 @@ Py_DECREF(bytes); return res; } - fclose(fp); /* we have the module, but no source */ - Py_RETURN_NONE; + Py_INCREF(Py_None); + return Py_None; } PyDoc_STRVAR(doc_find_module, @@ -882,133 +831,10 @@ return x; } -/* Return 1 if objects a and b fail a Py_EQ test for an attr. */ -static int -compare_obj_attr_strings(PyObject *obj_a, PyObject *obj_b, char *attr_name) -{ - int problem = 0; - PyObject *attr_a = PyObject_GetAttrString(obj_a, attr_name); - PyObject *attr_b = PyObject_GetAttrString(obj_b, attr_name); - if (attr_a == NULL || attr_b == NULL) - problem = 1; - else - problem = (PyObject_RichCompareBool(attr_a, attr_b, Py_EQ) != 1); - Py_XDECREF(attr_a); - Py_XDECREF(attr_b); - return problem; -} +/* + read_directory(archive) -> files dict (new reference) -/* - * Returns an open FILE * on success. - * Returns NULL on error with the Python error context set. - */ -static FILE * -safely_reopen_archive(ZipImporter *self) -{ - FILE *fp; - PyObject *stat_now = NULL; - - fp = fopen_rb_and_stat(self->archive, &stat_now); - if (!fp) { - PyErr_Format(ZipImportError, - "zipimport: can not open file %U", self->archive); - Py_XDECREF(stat_now); - return NULL; - } - - if (stat_now != NULL) { - int problem = 0; - PyObject *files; - PyObject *prev_stat = PyDict_GetItem(zip_stat_cache, self->archive); - /* Test stat_now vs the old cached stat on some key attributes. */ - if (prev_stat != NULL) { - problem = compare_obj_attr_strings(prev_stat, stat_now, - "st_ino"); - problem |= compare_obj_attr_strings(prev_stat, stat_now, - "st_size"); - problem |= compare_obj_attr_strings(prev_stat, stat_now, - "st_mtime"); - } else { - if (Py_VerboseFlag) - PySys_FormatStderr("# zipimport: no stat data for %U!\n", - self->archive); - problem = 1; - } - - if (problem) { - if (Py_VerboseFlag) - PySys_FormatStderr("# zipimport: %U modified since last" - " import, rereading TOC.\n", self->archive); - files = read_directory(fp, self->archive); - if (files == NULL) { - Py_DECREF(stat_now); - fclose(fp); - return NULL; - } - if (PyDict_SetItem(zip_directory_cache, self->archive, - files) != 0) { - Py_DECREF(files); - Py_DECREF(stat_now); - fclose(fp); - return NULL; - } - if (stat_now && PyDict_SetItem(zip_stat_cache, self->archive, - stat_now) != 0) { - Py_DECREF(files); - Py_DECREF(stat_now); - fclose(fp); - return NULL; - } - Py_XDECREF(self->files); /* free the old value. */ - self->files = files; - } - Py_DECREF(stat_now); - } /* stat succeeded */ - - return fp; -} - -/* - fopen_rb_and_stat(path, &py_stat) -> FILE * - - Opens path in "rb" mode and populates the Python py_stat stat_result - with information about the opened file. *py_stat may not be changed - if there is no fstat_function or if fstat_function fails. - - Returns NULL and does nothing to *py_stat if the open failed. -*/ -static FILE * -fopen_rb_and_stat(PyObject *path, PyObject **py_stat_p) -{ - FILE *fp; - assert(py_stat_p != NULL); - assert(*py_stat_p == NULL); - - fp = _Py_fopen_obj(path, "rb"); - if (fp == NULL) { - if (!PyErr_Occurred()) - PyErr_Format(ZipImportError, - "zipimport: can not open file %U", path); - return NULL; - } - - if (fstat_function) { - PyObject *stat_result = PyObject_CallFunction(fstat_function, - "i", fileno(fp)); - if (stat_result == NULL) { - PyErr_Clear(); /* We can function without it. */ - } else { - *py_stat_p = stat_result; - } - } - - return fp; -} - -/* - read_directory(fp, archive) -> files dict (new reference) - - Given an open Zip archive, build a dict, mapping file names + Given a path to a Zip archive, build a dict, mapping file names (local to the archive, using SEP as a separator) to toc entries. A toc_entry is a tuple: @@ -1028,9 +854,10 @@ data_size and file_offset are 0. */ static PyObject * -read_directory(FILE *fp, PyObject *archive) +read_directory(PyObject *archive) { PyObject *files = NULL; + FILE *fp; unsigned short flags; short compress, time, date, name_size; long crc, data_size, file_size, header_size; @@ -1046,18 +873,27 @@ const char *charset; int bootstrap; - assert(fp != NULL); + fp = _Py_fopen_obj(archive, "rb"); + if (fp == NULL) { + if (!PyErr_Occurred()) + PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); + return NULL; + } + if (fseek(fp, -22, SEEK_END) == -1) { + fclose(fp); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; } header_position = ftell(fp); if (fread(endof_central_dir, 1, 22, fp) != 22) { + fclose(fp); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; } if (get_long((unsigned char *)endof_central_dir) != 0x06054B50) { /* Bad: End of Central Dir signature */ + fclose(fp); PyErr_Format(ZipImportError, "not a Zip file: %R", archive); return NULL; } @@ -1164,16 +1000,19 @@ goto error; count++; } + fclose(fp); if (Py_VerboseFlag) PySys_FormatStderr("# zipimport: found %ld names in %R\n", count, archive); return files; file_error: + fclose(fp); Py_XDECREF(files); Py_XDECREF(nameobj); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; error: + fclose(fp); Py_XDECREF(files); Py_XDECREF(nameobj); return NULL; @@ -1212,13 +1051,14 @@ return decompress; } -/* Given a FILE* to a Zip file and a toc_entry, return the (uncompressed) +/* Given a path to a Zip file and a toc_entry, return the (uncompressed) data as a new reference. */ static PyObject * -get_data(FILE *fp, PyObject *archive, PyObject *toc_entry) +get_data(PyObject *archive, PyObject *toc_entry) { PyObject *raw_data, *data = NULL, *decompress; char *buf; + FILE *fp; int err; Py_ssize_t bytes_read = 0; long l; @@ -1232,8 +1072,17 @@ return NULL; } + fp = _Py_fopen_obj(archive, "rb"); + if (!fp) { + if (!PyErr_Occurred()) + PyErr_Format(PyExc_IOError, + "zipimport: can not open file %U", archive); + return NULL; + } + /* Check to make sure the local file header is correct */ if (fseek(fp, file_offset, 0) == -1) { + fclose(fp); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; } @@ -1245,9 +1094,11 @@ PyErr_Format(ZipImportError, "bad local file header in %U", archive); + fclose(fp); return NULL; } if (fseek(fp, file_offset + 26, 0) == -1) { + fclose(fp); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; } @@ -1255,6 +1106,7 @@ l = 30 + PyMarshal_ReadShortFromFile(fp) + PyMarshal_ReadShortFromFile(fp); /* local header size */ if (PyErr_Occurred()) { + fclose(fp); return NULL; } file_offset += l; /* Start of file data */ @@ -1265,6 +1117,7 @@ raw_data = PyBytes_FromStringAndSize((char *)NULL, bytes_size); if (raw_data == NULL) { + fclose(fp); return NULL; } buf = PyBytes_AsString(raw_data); @@ -1273,9 +1126,11 @@ if (err == 0) { bytes_read = fread(buf, 1, data_size, fp); } else { + fclose(fp); PyErr_Format(ZipImportError, "can't read Zip file: %R", archive); return NULL; } + fclose(fp); if (err || bytes_read != data_size) { PyErr_SetString(PyExc_IOError, "zipimport: can't read data"); @@ -1496,12 +1351,12 @@ /* Return the code object for the module named by 'fullname' from the Zip archive as a new reference. */ static PyObject * -get_code_from_data(ZipImporter *self, FILE *fp, int ispackage, int isbytecode, +get_code_from_data(ZipImporter *self, int ispackage, int isbytecode, time_t mtime, PyObject *toc_entry) { PyObject *data, *modpath, *code; - data = get_data(fp, self->archive, toc_entry); + data = get_data(self->archive, toc_entry); if (data == NULL) return NULL; @@ -1523,7 +1378,6 @@ PyObject *code = NULL, *toc_entry, *subname; PyObject *path, *fullpath = NULL; struct st_zip_searchorder *zso; - FILE *fp; subname = get_subname(fullname); if (subname == NULL) @@ -1534,12 +1388,6 @@ if (path == NULL) return NULL; - fp = safely_reopen_archive(self); - if (fp == NULL) { - Py_DECREF(path); - return NULL; - } - for (zso = zip_searchorder; *zso->suffix; zso++) { code = NULL; @@ -1550,7 +1398,6 @@ if (Py_VerboseFlag > 1) PySys_FormatStderr("# trying %U%c%U\n", self->archive, (int)SEP, fullpath); - toc_entry = PyDict_GetItem(self->files, fullpath); if (toc_entry != NULL) { time_t mtime = 0; @@ -1566,7 +1413,7 @@ Py_CLEAR(fullpath); if (p_ispackage != NULL) *p_ispackage = ispackage; - code = get_code_from_data(self, fp, ispackage, + code = get_code_from_data(self, ispackage, isbytecode, mtime, toc_entry); if (code == Py_None) { @@ -1586,7 +1433,6 @@ } PyErr_Format(ZipImportError, "can't find module %R", fullname); exit: - fclose(fp); Py_DECREF(path); Py_XDECREF(fullpath); return code; @@ -1604,8 +1450,6 @@ subclass of ImportError, so it can be caught as ImportError, too.\n\ - _zip_directory_cache: a dict, mapping archive paths to zip directory\n\ info dicts, as used in zipimporter._files.\n\ -- _zip_stat_cache: a dict, mapping archive paths to stat_result\n\ - info for the .zip the last time anything was imported from it.\n\ \n\ It is usually not needed to use the zipimport module explicitly; it is\n\ used by the builtin import mechanism for sys.path items that are paths\n\ @@ -1665,7 +1509,6 @@ (PyObject *)&ZipImporter_Type) < 0) return NULL; - Py_XDECREF(zip_directory_cache); /* Avoid embedded interpreter leaks. */ zip_directory_cache = PyDict_New(); if (zip_directory_cache == NULL) return NULL; @@ -1673,36 +1516,5 @@ if (PyModule_AddObject(mod, "_zip_directory_cache", zip_directory_cache) < 0) return NULL; - - Py_XDECREF(zip_stat_cache); /* Avoid embedded interpreter leaks. */ - zip_stat_cache = PyDict_New(); - if (zip_stat_cache == NULL) - return NULL; - Py_INCREF(zip_stat_cache); - if (PyModule_AddObject(mod, "_zip_stat_cache", zip_stat_cache) < 0) - return NULL; - - { - /* We cannot import "os" here as that is a .py/.pyc file that could - * live within a zipped up standard library. Import the posix or nt - * builtin that provides the fstat() function we want instead. */ - PyObject *os_like_module; - Py_CLEAR(fstat_function); /* Avoid embedded interpreter leaks. */ - os_like_module = PyImport_ImportModule("posix"); - if (os_like_module == NULL) { - PyErr_Clear(); - os_like_module = PyImport_ImportModule("nt"); - } - if (os_like_module != NULL) { - fstat_function = PyObject_GetAttrString(os_like_module, "fstat"); - Py_DECREF(os_like_module); - } - if (fstat_function == NULL) { - PyErr_Clear(); /* non-fatal, we'll go on without it. */ - if (Py_VerboseFlag) - PySys_WriteStderr("# zipimport unable to use os.fstat().\n"); - } - } - return mod; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:46 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjE2?= =?utf-8?q?=3A_Add_a_format=28=29_method_to_tracemalloc=2ETraceback=2E?= Message-ID: <3fnSN271J0z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/132821b65e22 changeset: 89715:132821b65e22 branch: 3.4 user: Victor Stinner date: Sun Feb 16 23:53:38 2014 +0100 summary: Issue #20616: Add a format() method to tracemalloc.Traceback. files: Doc/library/tracemalloc.rst | 31 ++++++++++++++++++----- Lib/test/test_tracemalloc.py | 20 +++++++++++++++ Lib/tracemalloc.py | 13 ++++++++++ Misc/NEWS | 2 + 4 files changed, 59 insertions(+), 7 deletions(-) diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -118,7 +118,6 @@ Code to display the traceback of the biggest memory block:: - import linecache import tracemalloc # Store 25 frames @@ -132,12 +131,8 @@ # pick the biggest memory block stat = top_stats[0] print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024)) - for frame in stat.traceback: - print(' File "%s", line %s' % (frame.filename, frame.lineno)) - line = linecache.getline(frame.filename, frame.lineno) - line = line.strip() - if line: - print(' ' + line) + for line in stat.traceback.format(): + print(line) Example of output of the Python test suite (traceback limited to 25 frames):: @@ -602,4 +597,26 @@ The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback` instance. + .. method:: format(limit=None) + Format the traceback as a list of lines with newlines. Use the + :mod:`linecache` module to retrieve lines from the source code. If + *limit* is set, only format the *limit* most recent frames. + + Similar to the :func:`traceback.format_tb` function, except that + :meth:`format` does not include newlines. + + Example:: + + print("Traceback (most recent call first):") + for line in traceback: + print(line) + + Output:: + + Traceback (most recent call first): + File "test.py", line 9 + obj = Object() + File "test.py", line 12 + tb = tracemalloc.get_object_traceback(f()) + diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -510,6 +510,26 @@ self.assertEqual(traceback[:2], (traceback[0], traceback[1])) + def test_format_traceback(self): + snapshot, snapshot2 = create_snapshots() + def getline(filename, lineno): + return ' <%s, %s>' % (filename, lineno) + with unittest.mock.patch('tracemalloc.linecache.getline', + side_effect=getline): + tb = snapshot.traces[0].traceback + self.assertEqual(tb.format(), + [' File "a.py", line 2', + ' ', + ' File "b.py", line 4', + ' ']) + + self.assertEqual(tb.format(limit=1), + [' File "a.py", line 2', + ' ']) + + self.assertEqual(tb.format(limit=-1), + []) + class TestFilters(unittest.TestCase): maxDiff = 2048 diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py --- a/Lib/tracemalloc.py +++ b/Lib/tracemalloc.py @@ -1,6 +1,7 @@ from collections import Sequence from functools import total_ordering import fnmatch +import linecache import os.path import pickle @@ -205,6 +206,18 @@ def __repr__(self): return "" % (tuple(self),) + def format(self, limit=None): + lines = [] + if limit is not None and limit < 0: + return lines + for frame in self[:limit]: + lines.append(' File "%s", line %s' + % (frame.filename, frame.lineno)) + line = linecache.getline(frame.filename, frame.lineno).strip() + if line: + lines.append(' %s' % line) + return lines + def get_object_traceback(obj): """ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #20616: Add a format() method to tracemalloc.Traceback. + - Issue #19744: the ensurepip installation step now just prints a warning to stderr rather than failing outright if SSL/TLS is unavailable. This allows local installation of POSIX builds without SSL/TLS support. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:48 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwNjUy?= =?utf-8?q?=3A_asyncio_doc=3A_close_the_event_loop_in_run=5Fforever=28=29_?= =?utf-8?q?example=2E_Fix?= Message-ID: <3fnSN41clfz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/36005fe2ab9b changeset: 89716:36005fe2ab9b branch: 3.4 user: Victor Stinner date: Mon Feb 17 10:54:30 2014 +0100 summary: Close #20652: asyncio doc: close the event loop in run_forever() example. Fix also typo. Patch written by Vajrasky Kok. files: Doc/library/asyncio-task.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -229,7 +229,7 @@ @asyncio.coroutine def slow_operation(future): yield from asyncio.sleep(1) - future.set_result('Future in done!') + future.set_result('Future is done!') loop = asyncio.get_event_loop() future = asyncio.Future() @@ -261,7 +261,7 @@ @asyncio.coroutine def slow_operation(future): yield from asyncio.sleep(1) - future.set_result('Future in done!') + future.set_result('Future is done!') def got_result(future): print(future.result()) @@ -271,7 +271,10 @@ future = asyncio.Future() asyncio.Task(slow_operation(future)) future.add_done_callback(got_result) - loop.run_forever() + try: + loop.run_forever() + finally: + loop.close() In this example, the future is responsible to display the result and to stop the loop. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:49 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjU1?= =?utf-8?q?=3A_Fix_test=5Fasyncio=2C_run_also_subprocess_tests=2E_Patch_wr?= =?utf-8?q?itten_by?= Message-ID: <3fnSN53JKNz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/b22b83ea719d changeset: 89717:b22b83ea719d branch: 3.4 user: Victor Stinner date: Tue Feb 18 00:11:21 2014 +0100 summary: Issue #20655: Fix test_asyncio, run also subprocess tests. Patch written by Vajrasky Kok. files: Lib/test/test_asyncio/tests.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/tests.txt b/Lib/test/test_asyncio/tests.txt --- a/Lib/test/test_asyncio/tests.txt +++ b/Lib/test/test_asyncio/tests.txt @@ -6,6 +6,7 @@ test_asyncio.test_queues test_asyncio.test_selector_events test_asyncio.test_streams +test_asyncio.test_subprocess test_asyncio.test_tasks test_asyncio.test_transports test_asyncio.test_unix_events -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:50 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjY3?= =?utf-8?q?=3A_test=5Fasyncio=3A_Skip_KqueueEventLoopTests=2Etest=5Fread?= =?utf-8?q?=5Fpty=5Foutput=28=29_on?= Message-ID: <3fnSN65TTLz7LkY@mail.python.org> http://hg.python.org/cpython/rev/5f1a4782184f changeset: 89718:5f1a4782184f branch: 3.4 user: Victor Stinner date: Tue Feb 18 01:30:03 2014 +0100 summary: Issue #20667: test_asyncio: Skip KqueueEventLoopTests.test_read_pty_output() on OpenBSD older than 5.5 files: Lib/test/support/__init__.py | 10 ++++++++++ Lib/test/test_asyncio/test_events.py | 3 +++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -78,6 +78,7 @@ "create_empty_file", "can_symlink", "fs_is_case_insensitive", # unittest "is_resource_enabled", "requires", "requires_freebsd_version", + "requires_openbsd_version", "requires_linux_version", "requires_mac_ver", "check_syntax_error", "TransientResource", "time_out", "socket_peer_reset", "ioerror_peer_reset", "transient_internet", "BasicTestRunner", "run_unittest", "run_doctest", @@ -467,6 +468,15 @@ """ return _requires_unix_version('FreeBSD', min_version) +def requires_openbsd_version(*min_version): + """Decorator raising SkipTest if the OS is OpenBSD and the OpenBSD version + is less than `min_version`. + + For example, @requires_freebsd_version(5, 4) raises SkipTest if the FreeBSD + version is less than 5.4. + """ + return _requires_unix_version('OpenBSD', min_version) + def requires_linux_version(*min_version): """Decorator raising SkipTest if the OS is Linux and the Linux version is less than `min_version`. diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1622,6 +1622,9 @@ # kqueue doesn't support character devices (PTY) on Mac OS X older # than 10.9 (Maverick) @support.requires_mac_ver(10, 9) + # Issue #20667: KqueueEventLoopTests.test_read_pty_output() + # hangs on OpenBSD 5.4 + @support.requires_openbsd_version(5, 5) def test_read_pty_output(self): super().test_read_pty_output() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:52 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjY3?= =?utf-8?q?=3A_KqueueEventLoopTests=2Etest=5Fread=5Fpty=5Foutput=28=29_han?= =?utf-8?q?gs_also_on_OpenBSD?= Message-ID: <3fnSN80FMbz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/add2c31ecfb1 changeset: 89719:add2c31ecfb1 branch: 3.4 user: Victor Stinner date: Tue Feb 18 09:13:47 2014 +0100 summary: Issue #20667: KqueueEventLoopTests.test_read_pty_output() hangs also on OpenBSD 5.5. files: Lib/test/support/__init__.py | 10 ---------- Lib/test/test_asyncio/test_events.py | 5 +++-- 2 files changed, 3 insertions(+), 12 deletions(-) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -78,7 +78,6 @@ "create_empty_file", "can_symlink", "fs_is_case_insensitive", # unittest "is_resource_enabled", "requires", "requires_freebsd_version", - "requires_openbsd_version", "requires_linux_version", "requires_mac_ver", "check_syntax_error", "TransientResource", "time_out", "socket_peer_reset", "ioerror_peer_reset", "transient_internet", "BasicTestRunner", "run_unittest", "run_doctest", @@ -468,15 +467,6 @@ """ return _requires_unix_version('FreeBSD', min_version) -def requires_openbsd_version(*min_version): - """Decorator raising SkipTest if the OS is OpenBSD and the OpenBSD version - is less than `min_version`. - - For example, @requires_freebsd_version(5, 4) raises SkipTest if the FreeBSD - version is less than 5.4. - """ - return _requires_unix_version('OpenBSD', min_version) - def requires_linux_version(*min_version): """Decorator raising SkipTest if the OS is Linux and the Linux version is less than `min_version`. diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1623,8 +1623,9 @@ # than 10.9 (Maverick) @support.requires_mac_ver(10, 9) # Issue #20667: KqueueEventLoopTests.test_read_pty_output() - # hangs on OpenBSD 5.4 - @support.requires_openbsd_version(5, 5) + # hangs on OpenBSD 5.5 + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'test hangs on OpenBSD') def test_read_pty_output(self): super().test_read_pty_output() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:53 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwNjQ5?= =?utf-8?q?=3A_Fix_typo_in_asyncio_doc=2E_Patch_written_by_Brett_Cannon=2E?= Message-ID: <3fnSN9231Wz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/38516b2ffe92 changeset: 89720:38516b2ffe92 branch: 3.4 user: Victor Stinner date: Tue Feb 18 09:22:00 2014 +0100 summary: Close #20649: Fix typo in asyncio doc. Patch written by Brett Cannon. files: Doc/library/asyncio-dev.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -13,7 +13,7 @@ ------------------------------ An event loop runs in a thread and executes all callbacks and tasks in the same -thread. While a task in running in the event loop, no other task is running in +thread. While a task is running in the event loop, no other task is running in the same thread. But when the task uses ``yield from``, the task is suspended and the event loop executes the next task. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:54 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDkz?= =?utf-8?q?=3A_Document_that_asyncio_should_not_exceed_one_day?= Message-ID: <3fnSNB3X8Tz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/41c6c066feb2 changeset: 89721:41c6c066feb2 branch: 3.4 user: Victor Stinner date: Tue Feb 18 09:37:43 2014 +0100 summary: Issue #20493: Document that asyncio should not exceed one day files: Doc/library/asyncio-eventloop.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -157,6 +157,10 @@ implementation; ideally it is a monotonic clock. This will generally be a different clock than :func:`time.time`. +.. note:: + + Timeouts (relative *delay* or absolute *when*) should not exceed one day. + .. method:: BaseEventLoop.call_later(delay, callback, *args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:56 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQWRk?= =?utf-8?q?_support_for_UNIX_Domain_Sockets=2E?= Message-ID: <3fnSND2ws4z7LkN@mail.python.org> http://hg.python.org/cpython/rev/50c4176d865d changeset: 89722:50c4176d865d branch: 3.4 user: Yury Selivanov date: Tue Feb 18 12:15:06 2014 -0500 summary: asyncio: Add support for UNIX Domain Sockets. files: Lib/asyncio/base_events.py | 7 + Lib/asyncio/events.py | 26 + Lib/asyncio/streams.py | 39 +- Lib/asyncio/test_utils.py | 153 +++- Lib/asyncio/unix_events.py | 75 ++- Lib/test/test_asyncio/test_base_events.py | 2 +- Lib/test/test_asyncio/test_events.py | 353 ++++++--- Lib/test/test_asyncio/test_selector_events.py | 3 +- Lib/test/test_asyncio/test_streams.py | 215 ++++- Lib/test/test_asyncio/test_unix_events.py | 82 ++- 10 files changed, 750 insertions(+), 205 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -407,6 +407,13 @@ sock.setblocking(False) + transport, protocol = yield from self._create_connection_transport( + sock, protocol_factory, ssl, server_hostname) + return transport, protocol + + @tasks.coroutine + def _create_connection_transport(self, sock, protocol_factory, ssl, + server_hostname): protocol = protocol_factory() waiter = futures.Future(loop=self) if ssl: diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -220,6 +220,32 @@ """ raise NotImplementedError + def create_unix_connection(self, protocol_factory, path, *, + ssl=None, sock=None, + server_hostname=None): + raise NotImplementedError + + def create_unix_server(self, protocol_factory, path, *, + sock=None, backlog=100, ssl=None): + """A coroutine which creates a UNIX Domain Socket server. + + The return valud is a Server object, which can be used to stop + the service. + + path is a str, representing a file systsem path to bind the + server socket to. + + sock can optionally be specified in order to use a preexisting + socket object. + + backlog is the maximum number of queued connections passed to + listen() (defaults to 100). + + ssl can be set to an SSLContext to enable SSL over the + accepted connections. + """ + raise NotImplementedError + def create_datagram_endpoint(self, protocol_factory, local_addr=None, remote_addr=None, *, family=0, proto=0, flags=0): diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -1,9 +1,13 @@ """Stream-related things.""" __all__ = ['StreamReader', 'StreamWriter', 'StreamReaderProtocol', - 'open_connection', 'start_server', 'IncompleteReadError', + 'open_connection', 'start_server', + 'open_unix_connection', 'start_unix_server', + 'IncompleteReadError', ] +import socket + from . import events from . import futures from . import protocols @@ -93,6 +97,39 @@ return (yield from loop.create_server(factory, host, port, **kwds)) +if hasattr(socket, 'AF_UNIX'): + # UNIX Domain Sockets are supported on this platform + + @tasks.coroutine + def open_unix_connection(path=None, *, + loop=None, limit=_DEFAULT_LIMIT, **kwds): + """Similar to `open_connection` but works with UNIX Domain Sockets.""" + if loop is None: + loop = events.get_event_loop() + reader = StreamReader(limit=limit, loop=loop) + protocol = StreamReaderProtocol(reader, loop=loop) + transport, _ = yield from loop.create_unix_connection( + lambda: protocol, path, **kwds) + writer = StreamWriter(transport, protocol, reader, loop) + return reader, writer + + + @tasks.coroutine + def start_unix_server(client_connected_cb, path=None, *, + loop=None, limit=_DEFAULT_LIMIT, **kwds): + """Similar to `start_server` but works with UNIX Domain Sockets.""" + if loop is None: + loop = events.get_event_loop() + + def factory(): + reader = StreamReader(limit=limit, loop=loop) + protocol = StreamReaderProtocol(reader, client_connected_cb, + loop=loop) + return protocol + + return (yield from loop.create_unix_server(factory, path, **kwds)) + + class FlowControlMixin(protocols.Protocol): """Reusable flow control logic for StreamWriter.drain(). diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -4,12 +4,18 @@ import contextlib import io import os +import socket +import socketserver import sys +import tempfile import threading import time import unittest import unittest.mock + +from http.server import HTTPServer from wsgiref.simple_server import make_server, WSGIRequestHandler, WSGIServer + try: import ssl except ImportError: # pragma: no cover @@ -70,42 +76,51 @@ loop.run_forever() - at contextlib.contextmanager -def run_test_server(*, host='127.0.0.1', port=0, use_ssl=False): +class SilentWSGIRequestHandler(WSGIRequestHandler): - class SilentWSGIRequestHandler(WSGIRequestHandler): - def get_stderr(self): - return io.StringIO() + def get_stderr(self): + return io.StringIO() - def log_message(self, format, *args): + def log_message(self, format, *args): + pass + + +class SilentWSGIServer(WSGIServer): + + def handle_error(self, request, client_address): + pass + + +class SSLWSGIServerMixin: + + def finish_request(self, request, client_address): + # The relative location of our test directory (which + # contains the ssl key and certificate files) differs + # between the stdlib and stand-alone asyncio. + # Prefer our own if we can find it. + here = os.path.join(os.path.dirname(__file__), '..', 'tests') + if not os.path.isdir(here): + here = os.path.join(os.path.dirname(os.__file__), + 'test', 'test_asyncio') + keyfile = os.path.join(here, 'ssl_key.pem') + certfile = os.path.join(here, 'ssl_cert.pem') + ssock = ssl.wrap_socket(request, + keyfile=keyfile, + certfile=certfile, + server_side=True) + try: + self.RequestHandlerClass(ssock, client_address, self) + ssock.close() + except OSError: + # maybe socket has been closed by peer pass - class SilentWSGIServer(WSGIServer): - def handle_error(self, request, client_address): - pass - class SSLWSGIServer(SilentWSGIServer): - def finish_request(self, request, client_address): - # The relative location of our test directory (which - # contains the ssl key and certificate files) differs - # between the stdlib and stand-alone asyncio. - # Prefer our own if we can find it. - here = os.path.join(os.path.dirname(__file__), '..', 'tests') - if not os.path.isdir(here): - here = os.path.join(os.path.dirname(os.__file__), - 'test', 'test_asyncio') - keyfile = os.path.join(here, 'ssl_key.pem') - certfile = os.path.join(here, 'ssl_cert.pem') - ssock = ssl.wrap_socket(request, - keyfile=keyfile, - certfile=certfile, - server_side=True) - try: - self.RequestHandlerClass(ssock, client_address, self) - ssock.close() - except OSError: - # maybe socket has been closed by peer - pass +class SSLWSGIServer(SSLWSGIServerMixin, SilentWSGIServer): + pass + + +def _run_test_server(*, address, use_ssl=False, server_cls, server_ssl_cls): def app(environ, start_response): status = '200 OK' @@ -115,9 +130,9 @@ # Run the test WSGI server in a separate thread in order not to # interfere with event handling in the main thread - server_class = SSLWSGIServer if use_ssl else SilentWSGIServer - httpd = make_server(host, port, app, - server_class, SilentWSGIRequestHandler) + server_class = server_ssl_cls if use_ssl else server_cls + httpd = server_class(address, SilentWSGIRequestHandler) + httpd.set_app(app) httpd.address = httpd.server_address server_thread = threading.Thread(target=httpd.serve_forever) server_thread.start() @@ -129,6 +144,75 @@ server_thread.join() +if hasattr(socket, 'AF_UNIX'): + + class UnixHTTPServer(socketserver.UnixStreamServer, HTTPServer): + + def server_bind(self): + socketserver.UnixStreamServer.server_bind(self) + self.server_name = '127.0.0.1' + self.server_port = 80 + + + class UnixWSGIServer(UnixHTTPServer, WSGIServer): + + def server_bind(self): + UnixHTTPServer.server_bind(self) + self.setup_environ() + + def get_request(self): + request, client_addr = super().get_request() + # Code in the stdlib expects that get_request + # will return a socket and a tuple (host, port). + # However, this isn't true for UNIX sockets, + # as the second return value will be a path; + # hence we return some fake data sufficient + # to get the tests going + return request, ('127.0.0.1', '') + + + class SilentUnixWSGIServer(UnixWSGIServer): + + def handle_error(self, request, client_address): + pass + + + class UnixSSLWSGIServer(SSLWSGIServerMixin, SilentUnixWSGIServer): + pass + + + def gen_unix_socket_path(): + with tempfile.NamedTemporaryFile() as file: + return file.name + + + @contextlib.contextmanager + def unix_socket_path(): + path = gen_unix_socket_path() + try: + yield path + finally: + try: + os.unlink(path) + except OSError: + pass + + + @contextlib.contextmanager + def run_test_unix_server(*, use_ssl=False): + with unix_socket_path() as path: + yield from _run_test_server(address=path, use_ssl=use_ssl, + server_cls=SilentUnixWSGIServer, + server_ssl_cls=UnixSSLWSGIServer) + + + at contextlib.contextmanager +def run_test_server(*, host='127.0.0.1', port=0, use_ssl=False): + yield from _run_test_server(address=(host, port), use_ssl=use_ssl, + server_cls=SilentWSGIServer, + server_ssl_cls=SSLWSGIServer) + + def make_test_protocol(base): dct = {} for name in dir(base): @@ -275,5 +359,6 @@ def _write_to_self(self): pass + def MockCallback(**kwargs): return unittest.mock.Mock(spec=['__call__'], **kwargs) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -11,6 +11,7 @@ import threading +from . import base_events from . import base_subprocess from . import constants from . import events @@ -31,9 +32,9 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop): - """Unix event loop + """Unix event loop. - Adds signal handling to SelectorEventLoop + Adds signal handling and UNIX Domain Socket support to SelectorEventLoop. """ def __init__(self, selector=None): @@ -164,6 +165,76 @@ def _child_watcher_callback(self, pid, returncode, transp): self.call_soon_threadsafe(transp._process_exited, returncode) + @tasks.coroutine + def create_unix_connection(self, protocol_factory, path, *, + ssl=None, sock=None, + server_hostname=None): + assert server_hostname is None or isinstance(server_hostname, str) + if ssl: + if server_hostname is None: + raise ValueError( + 'you have to pass server_hostname when using ssl') + else: + if server_hostname is not None: + raise ValueError('server_hostname is only meaningful with ssl') + + if path is not None: + if sock is not None: + raise ValueError( + 'path and sock can not be specified at the same time') + + try: + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) + sock.setblocking(False) + yield from self.sock_connect(sock, path) + except OSError: + if sock is not None: + sock.close() + raise + + else: + if sock is None: + raise ValueError('no path and sock were specified') + sock.setblocking(False) + + transport, protocol = yield from self._create_connection_transport( + sock, protocol_factory, ssl, server_hostname) + return transport, protocol + + @tasks.coroutine + def create_unix_server(self, protocol_factory, path=None, *, + sock=None, backlog=100, ssl=None): + if isinstance(ssl, bool): + raise TypeError('ssl argument must be an SSLContext or None') + + if path is not None: + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + + try: + sock.bind(path) + except OSError as exc: + if exc.errno == errno.EADDRINUSE: + # Let's improve the error message by adding + # with what exact address it occurs. + msg = 'Address {!r} is already in use'.format(path) + raise OSError(errno.EADDRINUSE, msg) from None + else: + raise + else: + if sock is None: + raise ValueError( + 'path was not specified, and no sock specified') + + if sock.family != socket.AF_UNIX: + raise ValueError( + 'A UNIX Domain Socket was expected, got {!r}'.format(sock)) + + server = base_events.Server(self, [sock]) + sock.listen(backlog) + sock.setblocking(False) + self._start_serving(protocol_factory, sock, ssl, server) + return server + def _set_nonblocking(fd): flags = fcntl.fcntl(fd, fcntl.F_GETFL) diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -212,7 +212,7 @@ idx = -1 data = [10.0, 10.0, 10.3, 13.0] - self.loop._scheduled = [asyncio.TimerHandle(11.0, lambda:True, ())] + self.loop._scheduled = [asyncio.TimerHandle(11.0, lambda: True, ())] self.loop._run_once() self.assertEqual(logging.DEBUG, m_logger.log.call_args[0][0]) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -39,13 +39,14 @@ return fullname raise FileNotFoundError(filename) + ONLYCERT = data_file('ssl_cert.pem') ONLYKEY = data_file('ssl_key.pem') SIGNED_CERTFILE = data_file('keycert3.pem') SIGNING_CA = data_file('pycacert.pem') -class MyProto(asyncio.Protocol): +class MyBaseProto(asyncio.Protocol): done = None def __init__(self, loop=None): @@ -59,7 +60,6 @@ self.transport = transport assert self.state == 'INITIAL', self.state self.state = 'CONNECTED' - transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n') def data_received(self, data): assert self.state == 'CONNECTED', self.state @@ -76,6 +76,12 @@ self.done.set_result(None) +class MyProto(MyBaseProto): + def connection_made(self, transport): + super().connection_made(transport) + transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n') + + class MyDatagramProto(asyncio.DatagramProtocol): done = None @@ -357,22 +363,30 @@ r.close() self.assertGreaterEqual(len(data), 200) + def _basetest_sock_client_ops(self, httpd, sock): + sock.setblocking(False) + self.loop.run_until_complete( + self.loop.sock_connect(sock, httpd.address)) + self.loop.run_until_complete( + self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) + data = self.loop.run_until_complete( + self.loop.sock_recv(sock, 1024)) + # consume data + self.loop.run_until_complete( + self.loop.sock_recv(sock, 1024)) + sock.close() + self.assertTrue(data.startswith(b'HTTP/1.0 200 OK')) + def test_sock_client_ops(self): with test_utils.run_test_server() as httpd: sock = socket.socket() - sock.setblocking(False) - self.loop.run_until_complete( - self.loop.sock_connect(sock, httpd.address)) - self.loop.run_until_complete( - self.loop.sock_sendall(sock, b'GET / HTTP/1.0\r\n\r\n')) - data = self.loop.run_until_complete( - self.loop.sock_recv(sock, 1024)) - # consume data - self.loop.run_until_complete( - self.loop.sock_recv(sock, 1024)) - sock.close() + self._basetest_sock_client_ops(httpd, sock) - self.assertTrue(data.startswith(b'HTTP/1.0 200 OK')) + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_unix_sock_client_ops(self): + with test_utils.run_test_unix_server() as httpd: + sock = socket.socket(socket.AF_UNIX) + self._basetest_sock_client_ops(httpd, sock) def test_sock_client_fail(self): # Make sure that we will get an unused port @@ -485,16 +499,26 @@ self.loop.run_forever() self.assertEqual(caught, 1) + def _basetest_create_connection(self, connection_fut): + tr, pr = self.loop.run_until_complete(connection_fut) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, asyncio.Protocol) + self.loop.run_until_complete(pr.done) + self.assertGreater(pr.nbytes, 0) + tr.close() + def test_create_connection(self): with test_utils.run_test_server() as httpd: - f = self.loop.create_connection( + conn_fut = self.loop.create_connection( lambda: MyProto(loop=self.loop), *httpd.address) - tr, pr = self.loop.run_until_complete(f) - self.assertIsInstance(tr, asyncio.Transport) - self.assertIsInstance(pr, asyncio.Protocol) - self.loop.run_until_complete(pr.done) - self.assertGreater(pr.nbytes, 0) - tr.close() + self._basetest_create_connection(conn_fut) + + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_connection(self): + with test_utils.run_test_unix_server() as httpd: + conn_fut = self.loop.create_unix_connection( + lambda: MyProto(loop=self.loop), httpd.address) + self._basetest_create_connection(conn_fut) def test_create_connection_sock(self): with test_utils.run_test_server() as httpd: @@ -524,20 +548,37 @@ self.assertGreater(pr.nbytes, 0) tr.close() + def _basetest_create_ssl_connection(self, connection_fut): + tr, pr = self.loop.run_until_complete(connection_fut) + self.assertIsInstance(tr, asyncio.Transport) + self.assertIsInstance(pr, asyncio.Protocol) + self.assertTrue('ssl' in tr.__class__.__name__.lower()) + self.assertIsNotNone(tr.get_extra_info('sockname')) + self.loop.run_until_complete(pr.done) + self.assertGreater(pr.nbytes, 0) + tr.close() + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_ssl_connection(self): with test_utils.run_test_server(use_ssl=True) as httpd: - f = self.loop.create_connection( - lambda: MyProto(loop=self.loop), *httpd.address, + conn_fut = self.loop.create_connection( + lambda: MyProto(loop=self.loop), + *httpd.address, ssl=test_utils.dummy_ssl_context()) - tr, pr = self.loop.run_until_complete(f) - self.assertIsInstance(tr, asyncio.Transport) - self.assertIsInstance(pr, asyncio.Protocol) - self.assertTrue('ssl' in tr.__class__.__name__.lower()) - self.assertIsNotNone(tr.get_extra_info('sockname')) - self.loop.run_until_complete(pr.done) - self.assertGreater(pr.nbytes, 0) - tr.close() + + self._basetest_create_ssl_connection(conn_fut) + + @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_ssl_unix_connection(self): + with test_utils.run_test_unix_server(use_ssl=True) as httpd: + conn_fut = self.loop.create_unix_connection( + lambda: MyProto(loop=self.loop), + httpd.address, + ssl=test_utils.dummy_ssl_context(), + server_hostname='127.0.0.1') + + self._basetest_create_ssl_connection(conn_fut) def test_create_connection_local_addr(self): with test_utils.run_test_server() as httpd: @@ -561,14 +602,8 @@ self.assertIn(str(httpd.address), cm.exception.strerror) def test_create_server(self): - proto = None - - def factory(): - nonlocal proto - proto = MyProto() - return proto - - f = self.loop.create_server(factory, '0.0.0.0', 0) + proto = MyProto() + f = self.loop.create_server(lambda: proto, '0.0.0.0', 0) server = self.loop.run_until_complete(f) self.assertEqual(len(server.sockets), 1) sock = server.sockets[0] @@ -605,38 +640,76 @@ # close server server.close() - def _make_ssl_server(self, factory, certfile, keyfile=None): + def _make_unix_server(self, factory, **kwargs): + path = test_utils.gen_unix_socket_path() + self.addCleanup(lambda: os.path.exists(path) and os.unlink(path)) + + f = self.loop.create_unix_server(factory, path, **kwargs) + server = self.loop.run_until_complete(f) + + return server, path + + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_server(self): + proto = MyProto() + server, path = self._make_unix_server(lambda: proto) + self.assertEqual(len(server.sockets), 1) + + client = socket.socket(socket.AF_UNIX) + client.connect(path) + client.sendall(b'xxx') + test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: proto is not None, 10) + + self.assertIsInstance(proto, MyProto) + self.assertEqual('INITIAL', proto.state) + test_utils.run_briefly(self.loop) + self.assertEqual('CONNECTED', proto.state) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0, + timeout=10) + self.assertEqual(3, proto.nbytes) + + # close connection + proto.transport.close() + test_utils.run_briefly(self.loop) # windows iocp + + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # close server + server.close() + + def _create_ssl_context(self, certfile, keyfile=None): sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) sslcontext.options |= ssl.OP_NO_SSLv2 sslcontext.load_cert_chain(certfile, keyfile) + return sslcontext - f = self.loop.create_server( - factory, '127.0.0.1', 0, ssl=sslcontext) + def _make_ssl_server(self, factory, certfile, keyfile=None): + sslcontext = self._create_ssl_context(certfile, keyfile) + f = self.loop.create_server(factory, '127.0.0.1', 0, ssl=sslcontext) server = self.loop.run_until_complete(f) + sock = server.sockets[0] host, port = sock.getsockname() self.assertEqual(host, '127.0.0.1') return server, host, port + def _make_ssl_unix_server(self, factory, certfile, keyfile=None): + sslcontext = self._create_ssl_context(certfile, keyfile) + return self._make_unix_server(factory, ssl=sslcontext) + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl(self): - proto = None + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, ONLYCERT, ONLYKEY) - class ClientMyProto(MyProto): - def connection_made(self, transport): - self.transport = transport - assert self.state == 'INITIAL', self.state - self.state = 'CONNECTED' - - def factory(): - nonlocal proto - proto = MyProto(loop=self.loop) - return proto - - server, host, port = self._make_ssl_server(factory, ONLYCERT, ONLYKEY) - - f_c = self.loop.create_connection(ClientMyProto, host, port, + f_c = self.loop.create_connection(MyBaseProto, host, port, ssl=test_utils.dummy_ssl_context()) client, pr = self.loop.run_until_complete(f_c) @@ -667,16 +740,45 @@ server.close() @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_server_ssl(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, ONLYCERT, ONLYKEY) + + f_c = self.loop.create_unix_connection( + MyBaseProto, path, ssl=test_utils.dummy_ssl_context(), + server_hostname='') + + client, pr = self.loop.run_until_complete(f_c) + + client.write(b'xxx') + test_utils.run_briefly(self.loop) + self.assertIsInstance(proto, MyProto) + test_utils.run_briefly(self.loop) + self.assertEqual('CONNECTED', proto.state) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0, + timeout=10) + self.assertEqual(3, proto.nbytes) + + # close connection + proto.transport.close() + self.loop.run_until_complete(proto.done) + self.assertEqual('CLOSED', proto.state) + + # the client socket must be closed after to avoid ECONNRESET upon + # recv()/send() on the serving socket + client.close() + + # stop serving + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_verify_failed(self): - proto = None - - def factory(): - nonlocal proto - proto = MyProto(loop=self.loop) - return proto - - server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, SIGNED_CERTFILE) sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) sslcontext_client.options |= ssl.OP_NO_SSLv2 @@ -697,15 +799,36 @@ @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_server_ssl_verify_failed(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # no CA loaded + f_c = self.loop.create_unix_connection(MyProto, path, + ssl=sslcontext_client, + server_hostname='invalid') + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # close connection + self.assertIsNone(proto.transport) + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_match_failed(self): - proto = None - - def factory(): - nonlocal proto - proto = MyProto(loop=self.loop) - return proto - - server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, SIGNED_CERTFILE) sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) sslcontext_client.options |= ssl.OP_NO_SSLv2 @@ -729,15 +852,36 @@ @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_create_unix_server_ssl_verified(self): + proto = MyProto(loop=self.loop) + server, path = self._make_ssl_unix_server( + lambda: proto, SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations(cafile=SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # Connection succeeds with correct CA and server hostname. + f_c = self.loop.create_unix_connection(MyProto, path, + ssl=sslcontext_client, + server_hostname='localhost') + client, pr = self.loop.run_until_complete(f_c) + + # close connection + proto.transport.close() + client.close() + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_verified(self): - proto = None - - def factory(): - nonlocal proto - proto = MyProto(loop=self.loop) - return proto - - server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + proto = MyProto(loop=self.loop) + server, host, port = self._make_ssl_server( + lambda: proto, SIGNED_CERTFILE) sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) sslcontext_client.options |= ssl.OP_NO_SSLv2 @@ -915,19 +1059,15 @@ @unittest.skipUnless(sys.platform != 'win32', "Don't support pipes for Windows") def test_read_pipe(self): - proto = None - - def factory(): - nonlocal proto - proto = MyReadPipeProto(loop=self.loop) - return proto + proto = MyReadPipeProto(loop=self.loop) rpipe, wpipe = os.pipe() pipeobj = io.open(rpipe, 'rb', 1024) @asyncio.coroutine def connect(): - t, p = yield from self.loop.connect_read_pipe(factory, pipeobj) + t, p = yield from self.loop.connect_read_pipe( + lambda: proto, pipeobj) self.assertIs(p, proto) self.assertIs(t, proto.transport) self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) @@ -959,19 +1099,14 @@ # Issue #20495: The test hangs on FreeBSD 7.2 but pass on FreeBSD 9 @support.requires_freebsd_version(8) def test_read_pty_output(self): - proto = None - - def factory(): - nonlocal proto - proto = MyReadPipeProto(loop=self.loop) - return proto + proto = MyReadPipeProto(loop=self.loop) master, slave = os.openpty() master_read_obj = io.open(master, 'rb', 0) @asyncio.coroutine def connect(): - t, p = yield from self.loop.connect_read_pipe(factory, + t, p = yield from self.loop.connect_read_pipe(lambda: proto, master_read_obj) self.assertIs(p, proto) self.assertIs(t, proto.transport) @@ -999,21 +1134,17 @@ @unittest.skipUnless(sys.platform != 'win32', "Don't support pipes for Windows") def test_write_pipe(self): - proto = None + proto = MyWritePipeProto(loop=self.loop) transport = None - def factory(): - nonlocal proto - proto = MyWritePipeProto(loop=self.loop) - return proto - rpipe, wpipe = os.pipe() pipeobj = io.open(wpipe, 'wb', 1024) @asyncio.coroutine def connect(): nonlocal transport - t, p = yield from self.loop.connect_write_pipe(factory, pipeobj) + t, p = yield from self.loop.connect_write_pipe( + lambda: proto, pipeobj) self.assertIs(p, proto) self.assertIs(t, proto.transport) self.assertEqual('CONNECTED', proto.state) @@ -1045,21 +1176,16 @@ @unittest.skipUnless(sys.platform != 'win32', "Don't support pipes for Windows") def test_write_pipe_disconnect_on_close(self): - proto = None + proto = MyWritePipeProto(loop=self.loop) transport = None - def factory(): - nonlocal proto - proto = MyWritePipeProto(loop=self.loop) - return proto - rsock, wsock = test_utils.socketpair() pipeobj = io.open(wsock.detach(), 'wb', 1024) @asyncio.coroutine def connect(): nonlocal transport - t, p = yield from self.loop.connect_write_pipe(factory, + t, p = yield from self.loop.connect_write_pipe(lambda: proto, pipeobj) self.assertIs(p, proto) self.assertIs(t, proto.transport) @@ -1084,21 +1210,16 @@ # older than 10.6 (Snow Leopard) @support.requires_mac_ver(10, 6) def test_write_pty(self): - proto = None + proto = MyWritePipeProto(loop=self.loop) transport = None - def factory(): - nonlocal proto - proto = MyWritePipeProto(loop=self.loop) - return proto - master, slave = os.openpty() slave_write_obj = io.open(slave, 'wb', 0) @asyncio.coroutine def connect(): nonlocal transport - t, p = yield from self.loop.connect_write_pipe(factory, + t, p = yield from self.loop.connect_write_pipe(lambda: proto, slave_write_obj) self.assertIs(p, proto) self.assertIs(t, proto.transport) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -55,7 +55,8 @@ self.loop.remove_reader = unittest.mock.Mock() self.loop.remove_writer = unittest.mock.Mock() waiter = asyncio.Future(loop=self.loop) - transport = self.loop._make_ssl_transport(m, asyncio.Protocol(), m, waiter) + transport = self.loop._make_ssl_transport( + m, asyncio.Protocol(), m, waiter) self.assertIsInstance(transport, _SelectorSslTransport) @unittest.mock.patch('asyncio.selector_events.ssl', None) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -1,6 +1,8 @@ """Tests for streams.py.""" +import functools import gc +import socket import unittest import unittest.mock try: @@ -32,48 +34,85 @@ stream = asyncio.StreamReader() self.assertIs(stream._loop, m_events.get_event_loop.return_value) + def _basetest_open_connection(self, open_connection_fut): + reader, writer = self.loop.run_until_complete(open_connection_fut) + writer.write(b'GET / HTTP/1.0\r\n\r\n') + f = reader.readline() + data = self.loop.run_until_complete(f) + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + f = reader.read() + data = self.loop.run_until_complete(f) + self.assertTrue(data.endswith(b'\r\n\r\nTest message')) + writer.close() + def test_open_connection(self): with test_utils.run_test_server() as httpd: - f = asyncio.open_connection(*httpd.address, loop=self.loop) - reader, writer = self.loop.run_until_complete(f) - writer.write(b'GET / HTTP/1.0\r\n\r\n') - f = reader.readline() - data = self.loop.run_until_complete(f) - self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') - f = reader.read() - data = self.loop.run_until_complete(f) - self.assertTrue(data.endswith(b'\r\n\r\nTest message')) + conn_fut = asyncio.open_connection(*httpd.address, + loop=self.loop) + self._basetest_open_connection(conn_fut) - writer.close() + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_open_unix_connection(self): + with test_utils.run_test_unix_server() as httpd: + conn_fut = asyncio.open_unix_connection(httpd.address, + loop=self.loop) + self._basetest_open_connection(conn_fut) + + def _basetest_open_connection_no_loop_ssl(self, open_connection_fut): + try: + reader, writer = self.loop.run_until_complete(open_connection_fut) + finally: + asyncio.set_event_loop(None) + writer.write(b'GET / HTTP/1.0\r\n\r\n') + f = reader.read() + data = self.loop.run_until_complete(f) + self.assertTrue(data.endswith(b'\r\n\r\nTest message')) + + writer.close() @unittest.skipIf(ssl is None, 'No ssl module') def test_open_connection_no_loop_ssl(self): with test_utils.run_test_server(use_ssl=True) as httpd: - try: - asyncio.set_event_loop(self.loop) - f = asyncio.open_connection(*httpd.address, - ssl=test_utils.dummy_ssl_context()) - reader, writer = self.loop.run_until_complete(f) - finally: - asyncio.set_event_loop(None) - writer.write(b'GET / HTTP/1.0\r\n\r\n') - f = reader.read() - data = self.loop.run_until_complete(f) - self.assertTrue(data.endswith(b'\r\n\r\nTest message')) + conn_fut = asyncio.open_connection( + *httpd.address, + ssl=test_utils.dummy_ssl_context(), + loop=self.loop) - writer.close() + self._basetest_open_connection_no_loop_ssl(conn_fut) + + @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_open_unix_connection_no_loop_ssl(self): + with test_utils.run_test_unix_server(use_ssl=True) as httpd: + conn_fut = asyncio.open_unix_connection( + httpd.address, + ssl=test_utils.dummy_ssl_context(), + server_hostname='', + loop=self.loop) + + self._basetest_open_connection_no_loop_ssl(conn_fut) + + def _basetest_open_connection_error(self, open_connection_fut): + reader, writer = self.loop.run_until_complete(open_connection_fut) + writer._protocol.connection_lost(ZeroDivisionError()) + f = reader.read() + with self.assertRaises(ZeroDivisionError): + self.loop.run_until_complete(f) + writer.close() + test_utils.run_briefly(self.loop) def test_open_connection_error(self): with test_utils.run_test_server() as httpd: - f = asyncio.open_connection(*httpd.address, loop=self.loop) - reader, writer = self.loop.run_until_complete(f) - writer._protocol.connection_lost(ZeroDivisionError()) - f = reader.read() - with self.assertRaises(ZeroDivisionError): - self.loop.run_until_complete(f) + conn_fut = asyncio.open_connection(*httpd.address, + loop=self.loop) + self._basetest_open_connection_error(conn_fut) - writer.close() - test_utils.run_briefly(self.loop) + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_open_unix_connection_error(self): + with test_utils.run_test_unix_server() as httpd: + conn_fut = asyncio.open_unix_connection(httpd.address, + loop=self.loop) + self._basetest_open_connection_error(conn_fut) def test_feed_empty_data(self): stream = asyncio.StreamReader(loop=self.loop) @@ -415,10 +454,86 @@ client_writer.write(data) def start(self): + sock = socket.socket() + sock.bind(('127.0.0.1', 0)) self.server = self.loop.run_until_complete( asyncio.start_server(self.handle_client, - '127.0.0.1', 12345, + sock=sock, loop=self.loop)) + return sock.getsockname() + + def handle_client_callback(self, client_reader, client_writer): + task = asyncio.Task(client_reader.readline(), loop=self.loop) + + def done(task): + client_writer.write(task.result()) + + task.add_done_callback(done) + + def start_callback(self): + sock = socket.socket() + sock.bind(('127.0.0.1', 0)) + addr = sock.getsockname() + sock.close() + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client_callback, + host=addr[0], port=addr[1], + loop=self.loop)) + return addr + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None + + @asyncio.coroutine + def client(addr): + reader, writer = yield from asyncio.open_connection( + *addr, loop=self.loop) + # send a line + writer.write(b"hello world!\n") + # read it back + msgback = yield from reader.readline() + writer.close() + return msgback + + # test the server variant with a coroutine as client handler + server = MyServer(self.loop) + addr = server.start() + msg = self.loop.run_until_complete(asyncio.Task(client(addr), + loop=self.loop)) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + # test the server variant with a callback as client handler + server = MyServer(self.loop) + addr = server.start_callback() + msg = self.loop.run_until_complete(asyncio.Task(client(addr), + loop=self.loop)) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') + def test_start_unix_server(self): + + class MyServer: + + def __init__(self, loop, path): + self.server = None + self.loop = loop + self.path = path + + @asyncio.coroutine + def handle_client(self, client_reader, client_writer): + data = yield from client_reader.readline() + client_writer.write(data) + + def start(self): + self.server = self.loop.run_until_complete( + asyncio.start_unix_server(self.handle_client, + path=self.path, + loop=self.loop)) def handle_client_callback(self, client_reader, client_writer): task = asyncio.Task(client_reader.readline(), loop=self.loop) @@ -430,9 +545,9 @@ def start_callback(self): self.server = self.loop.run_until_complete( - asyncio.start_server(self.handle_client_callback, - '127.0.0.1', 12345, - loop=self.loop)) + asyncio.start_unix_server(self.handle_client_callback, + path=self.path, + loop=self.loop)) def stop(self): if self.server is not None: @@ -441,9 +556,9 @@ self.server = None @asyncio.coroutine - def client(): - reader, writer = yield from asyncio.open_connection( - '127.0.0.1', 12345, loop=self.loop) + def client(path): + reader, writer = yield from asyncio.open_unix_connection( + path, loop=self.loop) # send a line writer.write(b"hello world!\n") # read it back @@ -452,20 +567,22 @@ return msgback # test the server variant with a coroutine as client handler - server = MyServer(self.loop) - server.start() - msg = self.loop.run_until_complete(asyncio.Task(client(), - loop=self.loop)) - server.stop() - self.assertEqual(msg, b"hello world!\n") + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start() + msg = self.loop.run_until_complete(asyncio.Task(client(path), + loop=self.loop)) + server.stop() + self.assertEqual(msg, b"hello world!\n") # test the server variant with a callback as client handler - server = MyServer(self.loop) - server.start_callback() - msg = self.loop.run_until_complete(asyncio.Task(client(), - loop=self.loop)) - server.stop() - self.assertEqual(msg, b"hello world!\n") + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start_callback() + msg = self.loop.run_until_complete(asyncio.Task(client(path), + loop=self.loop)) + server.stop() + self.assertEqual(msg, b"hello world!\n") if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -7,8 +7,10 @@ import os import pprint import signal +import socket import stat import sys +import tempfile import threading import unittest import unittest.mock @@ -24,7 +26,7 @@ @unittest.skipUnless(signal, 'Signals are not supported') -class SelectorEventLoopTests(unittest.TestCase): +class SelectorEventLoopSignalTests(unittest.TestCase): def setUp(self): self.loop = asyncio.SelectorEventLoop() @@ -200,6 +202,84 @@ m_signal.set_wakeup_fd.assert_called_once_with(-1) + at unittest.skipUnless(hasattr(socket, 'AF_UNIX'), + 'UNIX Sockets are not supported') +class SelectorEventLoopUnixSocketTests(unittest.TestCase): + + def setUp(self): + self.loop = asyncio.SelectorEventLoop() + asyncio.set_event_loop(None) + + def tearDown(self): + self.loop.close() + + def test_create_unix_server_existing_path_sock(self): + with test_utils.unix_socket_path() as path: + sock = socket.socket(socket.AF_UNIX) + sock.bind(path) + + coro = self.loop.create_unix_server(lambda: None, path) + with self.assertRaisesRegexp(OSError, + 'Address.*is already in use'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_existing_path_nonsock(self): + with tempfile.NamedTemporaryFile() as file: + coro = self.loop.create_unix_server(lambda: None, file.name) + with self.assertRaisesRegexp(OSError, + 'Address.*is already in use'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_ssl_bool(self): + coro = self.loop.create_unix_server(lambda: None, path='spam', + ssl=True) + with self.assertRaisesRegex(TypeError, + 'ssl argument must be an SSLContext'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_nopath_nosock(self): + coro = self.loop.create_unix_server(lambda: None, path=None) + with self.assertRaisesRegex(ValueError, + 'path was not specified, and no sock'): + self.loop.run_until_complete(coro) + + def test_create_unix_server_path_inetsock(self): + coro = self.loop.create_unix_server(lambda: None, path=None, + sock=socket.socket()) + with self.assertRaisesRegex(ValueError, + 'A UNIX Domain Socket was expected'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_path_sock(self): + coro = self.loop.create_unix_connection( + lambda: None, '/dev/null', sock=object()) + with self.assertRaisesRegex(ValueError, 'path and sock can not be'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_nopath_nosock(self): + coro = self.loop.create_unix_connection( + lambda: None, None) + with self.assertRaisesRegex(ValueError, + 'no path and sock were specified'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_nossl_serverhost(self): + coro = self.loop.create_unix_connection( + lambda: None, '/dev/null', server_hostname='spam') + with self.assertRaisesRegex(ValueError, + 'server_hostname is only meaningful'): + self.loop.run_until_complete(coro) + + def test_create_unix_connection_ssl_noserverhost(self): + coro = self.loop.create_unix_connection( + lambda: None, '/dev/null', ssl=True) + + with self.assertRaisesRegexp( + ValueError, 'you have to pass server_hostname when using ssl'): + + self.loop.run_until_complete(coro) + + class UnixReadPipeTransportTests(unittest.TestCase): def setUp(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:57 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogTWlzYy9ORVdTOiBB?= =?utf-8?q?dd_a_news_item_for_UNIX_Sockets_support_in_asyncio=2E_Cleanup_W?= =?utf-8?q?S=2E?= Message-ID: <3fnSNF5GQXz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/befaf4b93db7 changeset: 89723:befaf4b93db7 branch: 3.4 user: Yury Selivanov date: Tue Feb 18 12:21:57 2014 -0500 summary: Misc/NEWS: Add a news item for UNIX Sockets support in asyncio. Cleanup WS. files: Misc/NEWS | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,10 @@ Library ------- +- Issue #20673: Implement support for UNIX Domain Sockets in asyncio. + New APIs: loop.create_unix_connection(), loop.create_unix_server(), + streams.open_unix_connection(), and streams.start_unix_server(). + - Issue #20616: Add a format() method to tracemalloc.Traceback. - Issue #19744: the ensurepip installation step now just prints a warning to -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:31:59 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:31:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjA5?= =?utf-8?q?=3A_Merge_with_3=2E3=2E?= Message-ID: <3fnSNH16rGz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/6b4225607cf1 changeset: 89724:6b4225607cf1 branch: 3.4 user: Zachary Ware date: Tue Feb 18 11:35:15 2014 -0600 summary: Issue #20609: Merge with 3.3. files: Misc/NEWS | 6 +++ PCbuild/pythoncore.vcxproj | 48 +++++++++++++++++++++---- 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -35,6 +35,12 @@ - Issue #20594: Avoid name clash with the libc function posix_close. +Build +----- + +- Issue #20609: Restored the ability to build 64-bit Windows binaries on + 32-bit Windows, which was broken by the change in issue #19788. + What's New in Python 3.4.0 release candidate 1? =============================================== diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -182,7 +182,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -213,7 +217,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -244,7 +252,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -278,7 +290,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -307,7 +323,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -339,7 +359,11 @@ MachineX64 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -368,7 +392,11 @@ 0x1e000000 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... @@ -400,7 +428,11 @@ MachineX64 - $(KillPythonExe) + $(KillPythonExe) +IF %ERRORLEVEL% NEQ 0 ( + echo kill_python: warning: could not kill running Pythons, exit code %ERRORLEVEL% + exit /b 0 +) Killing any running $(PythonExe) instances... -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:00 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogTWFuZ2xlIF9fcGFy?= =?utf-8?q?ameters_in_=5F=5Fannotations=5F=5F_dict_properly=2E_Issue_=2320?= =?utf-8?q?625=2E?= Message-ID: <3fnSNJ2ss6z7Lk3@mail.python.org> http://hg.python.org/cpython/rev/4d7c3cbd8515 changeset: 89725:4d7c3cbd8515 branch: 3.4 user: Yury Selivanov date: Tue Feb 18 12:49:41 2014 -0500 summary: Mangle __parameters in __annotations__ dict properly. Issue #20625. files: Doc/whatsnew/3.4.rst | 4 ++++ Lib/test/test_grammar.py | 7 +++++++ Lib/test/test_inspect.py | 16 ++++++++++++++++ Misc/NEWS | 3 +++ Python/compile.c | 8 +++++++- 5 files changed, 37 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1704,6 +1704,10 @@ informative :exc:`ValueError` rather than the previous more mysterious :exc:`AttributeError` (:issue:`9177`). +* Parameter names in ``__annotations__`` dict are now mangled properly, + similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in + :issue:`20625`). + Changes in the C API -------------------- diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -314,6 +314,13 @@ self.assertEqual(f.__annotations__, {'b': 1, 'c': 2, 'e': 3, 'g': 6, 'h': 7, 'j': 9, 'k': 11, 'return': 12}) + # Check for issue #20625 -- annotations mangling + class Spam: + def f(self, *, __kw:1): + pass + class Ham(Spam): pass + self.assertEquals(Spam.f.__annotations__, {'_Spam__kw': 1}) + self.assertEquals(Ham.f.__annotations__, {'_Spam__kw': 1}) # Check for SF Bug #1697248 - mixing decorators and a return annotation def null(x): return x @null diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -2390,6 +2390,22 @@ self.assertEqual(sig.return_annotation, 42) self.assertEqual(sig, inspect.signature(test)) + def test_signature_on_mangled_parameters(self): + class Spam: + def foo(self, __p1:1=2, *, __p2:2=3): + pass + class Ham(Spam): + pass + + self.assertEqual(self.signature(Spam.foo), + ((('self', ..., ..., "positional_or_keyword"), + ('_Spam__p1', 2, 1, "positional_or_keyword"), + ('_Spam__p2', 3, 2, "keyword_only")), + ...)) + + self.assertEqual(self.signature(Spam.foo), + self.signature(Ham.foo)) + class TestParameterObject(unittest.TestCase): def test_signature_parameter_kinds(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #20625: Parameter names in __annotations__ were not mangled properly. + Discovered by Jonas Wielicki, patch by Yury Selivanov. + - Issue #20261: In pickle, lookup __getnewargs__ and __getnewargs_ex__ on the type of the object. diff --git a/Python/compile.c b/Python/compile.c --- a/Python/compile.c +++ b/Python/compile.c @@ -1533,8 +1533,14 @@ { if (annotation) { VISIT(c, expr, annotation); - if (PyList_Append(names, id)) + PyObject *mangled = _Py_Mangle(c->u->u_private, id); + if (!mangled) return -1; + if (PyList_Append(names, mangled) < 0) { + Py_DECREF(mangled); + return -1; + } + Py_DECREF(mangled); } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:01 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogTWFr?= =?utf-8?q?e_tests_pass_on_Windows=2E?= Message-ID: <3fnSNK4Wzdz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/8439d0becc62 changeset: 89726:8439d0becc62 branch: 3.4 user: Guido van Rossum date: Tue Feb 18 10:24:30 2014 -0800 summary: asyncio: Make tests pass on Windows. files: Lib/asyncio/streams.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -2,12 +2,14 @@ __all__ = ['StreamReader', 'StreamWriter', 'StreamReaderProtocol', 'open_connection', 'start_server', - 'open_unix_connection', 'start_unix_server', 'IncompleteReadError', ] import socket +if hasattr(socket, 'AF_UNIX'): + __all__.extend(['open_unix_connection', 'start_unix_server']) + from . import events from . import futures from . import protocols -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:02 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Close_issue206?= =?utf-8?q?53=3A_allow_Enum_subclasses_to_override_=5F=5Freduce=5Fex=5F=5F?= Message-ID: <3fnSNL737yz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/010723a7bd25 changeset: 89727:010723a7bd25 branch: 3.4 user: Ethan Furman date: Tue Feb 18 12:37:12 2014 -0800 summary: Close issue20653: allow Enum subclasses to override __reduce_ex__ files: Lib/enum.py | 26 ++++++------ Lib/test/test_enum.py | 61 ++++++++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 14 deletions(-) diff --git a/Lib/enum.py b/Lib/enum.py --- a/Lib/enum.py +++ b/Lib/enum.py @@ -116,12 +116,14 @@ enum_class._value2member_map_ = {} # check for a supported pickle protocols, and if not present sabotage - # pickling, since it won't work anyway - if member_type is not object: - methods = ('__getnewargs_ex__', '__getnewargs__', - '__reduce_ex__', '__reduce__') - if not any(map(member_type.__dict__.get, methods)): - _make_class_unpicklable(enum_class) + # pickling, since it won't work anyway. + # if new class implements its own __reduce_ex__, do not sabotage + if classdict.get('__reduce_ex__') is None: + if member_type is not object: + methods = ('__getnewargs_ex__', '__getnewargs__', + '__reduce_ex__', '__reduce__') + if not any(map(member_type.__dict__.get, methods)): + _make_class_unpicklable(enum_class) # instantiate them, checking for duplicates as we go # we instantiate first instead of checking for duplicates first in case @@ -167,7 +169,7 @@ # double check that repr and friends are not the mixin's or various # things break (such as pickle) - for name in ('__repr__', '__str__', '__format__', '__getnewargs__', '__reduce_ex__'): + for name in ('__repr__', '__str__', '__format__', '__reduce_ex__'): class_method = getattr(enum_class, name) obj_method = getattr(member_type, name, None) enum_method = getattr(first_enum, name, None) @@ -192,8 +194,9 @@ (i.e. Color = Enum('Color', names='red green blue')). When used for the functional API: `module`, if set, will be stored in - the new class' __module__ attribute; `type`, if set, will be mixed in - as the first base class. + the new class' __module__ attribute; `qualname`, if set, will be stored + in the new class' __qualname__ attribute; `type`, if set, will be mixed + in as the first base class. Note: if `module` is not set this routine will attempt to discover the calling module by walking the frame stack; if this is unsuccessful @@ -465,14 +468,11 @@ val = self.value return cls.__format__(val, format_spec) - def __getnewargs__(self): - return (self._value_, ) - def __hash__(self): return hash(self._name_) def __reduce_ex__(self, proto): - return self.__class__, self.__getnewargs__() + return self.__class__, (self._value_, ) # DynamicClassAttribute is used to provide access to the `name` and # `value` properties of enum members while keeping some measure of diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -956,6 +956,7 @@ test_pickle_dump_load(self.assertEqual, NI5, 5) self.assertEqual(NEI.y.value, 2) test_pickle_dump_load(self.assertIs, NEI.y) + test_pickle_dump_load(self.assertIs, NEI) def test_subclasses_with_getnewargs_ex(self): class NamedInt(int): @@ -1012,6 +1013,7 @@ test_pickle_dump_load(self.assertEqual, NI5, 5, protocol=(4, 4)) self.assertEqual(NEI.y.value, 2) test_pickle_dump_load(self.assertIs, NEI.y, protocol=(4, 4)) + test_pickle_dump_load(self.assertIs, NEI) def test_subclasses_with_reduce(self): class NamedInt(int): @@ -1068,6 +1070,7 @@ test_pickle_dump_load(self.assertEqual, NI5, 5) self.assertEqual(NEI.y.value, 2) test_pickle_dump_load(self.assertIs, NEI.y) + test_pickle_dump_load(self.assertIs, NEI) def test_subclasses_with_reduce_ex(self): class NamedInt(int): @@ -1124,8 +1127,9 @@ test_pickle_dump_load(self.assertEqual, NI5, 5) self.assertEqual(NEI.y.value, 2) test_pickle_dump_load(self.assertIs, NEI.y) + test_pickle_dump_load(self.assertIs, NEI) - def test_subclasses_without_getnewargs(self): + def test_subclasses_without_direct_pickle_support(self): class NamedInt(int): __qualname__ = 'NamedInt' def __new__(cls, *args): @@ -1178,6 +1182,61 @@ test_pickle_exception(self.assertRaises, TypeError, NEI.x) test_pickle_exception(self.assertRaises, PicklingError, NEI) + def test_subclasses_without_direct_pickle_support_using_name(self): + class NamedInt(int): + __qualname__ = 'NamedInt' + def __new__(cls, *args): + _args = args + name, *args = args + if len(args) == 0: + raise TypeError("name and value must be specified") + self = int.__new__(cls, *args) + self._intname = name + self._args = _args + return self + @property + def __name__(self): + return self._intname + def __repr__(self): + # repr() is updated to include the name and type info + return "{}({!r}, {})".format(type(self).__name__, + self.__name__, + int.__repr__(self)) + def __str__(self): + # str() is unchanged, even if it relies on the repr() fallback + base = int + base_str = base.__str__ + if base_str.__objclass__ is object: + return base.__repr__(self) + return base_str(self) + # for simplicity, we only define one operator that + # propagates expressions + def __add__(self, other): + temp = int(self) + int( other) + if isinstance(self, NamedInt) and isinstance(other, NamedInt): + return NamedInt( + '({0} + {1})'.format(self.__name__, other.__name__), + temp ) + else: + return temp + + class NEI(NamedInt, Enum): + __qualname__ = 'NEI' + x = ('the-x', 1) + y = ('the-y', 2) + def __reduce_ex__(self, proto): + return getattr, (self.__class__, self._name_) + + self.assertIs(NEI.__new__, Enum.__new__) + self.assertEqual(repr(NEI.x + NEI.y), "NamedInt('(the-x + the-y)', 3)") + globals()['NamedInt'] = NamedInt + globals()['NEI'] = NEI + NI5 = NamedInt('test', 5) + self.assertEqual(NI5, 5) + self.assertEqual(NEI.y.value, 2) + test_pickle_dump_load(self.assertIs, NEI.y) + test_pickle_dump_load(self.assertIs, NEI) + def test_tuple_subclass(self): class SomeTuple(tuple, Enum): __qualname__ = 'SomeTuple' # needed for pickle protocol 4 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:04 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjI1?= =?utf-8?q?=3A_Fix_compilation_issue?= Message-ID: <3fnSNN1vrQz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/3bced76d2706 changeset: 89728:3bced76d2706 branch: 3.4 user: Victor Stinner date: Tue Feb 18 22:07:56 2014 +0100 summary: Issue #20625: Fix compilation issue files: Python/compile.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Python/compile.c b/Python/compile.c --- a/Python/compile.c +++ b/Python/compile.c @@ -1532,8 +1532,9 @@ expr_ty annotation, PyObject *names) { if (annotation) { + PyObject *mangled; VISIT(c, expr, annotation); - PyObject *mangled = _Py_Mangle(c->u->u_private, id); + mangled = _Py_Mangle(c->u->u_private, id); if (!mangled) return -1; if (PyList_Append(names, mangled) < 0) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:06 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogTmV3?= =?utf-8?q?_error_handling_API=2E_Issue_=2320681=2E?= Message-ID: <3fnSNQ0CG3z7LkY@mail.python.org> http://hg.python.org/cpython/rev/3e592444da2f changeset: 89729:3e592444da2f branch: 3.4 user: Yury Selivanov date: Tue Feb 18 18:02:19 2014 -0500 summary: asyncio: New error handling API. Issue #20681. files: Lib/asyncio/base_events.py | 96 ++++- Lib/asyncio/events.py | 31 +- Lib/asyncio/futures.py | 22 +- Lib/asyncio/proactor_events.py | 33 +- Lib/asyncio/selector_events.py | 35 +- Lib/asyncio/test_utils.py | 18 +- Lib/asyncio/unix_events.py | 26 +- Lib/asyncio/windows_events.py | 8 +- Lib/test/test_asyncio/test_base_events.py | 194 +++++++++- Lib/test/test_asyncio/test_events.py | 44 +- Lib/test/test_asyncio/test_futures.py | 12 +- Lib/test/test_asyncio/test_proactor_events.py | 8 +- Lib/test/test_asyncio/test_selector_events.py | 20 +- Lib/test/test_asyncio/test_unix_events.py | 39 +- Misc/NEWS | 4 + 15 files changed, 491 insertions(+), 99 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -122,6 +122,7 @@ self._internal_fds = 0 self._running = False self._clock_resolution = time.get_clock_info('monotonic').resolution + self._exception_handler = None def _make_socket_transport(self, sock, protocol, waiter=None, *, extra=None, server=None): @@ -254,7 +255,7 @@ """Like call_later(), but uses an absolute time.""" if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_at()") - timer = events.TimerHandle(when, callback, args) + timer = events.TimerHandle(when, callback, args, self) heapq.heappush(self._scheduled, timer) return timer @@ -270,7 +271,7 @@ """ if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_soon()") - handle = events.Handle(callback, args) + handle = events.Handle(callback, args, self) self._ready.append(handle) return handle @@ -625,6 +626,97 @@ protocol, popen_args, False, stdin, stdout, stderr, bufsize, **kwargs) return transport, protocol + def set_exception_handler(self, handler): + """Set handler as the new event loop exception handler. + + If handler is None, the default exception handler will + be set. + + If handler is a callable object, it should have a + matching signature to '(loop, context)', where 'loop' + will be a reference to the active event loop, 'context' + will be a dict object (see `call_exception_handler()` + documentation for details about context). + """ + if handler is not None and not callable(handler): + raise TypeError('A callable object or None is expected, ' + 'got {!r}'.format(handler)) + self._exception_handler = handler + + def default_exception_handler(self, context): + """Default exception handler. + + This is called when an exception occurs and no exception + handler is set, and can be called by a custom exception + handler that wants to defer to the default behavior. + + context parameter has the same meaning as in + `call_exception_handler()`. + """ + message = context.get('message') + if not message: + message = 'Unhandled exception in event loop' + + exception = context.get('exception') + if exception is not None: + exc_info = (type(exception), exception, exception.__traceback__) + else: + exc_info = False + + log_lines = [message] + for key in sorted(context): + if key in {'message', 'exception'}: + continue + log_lines.append('{}: {!r}'.format(key, context[key])) + + logger.error('\n'.join(log_lines), exc_info=exc_info) + + def call_exception_handler(self, context): + """Call the current event loop exception handler. + + context is a dict object containing the following keys + (new keys maybe introduced later): + - 'message': Error message; + - 'exception' (optional): Exception object; + - 'future' (optional): Future instance; + - 'handle' (optional): Handle instance; + - 'protocol' (optional): Protocol instance; + - 'transport' (optional): Transport instance; + - 'socket' (optional): Socket instance. + + Note: this method should not be overloaded in subclassed + event loops. For any custom exception handling, use + `set_exception_handler()` method. + """ + if self._exception_handler is None: + try: + self.default_exception_handler(context) + except Exception: + # Second protection layer for unexpected errors + # in the default implementation, as well as for subclassed + # event loops with overloaded "default_exception_handler". + logger.error('Exception in default exception handler', + exc_info=True) + else: + try: + self._exception_handler(self, context) + except Exception as exc: + # Exception in the user set custom exception handler. + try: + # Let's try default handler. + self.default_exception_handler({ + 'message': 'Unhandled error in exception handler', + 'exception': exc, + 'context': context, + }) + except Exception: + # Guard 'default_exception_handler' in case it's + # overloaded. + logger.error('Exception in default exception handler ' + 'while handling an unexpected error ' + 'in custom exception handler', + exc_info=True) + def _add_callback(self, handle): """Add a Handle to ready or scheduled.""" assert isinstance(handle, events.Handle), 'A Handle is required here' diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -19,10 +19,11 @@ class Handle: """Object returned by callback registration methods.""" - __slots__ = ['_callback', '_args', '_cancelled'] + __slots__ = ['_callback', '_args', '_cancelled', '_loop'] - def __init__(self, callback, args): + def __init__(self, callback, args, loop): assert not isinstance(callback, Handle), 'A Handle is not a callback' + self._loop = loop self._callback = callback self._args = args self._cancelled = False @@ -39,9 +40,14 @@ def _run(self): try: self._callback(*self._args) - except Exception: - logger.exception('Exception in callback %s %r', - self._callback, self._args) + except Exception as exc: + msg = 'Exception in callback {}{!r}'.format(self._callback, + self._args) + self._loop.call_exception_handler({ + 'message': msg, + 'exception': exc, + 'handle': self, + }) self = None # Needed to break cycles when an exception occurs. @@ -50,9 +56,9 @@ __slots__ = ['_when'] - def __init__(self, when, callback, args): + def __init__(self, when, callback, args, loop): assert when is not None - super().__init__(callback, args) + super().__init__(callback, args, loop) self._when = when @@ -328,6 +334,17 @@ def remove_signal_handler(self, sig): raise NotImplementedError + # Error handlers. + + def set_exception_handler(self, handler): + raise NotImplementedError + + def default_exception_handler(self, context): + raise NotImplementedError + + def call_exception_handler(self, context): + raise NotImplementedError + class AbstractEventLoopPolicy: """Abstract policy for accessing the event loop.""" diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -83,9 +83,10 @@ in a discussion about closing files when they are collected. """ - __slots__ = ['exc', 'tb'] + __slots__ = ['exc', 'tb', 'loop'] - def __init__(self, exc): + def __init__(self, exc, loop): + self.loop = loop self.exc = exc self.tb = None @@ -102,8 +103,11 @@ def __del__(self): if self.tb: - logger.error('Future/Task exception was never retrieved:\n%s', - ''.join(self.tb)) + msg = 'Future/Task exception was never retrieved:\n{tb}' + context = { + 'message': msg.format(tb=''.join(self.tb)), + } + self.loop.call_exception_handler(context) class Future: @@ -173,8 +177,12 @@ # has consumed the exception return exc = self._exception - logger.error('Future/Task exception was never retrieved:', - exc_info=(exc.__class__, exc, exc.__traceback__)) + context = { + 'message': 'Future/Task exception was never retrieved', + 'exception': exc, + 'future': self, + } + self._loop.call_exception_handler(context) def cancel(self): """Cancel the future and schedule callbacks. @@ -309,7 +317,7 @@ if _PY34: self._log_traceback = True else: - self._tb_logger = _TracebackLogger(exception) + self._tb_logger = _TracebackLogger(exception, self._loop) # Arrange for the logger to be activated after all callbacks # have had a chance to call result() or exception(). self._loop.call_soon(self._tb_logger.activate) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -56,7 +56,12 @@ def _fatal_error(self, exc): if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): - logger.exception('Fatal error for %s', self) + self._loop.call_exception_handler({ + 'message': 'Fatal transport error', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) self._force_close(exc) def _force_close(self, exc): @@ -103,8 +108,13 @@ self._protocol_paused = True try: self._protocol.pause_writing() - except Exception: - logger.exception('pause_writing() failed') + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.pause_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) def _maybe_resume_protocol(self): if (self._protocol_paused and @@ -112,8 +122,13 @@ self._protocol_paused = False try: self._protocol.resume_writing() - except Exception: - logger.exception('resume_writing() failed') + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.resume_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) def set_write_buffer_limits(self, high=None, low=None): if high is None: @@ -465,9 +480,13 @@ conn, protocol, extra={'peername': addr}, server=server) f = self._proactor.accept(sock) - except OSError: + except OSError as exc: if sock.fileno() != -1: - logger.exception('Accept failed') + self.call_exception_handler({ + 'message': 'Accept failed', + 'exception': exc, + 'socket': sock, + }) sock.close() except futures.CancelledError: sock.close() diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -112,7 +112,11 @@ # Some platforms (e.g. Linux keep reporting the FD as # ready, so we remove the read handler temporarily. # We'll try again in a while. - logger.exception('Accept out of system resource (%s)', exc) + self.call_exception_handler({ + 'message': 'socket.accept() out of system resource', + 'exception': exc, + 'socket': sock, + }) self.remove_reader(sock.fileno()) self.call_later(constants.ACCEPT_RETRY_DELAY, self._start_serving, @@ -132,7 +136,7 @@ def add_reader(self, fd, callback, *args): """Add a reader callback.""" - handle = events.Handle(callback, args) + handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) except KeyError: @@ -167,7 +171,7 @@ def add_writer(self, fd, callback, *args): """Add a writer callback..""" - handle = events.Handle(callback, args) + handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) except KeyError: @@ -364,8 +368,13 @@ self._protocol_paused = True try: self._protocol.pause_writing() - except Exception: - logger.exception('pause_writing() failed') + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.pause_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) def _maybe_resume_protocol(self): if (self._protocol_paused and @@ -373,8 +382,13 @@ self._protocol_paused = False try: self._protocol.resume_writing() - except Exception: - logger.exception('resume_writing() failed') + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.resume_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) def set_write_buffer_limits(self, high=None, low=None): if high is None: @@ -435,7 +449,12 @@ def _fatal_error(self, exc): # Should be called from exception handler only. if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): - logger.exception('Fatal error for %s', self) + self._loop.call_exception_handler({ + 'message': 'Fatal transport error', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) self._force_close(exc) def _force_close(self, exc): diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -4,6 +4,7 @@ import contextlib import io import os +import re import socket import socketserver import sys @@ -301,7 +302,7 @@ raise AssertionError("Time generator is not finished") def add_reader(self, fd, callback, *args): - self.readers[fd] = events.Handle(callback, args) + self.readers[fd] = events.Handle(callback, args, self) def remove_reader(self, fd): self.remove_reader_count[fd] += 1 @@ -320,7 +321,7 @@ handle._args, args) def add_writer(self, fd, callback, *args): - self.writers[fd] = events.Handle(callback, args) + self.writers[fd] = events.Handle(callback, args, self) def remove_writer(self, fd): self.remove_writer_count[fd] += 1 @@ -362,3 +363,16 @@ def MockCallback(**kwargs): return unittest.mock.Mock(spec=['__call__'], **kwargs) + + +class MockPattern(str): + """A regex based str with a fuzzy __eq__. + + Use this helper with 'mock.assert_called_with', or anywhere + where a regexp comparison between strings is needed. + + For instance: + mock_call.assert_called_with(MockPattern('spam.*ham')) + """ + def __eq__(self, other): + return bool(re.search(str(self), other, re.S)) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -65,7 +65,7 @@ except ValueError as exc: raise RuntimeError(str(exc)) - handle = events.Handle(callback, args) + handle = events.Handle(callback, args, self) self._signal_handlers[sig] = handle try: @@ -294,7 +294,12 @@ def _fatal_error(self, exc): # should be called by exception handler only if not (isinstance(exc, OSError) and exc.errno == errno.EIO): - logger.exception('Fatal error for %s', self) + self._loop.call_exception_handler({ + 'message': 'Fatal transport error', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) self._close(exc) def _close(self, exc): @@ -441,7 +446,12 @@ def _fatal_error(self, exc): # should be called by exception handler only if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): - logger.exception('Fatal error for %s', self) + self._loop.call_exception_handler({ + 'message': 'Fatal transport error', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) self._close(exc) def _close(self, exc=None): @@ -582,8 +592,14 @@ def _sig_chld(self): try: self._do_waitpid_all() - except Exception: - logger.exception('Unknown exception in SIGCHLD handler') + except Exception as exc: + # self._loop should always be available here + # as '_sig_chld' is added as a signal handler + # in 'attach_loop' + self._loop.call_exception_handler({ + 'message': 'Unknown exception in SIGCHLD handler', + 'exception': exc, + }) def _compute_returncode(self, status): if os.WIFSIGNALED(status): diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -156,9 +156,13 @@ if pipe is None: return f = self._proactor.accept_pipe(pipe) - except OSError: + except OSError as exc: if pipe and pipe.fileno() != -1: - logger.exception('Pipe accept failed') + self.call_exception_handler({ + 'message': 'Pipe accept failed', + 'exception': exc, + 'pipe': pipe, + }) pipe.close() except futures.CancelledError: if pipe: diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -15,6 +15,10 @@ from asyncio import test_utils +MOCK_ANY = unittest.mock.ANY +PY34 = sys.version_info >= (3, 4) + + class BaseEventLoopTests(unittest.TestCase): def setUp(self): @@ -49,20 +53,21 @@ self.assertRaises(NotImplementedError, next, iter(gen)) def test__add_callback_handle(self): - h = asyncio.Handle(lambda: False, ()) + h = asyncio.Handle(lambda: False, (), self.loop) self.loop._add_callback(h) self.assertFalse(self.loop._scheduled) self.assertIn(h, self.loop._ready) def test__add_callback_timer(self): - h = asyncio.TimerHandle(time.monotonic()+10, lambda: False, ()) + h = asyncio.TimerHandle(time.monotonic()+10, lambda: False, (), + self.loop) self.loop._add_callback(h) self.assertIn(h, self.loop._scheduled) def test__add_callback_cancelled_handle(self): - h = asyncio.Handle(lambda: False, ()) + h = asyncio.Handle(lambda: False, (), self.loop) h.cancel() self.loop._add_callback(h) @@ -137,15 +142,15 @@ self.assertRaises( AssertionError, self.loop.run_in_executor, - None, asyncio.Handle(cb, ()), ('',)) + None, asyncio.Handle(cb, (), self.loop), ('',)) self.assertRaises( AssertionError, self.loop.run_in_executor, - None, asyncio.TimerHandle(10, cb, ())) + None, asyncio.TimerHandle(10, cb, (), self.loop)) def test_run_once_in_executor_cancelled(self): def cb(): pass - h = asyncio.Handle(cb, ()) + h = asyncio.Handle(cb, (), self.loop) h.cancel() f = self.loop.run_in_executor(None, h) @@ -156,7 +161,7 @@ def test_run_once_in_executor_plain(self): def cb(): pass - h = asyncio.Handle(cb, ()) + h = asyncio.Handle(cb, (), self.loop) f = asyncio.Future(loop=self.loop) executor = unittest.mock.Mock() executor.submit.return_value = f @@ -175,8 +180,10 @@ f.cancel() # Don't complain about abandoned Future. def test__run_once(self): - h1 = asyncio.TimerHandle(time.monotonic() + 5.0, lambda: True, ()) - h2 = asyncio.TimerHandle(time.monotonic() + 10.0, lambda: True, ()) + h1 = asyncio.TimerHandle(time.monotonic() + 5.0, lambda: True, (), + self.loop) + h2 = asyncio.TimerHandle(time.monotonic() + 10.0, lambda: True, (), + self.loop) h1.cancel() @@ -205,14 +212,15 @@ m_time.monotonic = monotonic self.loop._scheduled.append( - asyncio.TimerHandle(11.0, lambda: True, ())) + asyncio.TimerHandle(11.0, lambda: True, (), self.loop)) self.loop._process_events = unittest.mock.Mock() self.loop._run_once() self.assertEqual(logging.INFO, m_logger.log.call_args[0][0]) idx = -1 data = [10.0, 10.0, 10.3, 13.0] - self.loop._scheduled = [asyncio.TimerHandle(11.0, lambda: True, ())] + self.loop._scheduled = [asyncio.TimerHandle(11.0, lambda: True, (), + self.loop)] self.loop._run_once() self.assertEqual(logging.DEBUG, m_logger.log.call_args[0][0]) @@ -225,7 +233,8 @@ processed = True handle = loop.call_soon(lambda: True) - h = asyncio.TimerHandle(time.monotonic() - 1, cb, (self.loop,)) + h = asyncio.TimerHandle(time.monotonic() - 1, cb, (self.loop,), + self.loop) self.loop._process_events = unittest.mock.Mock() self.loop._scheduled.append(h) @@ -287,6 +296,163 @@ self.loop.run_until_complete, self.loop.subprocess_shell, asyncio.SubprocessProtocol, 'exit 0', bufsize=4096) + def test_default_exc_handler_callback(self): + self.loop._process_events = unittest.mock.Mock() + + def zero_error(fut): + fut.set_result(True) + 1/0 + + # Test call_soon (events.Handle) + with unittest.mock.patch('asyncio.base_events.logger') as log: + fut = asyncio.Future(loop=self.loop) + self.loop.call_soon(zero_error, fut) + fut.add_done_callback(lambda fut: self.loop.stop()) + self.loop.run_forever() + log.error.assert_called_with( + test_utils.MockPattern('Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + # Test call_later (events.TimerHandle) + with unittest.mock.patch('asyncio.base_events.logger') as log: + fut = asyncio.Future(loop=self.loop) + self.loop.call_later(0.01, zero_error, fut) + fut.add_done_callback(lambda fut: self.loop.stop()) + self.loop.run_forever() + log.error.assert_called_with( + test_utils.MockPattern('Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + def test_default_exc_handler_coro(self): + self.loop._process_events = unittest.mock.Mock() + + @asyncio.coroutine + def zero_error_coro(): + yield from asyncio.sleep(0.01, loop=self.loop) + 1/0 + + # Test Future.__del__ + with unittest.mock.patch('asyncio.base_events.logger') as log: + fut = asyncio.async(zero_error_coro(), loop=self.loop) + fut.add_done_callback(lambda *args: self.loop.stop()) + self.loop.run_forever() + fut = None # Trigger Future.__del__ or futures._TracebackLogger + if PY34: + # Future.__del__ in Python 3.4 logs error with + # an actual exception context + log.error.assert_called_with( + test_utils.MockPattern('.*exception was never retrieved'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + else: + # futures._TracebackLogger logs only textual traceback + log.error.assert_called_with( + test_utils.MockPattern( + '.*exception was never retrieved.*ZeroDiv'), + exc_info=False) + + def test_set_exc_handler_invalid(self): + with self.assertRaisesRegex(TypeError, 'A callable object or None'): + self.loop.set_exception_handler('spam') + + def test_set_exc_handler_custom(self): + def zero_error(): + 1/0 + + def run_loop(): + self.loop.call_soon(zero_error) + self.loop._run_once() + + self.loop._process_events = unittest.mock.Mock() + + mock_handler = unittest.mock.Mock() + self.loop.set_exception_handler(mock_handler) + run_loop() + mock_handler.assert_called_with(self.loop, { + 'exception': MOCK_ANY, + 'message': test_utils.MockPattern( + 'Exception in callback.*zero_error'), + 'handle': MOCK_ANY, + }) + mock_handler.reset_mock() + + self.loop.set_exception_handler(None) + with unittest.mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern( + 'Exception in callback.*zero'), + exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) + + assert not mock_handler.called + + def test_set_exc_handler_broken(self): + def run_loop(): + def zero_error(): + 1/0 + self.loop.call_soon(zero_error) + self.loop._run_once() + + def handler(loop, context): + raise AttributeError('spam') + + self.loop._process_events = unittest.mock.Mock() + + self.loop.set_exception_handler(handler) + + with unittest.mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern( + 'Unhandled error in exception handler'), + exc_info=(AttributeError, MOCK_ANY, MOCK_ANY)) + + def test_default_exc_handler_broken(self): + _context = None + + class Loop(base_events.BaseEventLoop): + + _selector = unittest.mock.Mock() + _process_events = unittest.mock.Mock() + + def default_exception_handler(self, context): + nonlocal _context + _context = context + # Simulates custom buggy "default_exception_handler" + raise ValueError('spam') + + loop = Loop() + asyncio.set_event_loop(loop) + + def run_loop(): + def zero_error(): + 1/0 + loop.call_soon(zero_error) + loop._run_once() + + with unittest.mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + 'Exception in default exception handler', + exc_info=True) + + def custom_handler(loop, context): + raise ValueError('ham') + + _context = None + loop.set_exception_handler(custom_handler) + with unittest.mock.patch('asyncio.base_events.logger') as log: + run_loop() + log.error.assert_called_with( + test_utils.MockPattern('Exception in default exception.*' + 'while handling.*in custom'), + exc_info=True) + + # Check that original context was passed to default + # exception handler. + self.assertIn('context', _context) + self.assertIs(type(_context['context']['exception']), + ZeroDivisionError) + class MyProto(asyncio.Protocol): done = None @@ -716,7 +882,7 @@ self.loop._accept_connection(MyProto, sock) self.assertFalse(sock.close.called) - @unittest.mock.patch('asyncio.selector_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_accept_connection_exception(self, m_log): sock = unittest.mock.Mock() sock.fileno.return_value = 10 @@ -725,7 +891,7 @@ self.loop.call_later = unittest.mock.Mock() self.loop._accept_connection(MyProto, sock) - self.assertTrue(m_log.exception.called) + self.assertTrue(m_log.error.called) self.assertFalse(sock.close.called) self.loop.remove_reader.assert_called_with(10) self.loop.call_later.assert_called_with(constants.ACCEPT_RETRY_DELAY, diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1788,7 +1788,7 @@ return args args = () - h = asyncio.Handle(callback, args) + h = asyncio.Handle(callback, args, unittest.mock.Mock()) self.assertIs(h._callback, callback) self.assertIs(h._args, args) self.assertFalse(h._cancelled) @@ -1808,28 +1808,37 @@ '.callback')) self.assertTrue(r.endswith('())'), r) - def test_handle(self): + def test_handle_from_handle(self): def callback(*args): return args - h1 = asyncio.Handle(callback, ()) + m_loop = object() + h1 = asyncio.Handle(callback, (), loop=m_loop) self.assertRaises( - AssertionError, asyncio.Handle, h1, ()) + AssertionError, asyncio.Handle, h1, (), m_loop) - @unittest.mock.patch('asyncio.events.logger') - def test_callback_with_exception(self, log): + def test_callback_with_exception(self): def callback(): raise ValueError() - h = asyncio.Handle(callback, ()) + m_loop = unittest.mock.Mock() + m_loop.call_exception_handler = unittest.mock.Mock() + + h = asyncio.Handle(callback, (), m_loop) h._run() - self.assertTrue(log.exception.called) + + m_loop.call_exception_handler.assert_called_with({ + 'message': test_utils.MockPattern('Exception in callback.*'), + 'exception': unittest.mock.ANY, + 'handle': h + }) class TimerTests(unittest.TestCase): def test_hash(self): when = time.monotonic() - h = asyncio.TimerHandle(when, lambda: False, ()) + h = asyncio.TimerHandle(when, lambda: False, (), + unittest.mock.Mock()) self.assertEqual(hash(h), hash(when)) def test_timer(self): @@ -1838,7 +1847,7 @@ args = () when = time.monotonic() - h = asyncio.TimerHandle(when, callback, args) + h = asyncio.TimerHandle(when, callback, args, unittest.mock.Mock()) self.assertIs(h._callback, callback) self.assertIs(h._args, args) self.assertFalse(h._cancelled) @@ -1853,16 +1862,19 @@ self.assertTrue(r.endswith('())'), r) self.assertRaises(AssertionError, - asyncio.TimerHandle, None, callback, args) + asyncio.TimerHandle, None, callback, args, + unittest.mock.Mock()) def test_timer_comparison(self): + loop = unittest.mock.Mock() + def callback(*args): return args when = time.monotonic() - h1 = asyncio.TimerHandle(when, callback, ()) - h2 = asyncio.TimerHandle(when, callback, ()) + h1 = asyncio.TimerHandle(when, callback, (), loop) + h2 = asyncio.TimerHandle(when, callback, (), loop) # TODO: Use assertLess etc. self.assertFalse(h1 < h2) self.assertFalse(h2 < h1) @@ -1878,8 +1890,8 @@ h2.cancel() self.assertFalse(h1 == h2) - h1 = asyncio.TimerHandle(when, callback, ()) - h2 = asyncio.TimerHandle(when + 10.0, callback, ()) + h1 = asyncio.TimerHandle(when, callback, (), loop) + h2 = asyncio.TimerHandle(when + 10.0, callback, (), loop) self.assertTrue(h1 < h2) self.assertFalse(h2 < h1) self.assertTrue(h1 <= h2) @@ -1891,7 +1903,7 @@ self.assertFalse(h1 == h2) self.assertTrue(h1 != h2) - h3 = asyncio.Handle(callback, ()) + h3 = asyncio.Handle(callback, (), loop) self.assertIs(NotImplemented, h1.__eq__(h3)) self.assertIs(NotImplemented, h1.__ne__(h3)) diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -174,20 +174,20 @@ self.assertRaises(AssertionError, test) fut.cancel() - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_abandoned(self, m_log): fut = asyncio.Future(loop=self.loop) del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_result_unretrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_result(42) del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_result_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_result(42) @@ -195,7 +195,7 @@ del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_unretrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) @@ -203,7 +203,7 @@ test_utils.run_briefly(self.loop) self.assertTrue(m_log.error.called) - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) @@ -211,7 +211,7 @@ del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.futures.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_result_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -207,13 +207,13 @@ test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) - @unittest.mock.patch('asyncio.proactor_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_fatal_error(self, m_logging): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._force_close = unittest.mock.Mock() tr._fatal_error(None) self.assertTrue(tr._force_close.called) - self.assertTrue(m_logging.exception.called) + self.assertTrue(m_logging.error.called) def test_force_close(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) @@ -432,7 +432,7 @@ def test_process_events(self): self.loop._process_events([]) - @unittest.mock.patch('asyncio.proactor_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_create_server(self, m_log): pf = unittest.mock.Mock() call_soon = self.loop.call_soon = unittest.mock.Mock() @@ -458,7 +458,7 @@ fut.result.side_effect = OSError() loop(fut) self.assertTrue(self.sock.close.called) - self.assertTrue(m_log.exception.called) + self.assertTrue(m_log.error.called) def test_create_server_cancel(self): pf = unittest.mock.Mock() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -23,6 +23,9 @@ from asyncio.selector_events import _SelectorDatagramTransport +MOCK_ANY = unittest.mock.ANY + + class TestBaseSelectorEventLoop(BaseSelectorEventLoop): def _make_self_pipe(self): @@ -643,14 +646,18 @@ self.assertFalse(self.loop.readers) self.assertEqual(1, self.loop.remove_reader_count[7]) - @unittest.mock.patch('asyncio.log.logger.exception') + @unittest.mock.patch('asyncio.log.logger.error') def test_fatal_error(self, m_exc): exc = OSError() tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) tr._force_close = unittest.mock.Mock() tr._fatal_error(exc) - m_exc.assert_called_with('Fatal error for %s', tr) + m_exc.assert_called_with( + test_utils.MockPattern( + 'Fatal transport error\nprotocol:.*\ntransport:.*'), + exc_info=(OSError, MOCK_ANY, MOCK_ANY)) + tr._force_close.assert_called_with(exc) def test_connection_lost(self): @@ -996,7 +1003,7 @@ transport._write_ready() transport._fatal_error.assert_called_with(err) - @unittest.mock.patch('asyncio.selector_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_write_ready_exception_and_close(self, m_log): self.sock.send.side_effect = OSError() remove_writer = self.loop.remove_writer = unittest.mock.Mock() @@ -1651,14 +1658,17 @@ self.assertFalse(transport._fatal_error.called) self.assertTrue(self.protocol.error_received.called) - @unittest.mock.patch('asyncio.log.logger.exception') + @unittest.mock.patch('asyncio.base_events.logger.error') def test_fatal_error_connected(self, m_exc): transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) err = ConnectionRefusedError() transport._fatal_error(err) self.assertFalse(self.protocol.error_received.called) - m_exc.assert_called_with('Fatal error for %s', transport) + m_exc.assert_called_with( + test_utils.MockPattern( + 'Fatal transport error\nprotocol:.*\ntransport:.*'), + exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -25,6 +25,9 @@ from asyncio import unix_events +MOCK_ANY = unittest.mock.ANY + + @unittest.skipUnless(signal, 'Signals are not supported') class SelectorEventLoopSignalTests(unittest.TestCase): @@ -45,7 +48,8 @@ self.loop._handle_signal(signal.NSIG + 1, ()) def test_handle_signal_cancelled_handler(self): - h = asyncio.Handle(unittest.mock.Mock(), ()) + h = asyncio.Handle(unittest.mock.Mock(), (), + loop=unittest.mock.Mock()) h.cancel() self.loop._signal_handlers[signal.NSIG + 1] = h self.loop.remove_signal_handler = unittest.mock.Mock() @@ -91,7 +95,7 @@ signal.SIGINT, lambda: True) @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.unix_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_add_signal_handler_install_error2(self, m_logging, m_signal): m_signal.NSIG = signal.NSIG @@ -108,7 +112,7 @@ self.assertEqual(1, m_signal.set_wakeup_fd.call_count) @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.unix_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_add_signal_handler_install_error3(self, m_logging, m_signal): class Err(OSError): errno = errno.EINVAL @@ -153,7 +157,7 @@ m_signal.signal.call_args[0]) @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.unix_events.logger') + @unittest.mock.patch('asyncio.base_events.logger') def test_remove_signal_handler_cleanup_error(self, m_logging, m_signal): m_signal.NSIG = signal.NSIG self.loop.add_signal_handler(signal.SIGHUP, lambda: True) @@ -347,7 +351,7 @@ test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.data_received.called) - @unittest.mock.patch('asyncio.log.logger.exception') + @unittest.mock.patch('asyncio.log.logger.error') @unittest.mock.patch('os.read') def test__read_ready_error(self, m_read, m_logexc): tr = unix_events._UnixReadPipeTransport( @@ -359,7 +363,10 @@ m_read.assert_called_with(5, tr.max_size) tr._close.assert_called_with(err) - m_logexc.assert_called_with('Fatal error for %s', tr) + m_logexc.assert_called_with( + test_utils.MockPattern( + 'Fatal transport error\nprotocol:.*\ntransport:.*'), + exc_info=(OSError, MOCK_ANY, MOCK_ANY)) @unittest.mock.patch('os.read') def test_pause_reading(self, m_read): @@ -423,7 +430,7 @@ self.assertEqual(2, sys.getrefcount(self.protocol), pprint.pformat(gc.get_referrers(self.protocol))) self.assertIsNone(tr._loop) - self.assertEqual(2, sys.getrefcount(self.loop), + self.assertEqual(4, sys.getrefcount(self.loop), pprint.pformat(gc.get_referrers(self.loop))) def test__call_connection_lost_with_err(self): @@ -436,10 +443,11 @@ self.pipe.close.assert_called_with() self.assertIsNone(tr._protocol) + self.assertEqual(2, sys.getrefcount(self.protocol), pprint.pformat(gc.get_referrers(self.protocol))) self.assertIsNone(tr._loop) - self.assertEqual(2, sys.getrefcount(self.loop), + self.assertEqual(4, sys.getrefcount(self.loop), pprint.pformat(gc.get_referrers(self.loop))) @@ -635,7 +643,7 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'data'], tr._buffer) - @unittest.mock.patch('asyncio.log.logger.exception') + @unittest.mock.patch('asyncio.log.logger.error') @unittest.mock.patch('os.write') def test__write_ready_err(self, m_write, m_logexc): tr = unix_events._UnixWritePipeTransport( @@ -650,7 +658,10 @@ self.assertFalse(self.loop.readers) self.assertEqual([], tr._buffer) self.assertTrue(tr._closing) - m_logexc.assert_called_with('Fatal error for %s', tr) + m_logexc.assert_called_with( + test_utils.MockPattern( + 'Fatal transport error\nprotocol:.*\ntransport:.*'), + exc_info=(OSError, MOCK_ANY, MOCK_ANY)) self.assertEqual(1, tr._conn_lost) test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(err) @@ -702,7 +713,7 @@ self.assertEqual(2, sys.getrefcount(self.protocol), pprint.pformat(gc.get_referrers(self.protocol))) self.assertIsNone(tr._loop) - self.assertEqual(2, sys.getrefcount(self.loop), + self.assertEqual(4, sys.getrefcount(self.loop), pprint.pformat(gc.get_referrers(self.loop))) def test__call_connection_lost_with_err(self): @@ -718,7 +729,7 @@ self.assertEqual(2, sys.getrefcount(self.protocol), pprint.pformat(gc.get_referrers(self.protocol))) self.assertIsNone(tr._loop) - self.assertEqual(2, sys.getrefcount(self.loop), + self.assertEqual(4, sys.getrefcount(self.loop), pprint.pformat(gc.get_referrers(self.loop))) def test_close(self): @@ -1285,10 +1296,10 @@ m.waitpid.side_effect = ValueError with unittest.mock.patch.object(log.logger, - "exception") as m_exception: + 'error') as m_error: self.assertEqual(self.watcher._sig_chld(), None) - self.assertTrue(m_exception.called) + self.assertTrue(m_error.called) @waitpid_mocks def test_sigchld_child_reaped_elsewhere(self, m): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,10 @@ Library ------- +- Issue #20681: Add new error handling API in asyncio. New APIs: + loop.set_exception_handler(), loop.default_exception_handler(), and + loop.call_exception_handler(). + - Issue #20673: Implement support for UNIX Domain Sockets in asyncio. New APIs: loop.create_unix_connection(), loop.create_unix_server(), streams.open_unix_connection(), and streams.start_unix_server(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:07 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby50cmFu?= =?utf-8?q?sports=3A_Make_=5FProactorBasePipeTransport_use_=5FFlowControlM?= =?utf-8?q?ixin?= Message-ID: <3fnSNR35m9z7LkJ@mail.python.org> http://hg.python.org/cpython/rev/7c93d2ec34f6 changeset: 89730:7c93d2ec34f6 branch: 3.4 user: Yury Selivanov date: Tue Feb 18 18:41:13 2014 -0500 summary: asyncio.transports: Make _ProactorBasePipeTransport use _FlowControlMixin files: Lib/asyncio/proactor_events.py | 55 +--------------- Lib/asyncio/selector_events.py | 73 +--------------------- Lib/asyncio/transports.py | 70 +++++++++++++++++++++ Lib/asyncio/unix_events.py | 2 +- 4 files changed, 75 insertions(+), 125 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -15,7 +15,8 @@ from .log import logger -class _ProactorBasePipeTransport(transports.BaseTransport): +class _ProactorBasePipeTransport(transports._FlowControlMixin, + transports.BaseTransport): """Base class for pipe and socket transports.""" def __init__(self, loop, sock, protocol, waiter=None, @@ -33,8 +34,6 @@ self._conn_lost = 0 self._closing = False # Set when close() called. self._eof_written = False - self._protocol_paused = False - self.set_write_buffer_limits() if self._server is not None: self._server.attach(self) self._loop.call_soon(self._protocol.connection_made, self) @@ -94,56 +93,6 @@ server.detach(self) self._server = None - # XXX The next four methods are nearly identical to corresponding - # ones in _SelectorTransport. Maybe refactor buffer management to - # share the implementations? (Also these are really only needed - # by _ProactorWritePipeTransport but since _buffer is defined on - # the base class I am putting it here for now.) - - def _maybe_pause_protocol(self): - size = self.get_write_buffer_size() - if size <= self._high_water: - return - if not self._protocol_paused: - self._protocol_paused = True - try: - self._protocol.pause_writing() - except Exception as exc: - self._loop.call_exception_handler({ - 'message': 'protocol.pause_writing() failed', - 'exception': exc, - 'transport': self, - 'protocol': self._protocol, - }) - - def _maybe_resume_protocol(self): - if (self._protocol_paused and - self.get_write_buffer_size() <= self._low_water): - self._protocol_paused = False - try: - self._protocol.resume_writing() - except Exception as exc: - self._loop.call_exception_handler({ - 'message': 'protocol.resume_writing() failed', - 'exception': exc, - 'transport': self, - 'protocol': self._protocol, - }) - - def set_write_buffer_limits(self, high=None, low=None): - if high is None: - if low is None: - high = 64*1024 - else: - high = 4*low - if low is None: - low = high // 4 - if not high >= low >= 0: - raise ValueError('high (%r) must be >= low (%r) must be >= 0' % - (high, low)) - self._high_water = high - self._low_water = low - def get_write_buffer_size(self): size = self._pending_write if self._buffer is not None: diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -338,77 +338,8 @@ sock.close() -class _FlowControlMixin(transports.Transport): - """All the logic for (write) flow control in a mix-in base class. - - The subclass must implement get_write_buffer_size(). It must call - _maybe_pause_protocol() whenever the write buffer size increases, - and _maybe_resume_protocol() whenever it decreases. It may also - override set_write_buffer_limits() (e.g. to specify different - defaults). - - The subclass constructor must call super().__init__(extra). This - will call set_write_buffer_limits(). - - The user may call set_write_buffer_limits() and - get_write_buffer_size(), and their protocol's pause_writing() and - resume_writing() may be called. - """ - - def __init__(self, extra=None): - super().__init__(extra) - self._protocol_paused = False - self.set_write_buffer_limits() - - def _maybe_pause_protocol(self): - size = self.get_write_buffer_size() - if size <= self._high_water: - return - if not self._protocol_paused: - self._protocol_paused = True - try: - self._protocol.pause_writing() - except Exception as exc: - self._loop.call_exception_handler({ - 'message': 'protocol.pause_writing() failed', - 'exception': exc, - 'transport': self, - 'protocol': self._protocol, - }) - - def _maybe_resume_protocol(self): - if (self._protocol_paused and - self.get_write_buffer_size() <= self._low_water): - self._protocol_paused = False - try: - self._protocol.resume_writing() - except Exception as exc: - self._loop.call_exception_handler({ - 'message': 'protocol.resume_writing() failed', - 'exception': exc, - 'transport': self, - 'protocol': self._protocol, - }) - - def set_write_buffer_limits(self, high=None, low=None): - if high is None: - if low is None: - high = 64*1024 - else: - high = 4*low - if low is None: - low = high // 4 - if not high >= low >= 0: - raise ValueError('high (%r) must be >= low (%r) must be >= 0' % - (high, low)) - self._high_water = high - self._low_water = low - - def get_write_buffer_size(self): - raise NotImplementedError - - -class _SelectorTransport(_FlowControlMixin, transports.Transport): +class _SelectorTransport(transports._FlowControlMixin, + transports.Transport): max_size = 256 * 1024 # Buffer size passed to recv(). diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -219,3 +219,73 @@ http://docs.python.org/3/library/subprocess#subprocess.Popen.kill """ raise NotImplementedError + + +class _FlowControlMixin(Transport): + """All the logic for (write) flow control in a mix-in base class. + + The subclass must implement get_write_buffer_size(). It must call + _maybe_pause_protocol() whenever the write buffer size increases, + and _maybe_resume_protocol() whenever it decreases. It may also + override set_write_buffer_limits() (e.g. to specify different + defaults). + + The subclass constructor must call super().__init__(extra). This + will call set_write_buffer_limits(). + + The user may call set_write_buffer_limits() and + get_write_buffer_size(), and their protocol's pause_writing() and + resume_writing() may be called. + """ + + def __init__(self, extra=None): + super().__init__(extra) + self._protocol_paused = False + self.set_write_buffer_limits() + + def _maybe_pause_protocol(self): + size = self.get_write_buffer_size() + if size <= self._high_water: + return + if not self._protocol_paused: + self._protocol_paused = True + try: + self._protocol.pause_writing() + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.pause_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) + + def _maybe_resume_protocol(self): + if (self._protocol_paused and + self.get_write_buffer_size() <= self._low_water): + self._protocol_paused = False + try: + self._protocol.resume_writing() + except Exception as exc: + self._loop.call_exception_handler({ + 'message': 'protocol.resume_writing() failed', + 'exception': exc, + 'transport': self, + 'protocol': self._protocol, + }) + + def set_write_buffer_limits(self, high=None, low=None): + if high is None: + if low is None: + high = 64*1024 + else: + high = 4*low + if low is None: + low = high // 4 + if not high >= low >= 0: + raise ValueError('high (%r) must be >= low (%r) must be >= 0' % + (high, low)) + self._high_water = high + self._low_water = low + + def get_write_buffer_size(self): + raise NotImplementedError diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -317,7 +317,7 @@ self._loop = None -class _UnixWritePipeTransport(selector_events._FlowControlMixin, +class _UnixWritePipeTransport(transports._FlowControlMixin, transports.WriteTransport): def __init__(self, loop, pipe, protocol, waiter=None, extra=None): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:09 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_139=3A_Improve_error_messages_on_=22fatal_errors=22?= Message-ID: <3fnSNT094nz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/e93a36a678da changeset: 89731:e93a36a678da branch: 3.4 user: Victor Stinner date: Wed Feb 19 01:40:41 2014 +0100 summary: asyncio, Tulip issue 139: Improve error messages on "fatal errors" Mention if the error was caused by a read or a write, and be more specific on the object (ex: "pipe transport" instead of "transport"). files: Lib/asyncio/proactor_events.py | 10 +- Lib/asyncio/selector_events.py | 22 +++-- Lib/asyncio/unix_events.py | 14 +- Lib/test/test_asyncio/test_proactor_events.py | 12 ++- Lib/test/test_asyncio/test_selector_events.py | 36 +++++++-- Lib/test/test_asyncio/test_unix_events.py | 8 +- 6 files changed, 64 insertions(+), 38 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -53,10 +53,10 @@ if self._read_fut is not None: self._read_fut.cancel() - def _fatal_error(self, exc): + def _fatal_error(self, exc, message='Fatal error on pipe transport'): if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): self._loop.call_exception_handler({ - 'message': 'Fatal transport error', + 'message': message, 'exception': exc, 'transport': self, 'protocol': self._protocol, @@ -151,11 +151,11 @@ self._read_fut = self._loop._proactor.recv(self._sock, 4096) except ConnectionAbortedError as exc: if not self._closing: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on pipe transport') except ConnectionResetError as exc: self._force_close(exc) except OSError as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on pipe transport') except futures.CancelledError: if not self._closing: raise @@ -246,7 +246,7 @@ except ConnectionResetError as exc: self._force_close(exc) except OSError as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on pipe transport') def can_write_eof(self): return True diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -377,11 +377,11 @@ self._conn_lost += 1 self._loop.call_soon(self._call_connection_lost, None) - def _fatal_error(self, exc): + def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): self._loop.call_exception_handler({ - 'message': 'Fatal transport error', + 'message': message, 'exception': exc, 'transport': self, 'protocol': self._protocol, @@ -452,7 +452,7 @@ except (BlockingIOError, InterruptedError): pass except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on socket transport') else: if data: self._protocol.data_received(data) @@ -488,7 +488,7 @@ except (BlockingIOError, InterruptedError): pass except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on socket transport') return else: data = data[n:] @@ -511,7 +511,7 @@ except Exception as exc: self._loop.remove_writer(self._sock_fd) self._buffer.clear() - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on socket transport') else: if n: del self._buffer[:n] @@ -678,7 +678,7 @@ self._loop.remove_reader(self._sock_fd) self._loop.add_writer(self._sock_fd, self._write_ready) except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on SSL transport') else: if data: self._protocol.data_received(data) @@ -712,7 +712,7 @@ except Exception as exc: self._loop.remove_writer(self._sock_fd) self._buffer.clear() - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on SSL transport') return if n: @@ -770,7 +770,7 @@ except OSError as exc: self._protocol.error_received(exc) except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on datagram transport') else: self._protocol.datagram_received(data, addr) @@ -805,7 +805,8 @@ self._protocol.error_received(exc) return except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, + 'Fatal write error on datagram transport') return # Ensure that what we buffer is immutable. @@ -827,7 +828,8 @@ self._protocol.error_received(exc) return except Exception as exc: - self._fatal_error(exc) + self._fatal_error(exc, + 'Fatal write error on datagram transport') return self._maybe_resume_protocol() # May append to buffer. diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -271,7 +271,7 @@ except (BlockingIOError, InterruptedError): pass except OSError as exc: - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal read error on pipe transport') else: if data: self._protocol.data_received(data) @@ -291,11 +291,11 @@ if not self._closing: self._close(None) - def _fatal_error(self, exc): + def _fatal_error(self, exc, message='Fatal error on pipe transport'): # should be called by exception handler only if not (isinstance(exc, OSError) and exc.errno == errno.EIO): self._loop.call_exception_handler({ - 'message': 'Fatal transport error', + 'message': message, 'exception': exc, 'transport': self, 'protocol': self._protocol, @@ -381,7 +381,7 @@ n = 0 except Exception as exc: self._conn_lost += 1 - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on pipe transport') return if n == len(data): return @@ -406,7 +406,7 @@ # Remove writer here, _fatal_error() doesn't it # because _buffer is empty. self._loop.remove_writer(self._fileno) - self._fatal_error(exc) + self._fatal_error(exc, 'Fatal write error on pipe transport') else: if n == len(data): self._loop.remove_writer(self._fileno) @@ -443,11 +443,11 @@ def abort(self): self._close(None) - def _fatal_error(self, exc): + def _fatal_error(self, exc, message='Fatal error on pipe transport'): # should be called by exception handler only if not isinstance(exc, (BrokenPipeError, ConnectionResetError)): self._loop.call_exception_handler({ - 'message': 'Fatal transport error', + 'message': message, 'exception': exc, 'transport': self, 'protocol': self._protocol, diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -69,7 +69,9 @@ tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._fatal_error = unittest.mock.Mock() tr._loop_reading() - tr._fatal_error.assert_called_with(err) + tr._fatal_error.assert_called_with( + err, + 'Fatal read error on pipe transport') def test_loop_reading_aborted_closing(self): self.loop._proactor.recv.side_effect = ConnectionAbortedError() @@ -105,7 +107,9 @@ tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._fatal_error = unittest.mock.Mock() tr._loop_reading() - tr._fatal_error.assert_called_with(err) + tr._fatal_error.assert_called_with( + err, + 'Fatal read error on pipe transport') def test_write(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) @@ -142,7 +146,9 @@ tr._fatal_error = unittest.mock.Mock() tr._buffer = [b'da', b'ta'] tr._loop_writing() - tr._fatal_error.assert_called_with(err) + tr._fatal_error.assert_called_with( + err, + 'Fatal write error on pipe transport') tr._conn_lost = 1 tr.write(b'data') diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -655,7 +655,7 @@ m_exc.assert_called_with( test_utils.MockPattern( - 'Fatal transport error\nprotocol:.*\ntransport:.*'), + 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) tr._force_close.assert_called_with(exc) @@ -785,7 +785,9 @@ transport._fatal_error = unittest.mock.Mock() transport._read_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on socket transport') def test_write(self): data = b'data' @@ -898,7 +900,9 @@ self.loop, self.sock, self.protocol) transport._fatal_error = unittest.mock.Mock() transport.write(data) - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on socket transport') transport._conn_lost = 1 self.sock.reset_mock() @@ -1001,7 +1005,9 @@ transport._fatal_error = unittest.mock.Mock() transport._buffer.extend(b'data') transport._write_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on socket transport') @unittest.mock.patch('asyncio.base_events.logger') def test_write_ready_exception_and_close(self, m_log): @@ -1237,7 +1243,9 @@ transport = self._make_one() transport._fatal_error = unittest.mock.Mock() transport._read_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on SSL transport') def test_write_ready_send(self): self.sslsock.send.return_value = 4 @@ -1319,7 +1327,9 @@ transport._buffer = list_to_buffer([b'data']) transport._fatal_error = unittest.mock.Mock() transport._write_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on SSL transport') self.assertEqual(list_to_buffer(), transport._buffer) def test_write_ready_read_wants_write(self): @@ -1407,7 +1417,9 @@ transport._fatal_error = unittest.mock.Mock() transport._read_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal read error on datagram transport') def test_read_ready_oserr(self): transport = _SelectorDatagramTransport( @@ -1517,7 +1529,9 @@ transport.sendto(data, ()) self.assertTrue(transport._fatal_error.called) - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') transport._conn_lost = 1 transport._address = ('123',) @@ -1633,7 +1647,9 @@ transport._buffer.append((b'data', ())) transport._sendto_ready() - transport._fatal_error.assert_called_with(err) + transport._fatal_error.assert_called_with( + err, + 'Fatal write error on datagram transport') def test_sendto_ready_error_received(self): self.sock.sendto.side_effect = ConnectionRefusedError @@ -1667,7 +1683,7 @@ self.assertFalse(self.protocol.error_received.called) m_exc.assert_called_with( test_utils.MockPattern( - 'Fatal transport error\nprotocol:.*\ntransport:.*'), + 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -365,7 +365,7 @@ tr._close.assert_called_with(err) m_logexc.assert_called_with( test_utils.MockPattern( - 'Fatal transport error\nprotocol:.*\ntransport:.*'), + 'Fatal read error on pipe transport\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) @unittest.mock.patch('os.read') @@ -558,7 +558,9 @@ m_write.assert_called_with(5, b'data') self.assertFalse(self.loop.writers) self.assertEqual([], tr._buffer) - tr._fatal_error.assert_called_with(err) + tr._fatal_error.assert_called_with( + err, + 'Fatal write error on pipe transport') self.assertEqual(1, tr._conn_lost) tr.write(b'data') @@ -660,7 +662,7 @@ self.assertTrue(tr._closing) m_logexc.assert_called_with( test_utils.MockPattern( - 'Fatal transport error\nprotocol:.*\ntransport:.*'), + 'Fatal write error on pipe transport\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) self.assertEqual(1, tr._conn_lost) test_utils.run_briefly(self.loop) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:10 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_143=3A_UNIX_domain_methods=2C_fix_ResourceWarning_and?= Message-ID: <3fnSNV2dhkz7LkB@mail.python.org> http://hg.python.org/cpython/rev/8ac1e30cddfb changeset: 89732:8ac1e30cddfb branch: 3.4 user: Victor Stinner date: Wed Feb 19 01:45:59 2014 +0100 summary: asyncio, Tulip issue 143: UNIX domain methods, fix ResourceWarning and DeprecationWarning warnings. create_unix_server() closes the socket on any error, not only on OSError. files: Lib/asyncio/unix_events.py | 8 +- Lib/test/test_asyncio/test_unix_events.py | 28 +++++----- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -183,13 +183,12 @@ raise ValueError( 'path and sock can not be specified at the same time') + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) try: - sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) sock.setblocking(False) yield from self.sock_connect(sock, path) - except OSError: - if sock is not None: - sock.close() + except: + sock.close() raise else: @@ -213,6 +212,7 @@ try: sock.bind(path) except OSError as exc: + sock.close() if exc.errno == errno.EADDRINUSE: # Let's improve the error message by adding # with what exact address it occurs. diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -221,17 +221,17 @@ with test_utils.unix_socket_path() as path: sock = socket.socket(socket.AF_UNIX) sock.bind(path) - - coro = self.loop.create_unix_server(lambda: None, path) - with self.assertRaisesRegexp(OSError, - 'Address.*is already in use'): - self.loop.run_until_complete(coro) + with sock: + coro = self.loop.create_unix_server(lambda: None, path) + with self.assertRaisesRegex(OSError, + 'Address.*is already in use'): + self.loop.run_until_complete(coro) def test_create_unix_server_existing_path_nonsock(self): with tempfile.NamedTemporaryFile() as file: coro = self.loop.create_unix_server(lambda: None, file.name) - with self.assertRaisesRegexp(OSError, - 'Address.*is already in use'): + with self.assertRaisesRegex(OSError, + 'Address.*is already in use'): self.loop.run_until_complete(coro) def test_create_unix_server_ssl_bool(self): @@ -248,11 +248,13 @@ self.loop.run_until_complete(coro) def test_create_unix_server_path_inetsock(self): - coro = self.loop.create_unix_server(lambda: None, path=None, - sock=socket.socket()) - with self.assertRaisesRegex(ValueError, - 'A UNIX Domain Socket was expected'): - self.loop.run_until_complete(coro) + sock = socket.socket() + with sock: + coro = self.loop.create_unix_server(lambda: None, path=None, + sock=sock) + with self.assertRaisesRegex(ValueError, + 'A UNIX Domain Socket was expected'): + self.loop.run_until_complete(coro) def test_create_unix_connection_path_sock(self): coro = self.loop.create_unix_connection( @@ -278,7 +280,7 @@ coro = self.loop.create_unix_connection( lambda: None, '/dev/null', ssl=True) - with self.assertRaisesRegexp( + with self.assertRaisesRegex( ValueError, 'you have to pass server_hostname when using ssl'): self.loop.run_until_complete(coro) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:11 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjgy?= =?utf-8?q?=3A_test=5Fasyncio=2C_=5Fbasetest=5Fcreate=5Fconnection=28=29_c?= =?utf-8?q?hecks_also_the?= Message-ID: <3fnSNW40Jmz7LkN@mail.python.org> http://hg.python.org/cpython/rev/9aac931d7bf5 changeset: 89733:9aac931d7bf5 branch: 3.4 user: Victor Stinner date: Wed Feb 19 02:21:08 2014 +0100 summary: Issue #20682: test_asyncio, _basetest_create_connection() checks also the sockname, as _basetest_create_ssl_connection(). files: Lib/test/test_asyncio/test_events.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -503,6 +503,7 @@ tr, pr = self.loop.run_until_complete(connection_fut) self.assertIsInstance(tr, asyncio.Transport) self.assertIsInstance(pr, asyncio.Protocol) + self.assertIsNotNone(tr.get_extra_info('sockname')) self.loop.run_until_complete(pr.done) self.assertGreater(pr.nbytes, 0) tr.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:12 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_spelling_and_typos=2E?= Message-ID: <3fnSNX6XbPz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/590d6b59996c changeset: 89734:590d6b59996c branch: 3.4 user: Yury Selivanov date: Tue Feb 18 22:27:48 2014 -0500 summary: asyncio: Fix spelling and typos. Thanks to Vajrasky Kok for discovering some of them. files: Lib/asyncio/events.py | 6 +++--- Lib/asyncio/protocols.py | 2 +- Lib/asyncio/selector_events.py | 2 +- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/test_utils.py | 4 ++-- Lib/asyncio/unix_events.py | 4 ++-- Lib/asyncio/windows_events.py | 2 +- Lib/selectors.py | 2 +- Lib/test/test_asyncio/test_base_events.py | 2 +- Lib/test/test_asyncio/test_events.py | 1 - Lib/test/test_asyncio/test_futures.py | 2 +- Lib/test/test_asyncio/test_streams.py | 2 +- Lib/test/test_asyncio/test_unix_events.py | 6 +++--- 13 files changed, 18 insertions(+), 19 deletions(-) diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -235,7 +235,7 @@ sock=None, backlog=100, ssl=None): """A coroutine which creates a UNIX Domain Socket server. - The return valud is a Server object, which can be used to stop + The return value is a Server object, which can be used to stop the service. path is a str, representing a file systsem path to bind the @@ -260,7 +260,7 @@ # Pipes and subprocesses. def connect_read_pipe(self, protocol_factory, pipe): - """Register read pipe in eventloop. + """Register read pipe in event loop. protocol_factory should instantiate object with Protocol interface. pipe is file-like object already switched to nonblocking. @@ -273,7 +273,7 @@ raise NotImplementedError def connect_write_pipe(self, protocol_factory, pipe): - """Register write pipe in eventloop. + """Register write pipe in event loop. protocol_factory should instantiate object with BaseProtocol interface. Pipe is file-like object already switched to nonblocking. diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -114,7 +114,7 @@ def pipe_data_received(self, fd, data): """Called when the subprocess writes data into stdout/stderr pipe. - fd is int file dascriptor. + fd is int file descriptor. data is bytes object. """ diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -213,7 +213,7 @@ def _sock_recv(self, fut, registered, sock, n): # _sock_recv() can add itself as an I/O callback if the operation can't - # be done immediatly. Don't use it directly, call sock_recv(). + # be done immediately. Don't use it directly, call sock_recv(). fd = sock.fileno() if registered: # Remove the callback early. It should be rare that the diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -181,7 +181,7 @@ The frames are always ordered from oldest to newest. - The optional limit gives the maximum nummber of frames to + The optional limit gives the maximum number of frames to return; by default all available frames are returned. Its meaning differs depending on whether a stack or a traceback is returned: the newest frames of a stack are returned, but the diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -259,7 +259,7 @@ when = yield ... ... = yield time_advance - Value retuned by yield is absolute time of next scheduled handler. + Value returned by yield is absolute time of next scheduled handler. Value passed to yield is time advance to move loop's time forward. """ @@ -369,7 +369,7 @@ """A regex based str with a fuzzy __eq__. Use this helper with 'mock.assert_called_with', or anywhere - where a regexp comparison between strings is needed. + where a regex comparison between strings is needed. For instance: mock_call.assert_called_with(MockPattern('spam.*ham')) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1,4 +1,4 @@ -"""Selector eventloop for Unix with signal handling.""" +"""Selector event loop for Unix with signal handling.""" import errno import fcntl @@ -244,7 +244,7 @@ class _UnixReadPipeTransport(transports.ReadTransport): - max_size = 256 * 1024 # max bytes we read in one eventloop iteration + max_size = 256 * 1024 # max bytes we read in one event loop iteration def __init__(self, loop, pipe, protocol, waiter=None, extra=None): super().__init__(extra) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -1,4 +1,4 @@ -"""Selector and proactor eventloops for Windows.""" +"""Selector and proactor event loops for Windows.""" import _winapi import errno diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -80,7 +80,7 @@ A selector can use various implementations (select(), poll(), epoll()...) depending on the platform. The default `Selector` class uses the most - performant implementation on the current platform. + efficient implementation on the current platform. """ @abstractmethod diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -277,7 +277,7 @@ asyncio.SubprocessProtocol, *args, bufsize=4096) def test_subprocess_shell_invalid_args(self): - # exepected a string, not an int or a list + # expected a string, not an int or a list self.assertRaises(TypeError, self.loop.run_until_complete, self.loop.subprocess_shell, asyncio.SubprocessProtocol, 123) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -503,7 +503,6 @@ tr, pr = self.loop.run_until_complete(connection_fut) self.assertIsInstance(tr, asyncio.Transport) self.assertIsInstance(pr, asyncio.Protocol) - self.assertIsNotNone(tr.get_extra_info('sockname')) self.loop.run_until_complete(pr.done) self.assertGreater(pr.nbytes, 0) tr.close() diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -38,7 +38,7 @@ asyncio.set_event_loop(None) def test_constructor_positional(self): - # Make sure Future does't accept a positional argument + # Make sure Future doesn't accept a positional argument self.assertRaises(TypeError, asyncio.Future, 42) def test_cancel(self): diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -239,7 +239,7 @@ # No b'\n' at the end. The 'limit' is set to 3. So before # waiting for the new data in buffer, 'readline' will consume # the entire buffer, and since the length of the consumed data - # is more than 3, it will raise a ValudError. The buffer is + # is more than 3, it will raise a ValueError. The buffer is # expected to be empty now. self.assertEqual(b'', stream._buffer) diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -965,7 +965,7 @@ self.assertFalse(m.WEXITSTATUS.called) self.assertFalse(m.WTERMSIG.called) - # childen are running + # children are running self.watcher._sig_chld() self.assertFalse(callback1.called) @@ -1069,7 +1069,7 @@ self.assertFalse(m.WEXITSTATUS.called) self.assertFalse(m.WTERMSIG.called) - # childen are running + # children are running self.watcher._sig_chld() self.assertFalse(callback1.called) @@ -1425,7 +1425,7 @@ self.add_zombie(61, 11) self.add_zombie(62, -5) - # SIGCHLD was not catched + # SIGCHLD was not caught self.assertFalse(callback1.called) self.assertFalse(callback2.called) self.assertFalse(callback3.called) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:14 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogcGVw?= =?utf-8?q?8-ify_the_code=2E?= Message-ID: <3fnSNZ21PSz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/6c5f3e12e318 changeset: 89735:6c5f3e12e318 branch: 3.4 user: Yury Selivanov date: Tue Feb 18 22:56:15 2014 -0500 summary: asyncio: pep8-ify the code. files: Lib/asyncio/base_events.py | 11 +++++---- Lib/asyncio/subprocess.py | 5 ++- Lib/test/test_asyncio/test_subprocess.py | 12 +++++++++- Lib/test/test_asyncio/test_tasks.py | 4 ++- Lib/test/test_asyncio/test_unix_events.py | 6 +++- 5 files changed, 26 insertions(+), 12 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -605,10 +605,10 @@ return transport, protocol @tasks.coroutine - def subprocess_exec(self, protocol_factory, program, *args, stdin=subprocess.PIPE, - stdout=subprocess.PIPE, stderr=subprocess.PIPE, - universal_newlines=False, shell=False, bufsize=0, - **kwargs): + def subprocess_exec(self, protocol_factory, program, *args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, universal_newlines=False, + shell=False, bufsize=0, **kwargs): if universal_newlines: raise ValueError("universal_newlines must be False") if shell: @@ -623,7 +623,8 @@ % type(arg).__name__) protocol = protocol_factory() transport = yield from self._make_subprocess_transport( - protocol, popen_args, False, stdin, stdout, stderr, bufsize, **kwargs) + protocol, popen_args, False, stdin, stdout, stderr, + bufsize, **kwargs) return transport, protocol def set_exception_handler(self, handler): diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -180,8 +180,9 @@ return Process(transport, protocol, loop) @tasks.coroutine -def create_subprocess_exec(program, *args, stdin=None, stdout=None, stderr=None, - loop=None, limit=streams._DEFAULT_LIMIT, **kwds): +def create_subprocess_exec(program, *args, stdin=None, stdout=None, + stderr=None, loop=None, + limit=streams._DEFAULT_LIMIT, **kwds): if loop is None: loop = events.get_event_loop() protocol_factory = lambda: SubprocessStreamProtocol(limit=limit, diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -21,6 +21,7 @@ 'sys.stdout.buffer.write(data)'))] class SubprocessMixin: + def test_stdin_stdout(self): args = PROGRAM_CAT @@ -132,6 +133,7 @@ if sys.platform != 'win32': # Unix class SubprocessWatcherMixin(SubprocessMixin): + Watcher = None def setUp(self): @@ -151,14 +153,20 @@ self.loop.close() policy.set_event_loop(None) - class SubprocessSafeWatcherTests(SubprocessWatcherMixin, unittest.TestCase): + class SubprocessSafeWatcherTests(SubprocessWatcherMixin, + unittest.TestCase): + Watcher = unix_events.SafeChildWatcher - class SubprocessFastWatcherTests(SubprocessWatcherMixin, unittest.TestCase): + class SubprocessFastWatcherTests(SubprocessWatcherMixin, + unittest.TestCase): + Watcher = unix_events.FastChildWatcher + else: # Windows class SubprocessProactorTests(SubprocessMixin, unittest.TestCase): + def setUp(self): policy = asyncio.get_event_loop_policy() self.loop = asyncio.ProactorEventLoop() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -876,6 +876,7 @@ self.assertEqual(set(f.result() for f in done), {'a', 'b'}) def test_as_completed_duplicate_coroutines(self): + @asyncio.coroutine def coro(s): return s @@ -884,7 +885,8 @@ def runner(): result = [] c = coro('ham') - for f in asyncio.as_completed([c, c, coro('spam')], loop=self.loop): + for f in asyncio.as_completed([c, c, coro('spam')], + loop=self.loop): result.append((yield from f)) return result diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -367,7 +367,8 @@ tr._close.assert_called_with(err) m_logexc.assert_called_with( test_utils.MockPattern( - 'Fatal read error on pipe transport\nprotocol:.*\ntransport:.*'), + 'Fatal read error on pipe transport' + '\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) @unittest.mock.patch('os.read') @@ -664,7 +665,8 @@ self.assertTrue(tr._closing) m_logexc.assert_called_with( test_utils.MockPattern( - 'Fatal write error on pipe transport\nprotocol:.*\ntransport:.*'), + 'Fatal write error on pipe transport' + '\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) self.assertEqual(1, tr._conn_lost) test_utils.run_briefly(self.loop) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:15 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogZG9j?= =?utf-8?q?ument_new_create=5Funix=5Fconnection=28=29_and_create=5Funix=5F?= =?utf-8?q?server=28=29_methods?= Message-ID: <3fnSNb3s5Mz7LkX@mail.python.org> http://hg.python.org/cpython/rev/0a8a5d435ecf changeset: 89736:0a8a5d435ecf branch: 3.4 user: Victor Stinner date: Wed Feb 19 13:32:34 2014 +0100 summary: asyncio: document new create_unix_connection() and create_unix_server() methods of BaseEventLoop files: Doc/library/asyncio-eventloop.rst | 43 ++++++++++++++++-- 1 files changed, 38 insertions(+), 5 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -201,8 +201,10 @@ .. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None) Create a streaming transport connection to a given Internet *host* and - *port*. *protocol_factory* must be a callable returning a - :ref:`protocol ` instance. + *port*: socket family :py:data:`~socket.AF_INET` or + :py:data:`~socket.AF_INET6` depending on *host* (or *family* if specified), + socket type :py:data:`~socket.SOCK_STREAM`. *protocol_factory* must be a + callable returning a :ref:`protocol ` instance. This method returns a :ref:`coroutine object ` which will try to establish the connection in the background. When successful, the @@ -265,6 +267,35 @@ (:class:`StreamReader`, :class:`StreamWriter`) instead of a protocol. +.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) + + Create datagram connection: socket family :py:data:`~socket.AF_INET` or + :py:data:`~socket.AF_INET6` depending on *host* (or *family* if specified), + socket type :py:data:`~socket.SOCK_DGRAM`. + + This method returns a :ref:`coroutine object ` which will try to + establish the connection in the background. When successful, the + coroutine returns a ``(transport, protocol)`` pair. + + See the :meth:`BaseEventLoop.create_connection` method for parameters. + + +.. method:: BaseEventLoop.create_unix_connection(protocol_factory, path, \*, ssl=None, sock=None, server_hostname=None) + + Create UNIX connection: socket family :py:data:`~socket.AF_UNIX`, socket + type :py:data:`~socket.SOCK_STREAM`. The :py:data:`~socket.AF_UNIX` socket + family is used to communicate between processes on the same machine + efficiently. + + This method returns a :ref:`coroutine object ` which will try to + establish the connection in the background. When successful, the + coroutine returns a ``(transport, protocol)`` pair. + + See the :meth:`BaseEventLoop.create_connection` method for parameters. + + Availability: UNIX. + + Creating listening connections ------------------------------ @@ -307,11 +338,13 @@ The function :func:`start_server` creates a (:class:`StreamReader`, :class:`StreamWriter`) pair and calls back a function with this pair. -.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) - Create datagram connection. +.. method:: BaseEventLoop.create_unix_server(protocol_factory, path=None, \*, sock=None, backlog=100, ssl=None) - This method returns a :ref:`coroutine object `. + Similar to :meth:`BaseEventLoop.create_server`, but specific to the + socket family :py:data:`~socket.AF_UNIX`. + + Availability: UNIX. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:16 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogV3Jp?= =?utf-8?q?teTransport=2Eset=5Fwrite=5Fbuffer=5Fsize_to_call_=5Fmaybe=5Fpa?= =?utf-8?q?use=5Fprotocol?= Message-ID: <3fnSNc5YX4z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/b83fbd8745d5 changeset: 89737:b83fbd8745d5 branch: 3.4 user: Yury Selivanov date: Wed Feb 19 11:10:52 2014 -0500 summary: asyncio: WriteTransport.set_write_buffer_size to call _maybe_pause_protocol files: Lib/asyncio/transports.py | 8 +++- Lib/test/test_asyncio/test_transports.py | 23 ++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -241,7 +241,7 @@ def __init__(self, extra=None): super().__init__(extra) self._protocol_paused = False - self.set_write_buffer_limits() + self._set_write_buffer_limits() def _maybe_pause_protocol(self): size = self.get_write_buffer_size() @@ -273,7 +273,7 @@ 'protocol': self._protocol, }) - def set_write_buffer_limits(self, high=None, low=None): + def _set_write_buffer_limits(self, high=None, low=None): if high is None: if low is None: high = 64*1024 @@ -287,5 +287,9 @@ self._high_water = high self._low_water = low + def set_write_buffer_limits(self, high=None, low=None): + self._set_write_buffer_limits(high=high, low=low) + self._maybe_pause_protocol() + def get_write_buffer_size(self): raise NotImplementedError diff --git a/Lib/test/test_asyncio/test_transports.py b/Lib/test/test_asyncio/test_transports.py --- a/Lib/test/test_asyncio/test_transports.py +++ b/Lib/test/test_asyncio/test_transports.py @@ -4,6 +4,7 @@ import unittest.mock import asyncio +from asyncio import transports class TransportTests(unittest.TestCase): @@ -60,6 +61,28 @@ self.assertRaises(NotImplementedError, transport.terminate) self.assertRaises(NotImplementedError, transport.kill) + def test_flowcontrol_mixin_set_write_limits(self): + + class MyTransport(transports._FlowControlMixin, + transports.Transport): + + def get_write_buffer_size(self): + return 512 + + transport = MyTransport() + transport._protocol = unittest.mock.Mock() + + self.assertFalse(transport._protocol_paused) + + with self.assertRaisesRegex(ValueError, 'high.*must be >= low'): + transport.set_write_buffer_limits(high=0, low=1) + + transport.set_write_buffer_limits(high=1024, low=128) + self.assertFalse(transport._protocol_paused) + + transport.set_write_buffer_limits(high=256, low=128) + self.assertTrue(transport._protocol_paused) + if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:18 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwNjgy?= =?utf-8?q?=3A_Fix_UNIX_sockets_tests_of_test=5Fasyncio_on_Mac_OS_X_Tiger?= Message-ID: <3fnSNf13ZZz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/472a4988489e changeset: 89738:472a4988489e branch: 3.4 user: Victor Stinner date: Wed Feb 19 18:10:32 2014 +0100 summary: Close #20682: Fix UNIX sockets tests of test_asyncio on Mac OS X Tiger On Mac OS X Tiger (and older), getsockname() returns a zero-length address for UNIX socket, and so 'sockname' extra info is None. files: Lib/test/test_asyncio/test_events.py | 32 +++++++++++++-- 1 files changed, 27 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -4,6 +4,7 @@ import gc import io import os +import platform import signal import socket try: @@ -40,6 +41,15 @@ raise FileNotFoundError(filename) +def osx_tiger(): + """Return True if the platform is Mac OS 10.4 or older.""" + if sys.platform != 'darwin': + return False + version = platform.mac_ver()[0] + version = tuple(map(int, version.split('.'))) + return version < (10, 5) + + ONLYCERT = data_file('ssl_cert.pem') ONLYKEY = data_file('ssl_key.pem') SIGNED_CERTFILE = data_file('keycert3.pem') @@ -499,10 +509,12 @@ self.loop.run_forever() self.assertEqual(caught, 1) - def _basetest_create_connection(self, connection_fut): + def _basetest_create_connection(self, connection_fut, check_sockname): tr, pr = self.loop.run_until_complete(connection_fut) self.assertIsInstance(tr, asyncio.Transport) self.assertIsInstance(pr, asyncio.Protocol) + if check_sockname: + self.assertIsNotNone(tr.get_extra_info('sockname')) self.loop.run_until_complete(pr.done) self.assertGreater(pr.nbytes, 0) tr.close() @@ -515,10 +527,14 @@ @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_connection(self): + # Issue #20682: On Mac OS X Tiger, getsockname() returns a + # zero-length address for UNIX socket. + check_sockname = not osx_tiger() + with test_utils.run_test_unix_server() as httpd: conn_fut = self.loop.create_unix_connection( lambda: MyProto(loop=self.loop), httpd.address) - self._basetest_create_connection(conn_fut) + self._basetest_create_connection(conn_fut, check_sockname) def test_create_connection_sock(self): with test_utils.run_test_server() as httpd: @@ -548,12 +564,14 @@ self.assertGreater(pr.nbytes, 0) tr.close() - def _basetest_create_ssl_connection(self, connection_fut): + def _basetest_create_ssl_connection(self, connection_fut, + check_sockname=True): tr, pr = self.loop.run_until_complete(connection_fut) self.assertIsInstance(tr, asyncio.Transport) self.assertIsInstance(pr, asyncio.Protocol) self.assertTrue('ssl' in tr.__class__.__name__.lower()) - self.assertIsNotNone(tr.get_extra_info('sockname')) + if check_sockname: + self.assertIsNotNone(tr.get_extra_info('sockname')) self.loop.run_until_complete(pr.done) self.assertGreater(pr.nbytes, 0) tr.close() @@ -571,6 +589,10 @@ @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_ssl_unix_connection(self): + # Issue #20682: On Mac OS X Tiger, getsockname() returns a + # zero-length address for UNIX socket. + check_sockname = not osx_tiger() + with test_utils.run_test_unix_server(use_ssl=True) as httpd: conn_fut = self.loop.create_unix_connection( lambda: MyProto(loop=self.loop), @@ -578,7 +600,7 @@ ssl=test_utils.dummy_ssl_context(), server_hostname='127.0.0.1') - self._basetest_create_ssl_connection(conn_fut) + self._basetest_create_ssl_connection(conn_fut, check_sockname) def test_create_connection_local_addr(self): with test_utils.run_test_server() as httpd: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:19 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjgy?= =?utf-8?q?=3A_Oops=2C_fix_test=5Fcreate=5Fconnection=28=29_of_test=5Fasyn?= =?utf-8?q?cio_=28fix_my?= Message-ID: <3fnSNg2TrVz7LkB@mail.python.org> http://hg.python.org/cpython/rev/f38a7d61c4c2 changeset: 89739:f38a7d61c4c2 branch: 3.4 user: Victor Stinner date: Wed Feb 19 18:32:03 2014 +0100 summary: Issue #20682: Oops, fix test_create_connection() of test_asyncio (fix my previous commit) files: Lib/test/test_asyncio/test_events.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -509,7 +509,7 @@ self.loop.run_forever() self.assertEqual(caught, 1) - def _basetest_create_connection(self, connection_fut, check_sockname): + def _basetest_create_connection(self, connection_fut, check_sockname=True): tr, pr = self.loop.run_until_complete(connection_fut) self.assertIsInstance(tr, asyncio.Transport) self.assertIsInstance(pr, asyncio.Protocol) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:20 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdDogRml4?= =?utf-8?q?_getfullargspec=28=29_to_not_to_follow_=5F=5Fwrapped=5F=5F_chai?= =?utf-8?q?ns?= Message-ID: <3fnSNh5YJmz7LkP@mail.python.org> http://hg.python.org/cpython/rev/4f5bd6726075 changeset: 89740:4f5bd6726075 branch: 3.4 user: Yury Selivanov date: Wed Feb 19 16:27:23 2014 -0500 summary: inspect: Fix getfullargspec() to not to follow __wrapped__ chains Initial patch by Nick Coghlan. files: Lib/inspect.py | 115 +++++++++++++++----------- Lib/test/test_inspect.py | 40 +++++++++ Misc/NEWS | 4 + 3 files changed, 111 insertions(+), 48 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -949,9 +949,9 @@ The first four items in the tuple correspond to getargspec(). """ - builtin_method_param = None - - if ismethod(func): + try: + # Re: `skip_bound_arg=False` + # # There is a notable difference in behaviour between getfullargspec # and Signature: the former always returns 'self' parameter for bound # methods, whereas the Signature always shows the actual calling @@ -960,20 +960,15 @@ # To simulate this behaviour, we "unbind" bound methods, to trick # inspect.signature to always return their first parameter ("self", # usually) - func = func.__func__ - - elif isbuiltin(func): - # We have a builtin function or method. For that, we check the - # special '__text_signature__' attribute, provided by the - # Argument Clinic. If it's a method, we'll need to make sure - # that its first parameter (usually "self") is always returned - # (see the previous comment). - text_signature = getattr(func, '__text_signature__', None) - if text_signature and text_signature.startswith('($'): - builtin_method_param = _signature_get_bound_param(text_signature) - - try: - sig = signature(func) + + # Re: `follow_wrapper_chains=False` + # + # getfullargspec() historically ignored __wrapped__ attributes, + # so we ensure that remains the case in 3.3+ + + sig = _signature_internal(func, + follow_wrapper_chains=False, + skip_bound_arg=False) except Exception as ex: # Most of the times 'signature' will raise ValueError. # But, it can also raise AttributeError, and, maybe something @@ -1023,13 +1018,6 @@ # compatibility with 'func.__defaults__' defaults = None - if builtin_method_param and (not args or args[0] != builtin_method_param): - # `func` is a method, and we always need to return its - # first parameter -- usually "self" (to be backwards - # compatible with the previous implementation of - # getfullargspec) - args.insert(0, builtin_method_param) - return FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwdefaults, annotations) @@ -1719,7 +1707,7 @@ return clean_signature, self_parameter, last_positional_only -def _signature_fromstr(cls, obj, s): +def _signature_fromstr(cls, obj, s, skip_bound_arg=True): # Internal helper to parse content of '__text_signature__' # and return a Signature based on it Parameter = cls._parameter_cls @@ -1840,7 +1828,7 @@ if self_parameter is not None: assert parameters - if getattr(obj, '__self__', None): + if getattr(obj, '__self__', None) and skip_bound_arg: # strip off self, it's already been bound parameters.pop(0) else: @@ -1851,8 +1839,21 @@ return cls(parameters, return_annotation=cls.empty) -def signature(obj): - '''Get a signature object for the passed callable.''' +def _signature_from_builtin(cls, func, skip_bound_arg=True): + # Internal helper function to get signature for + # builtin callables + if not _signature_is_builtin(func): + raise TypeError("{!r} is not a Python builtin " + "function".format(func)) + + s = getattr(func, "__text_signature__", None) + if not s: + raise ValueError("no signature found for builtin {!r}".format(func)) + + return _signature_fromstr(cls, func, s, skip_bound_arg) + + +def _signature_internal(obj, follow_wrapper_chains=True, skip_bound_arg=True): if not callable(obj): raise TypeError('{!r} is not a callable object'.format(obj)) @@ -1860,11 +1861,17 @@ if isinstance(obj, types.MethodType): # In this case we skip the first parameter of the underlying # function (usually `self` or `cls`). - sig = signature(obj.__func__) - return _signature_bound_method(sig) + sig = _signature_internal(obj.__func__, + follow_wrapper_chains, + skip_bound_arg) + if skip_bound_arg: + return _signature_bound_method(sig) + else: + return sig # Was this function wrapped by a decorator? - obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) + if follow_wrapper_chains: + obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) try: sig = obj.__signature__ @@ -1887,7 +1894,9 @@ # (usually `self`, or `cls`) will not be passed # automatically (as for boundmethods) - wrapped_sig = signature(partialmethod.func) + wrapped_sig = _signature_internal(partialmethod.func, + follow_wrapper_chains, + skip_bound_arg) sig = _signature_get_partial(wrapped_sig, partialmethod, (None,)) first_wrapped_param = tuple(wrapped_sig.parameters.values())[0] @@ -1896,7 +1905,8 @@ return sig.replace(parameters=new_params) if _signature_is_builtin(obj): - return Signature.from_builtin(obj) + return _signature_from_builtin(Signature, obj, + skip_bound_arg=skip_bound_arg) if isfunction(obj) or _signature_is_functionlike(obj): # If it's a pure Python function, or an object that is duck type @@ -1904,7 +1914,9 @@ return Signature.from_function(obj) if isinstance(obj, functools.partial): - wrapped_sig = signature(obj.func) + wrapped_sig = _signature_internal(obj.func, + follow_wrapper_chains, + skip_bound_arg) return _signature_get_partial(wrapped_sig, obj) sig = None @@ -1915,17 +1927,23 @@ # in its metaclass call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: - sig = signature(call) + sig = _signature_internal(call, + follow_wrapper_chains, + skip_bound_arg) else: # Now we check if the 'obj' class has a '__new__' method new = _signature_get_user_defined_method(obj, '__new__') if new is not None: - sig = signature(new) + sig = _signature_internal(new, + follow_wrapper_chains, + skip_bound_arg) else: # Finally, we should have at least __init__ implemented init = _signature_get_user_defined_method(obj, '__init__') if init is not None: - sig = signature(init) + sig = _signature_internal(init, + follow_wrapper_chains, + skip_bound_arg) if sig is None: # At this point we know, that `obj` is a class, with no user- @@ -1967,7 +1985,9 @@ call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: try: - sig = signature(call) + sig = _signature_internal(call, + follow_wrapper_chains, + skip_bound_arg) except ValueError as ex: msg = 'no signature found for {!r}'.format(obj) raise ValueError(msg) from ex @@ -1975,7 +1995,10 @@ if sig is not None: # For classes and objects we skip the first parameter of their # __call__, __new__, or __init__ methods - return _signature_bound_method(sig) + if skip_bound_arg: + return _signature_bound_method(sig) + else: + return sig if isinstance(obj, types.BuiltinFunctionType): # Raise a nicer error message for builtins @@ -1984,6 +2007,10 @@ raise ValueError('callable {!r} is not supported by signature'.format(obj)) +def signature(obj): + '''Get a signature object for the passed callable.''' + return _signature_internal(obj) + class _void: '''A private marker - used in Parameter & Signature''' @@ -2417,15 +2444,7 @@ @classmethod def from_builtin(cls, func): - if not _signature_is_builtin(func): - raise TypeError("{!r} is not a Python builtin " - "function".format(func)) - - s = getattr(func, "__text_signature__", None) - if not s: - raise ValueError("no signature found for builtin {!r}".format(func)) - - return _signature_fromstr(cls, func, s) + return _signature_from_builtin(cls, func) @property def parameters(self): diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -577,6 +577,46 @@ kwonlyargs_e=['arg'], formatted='(*, arg)') + def test_argspec_api_ignores_wrapped(self): + # Issue 20684: low level introspection API must ignore __wrapped__ + @functools.wraps(mod.spam) + def ham(x, y): + pass + # Basic check + self.assertArgSpecEquals(ham, ['x', 'y'], formatted='(x, y)') + self.assertFullArgSpecEquals(ham, ['x', 'y'], formatted='(x, y)') + self.assertFullArgSpecEquals(functools.partial(ham), + ['x', 'y'], formatted='(x, y)') + # Other variants + def check_method(f): + self.assertArgSpecEquals(f, ['self', 'x', 'y'], + formatted='(self, x, y)') + class C: + @functools.wraps(mod.spam) + def ham(self, x, y): + pass + pham = functools.partialmethod(ham) + @functools.wraps(mod.spam) + def __call__(self, x, y): + pass + check_method(C()) + check_method(C.ham) + check_method(C().ham) + check_method(C.pham) + check_method(C().pham) + + class C_new: + @functools.wraps(mod.spam) + def __new__(self, x, y): + pass + check_method(C_new) + + class C_init: + @functools.wraps(mod.spam) + def __init__(self, x, y): + pass + check_method(C_init) + def test_getfullargspec_signature_attr(self): def test(): pass diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,10 @@ Library ------- +- Issue #20684: Fix inspect.getfullargspec() to not to follow __wrapped__ + chains. Make its behaviour consistent with bound methods first argument. + Patch by Nick Coghlan and Yury Selivanov. + - Issue #20681: Add new error handling API in asyncio. New APIs: loop.set_exception_handler(), loop.default_exception_handler(), and loop.call_exception_handler(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:22 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_=23136=3A_Add_get/set=5Fdebug=28=29_methods_to_BaseEve?= =?utf-8?q?ntLoopTests=2E?= Message-ID: <3fnSNk1MM4z7LkR@mail.python.org> http://hg.python.org/cpython/rev/644acb7e3c22 changeset: 89741:644acb7e3c22 branch: 3.4 user: Victor Stinner date: Wed Feb 19 23:15:02 2014 +0100 summary: asyncio, Tulip issue #136: Add get/set_debug() methods to BaseEventLoopTests. Add also a PYTHONASYNCIODEBUG environment variable to debug coroutines since Python startup, to be able to debug coroutines defined directly in the asyncio module. files: Doc/library/asyncio-dev.rst | 10 ++- Doc/library/asyncio-eventloop.rst | 16 ++++++ Doc/using/cmdline.rst | 8 +++ Lib/asyncio/base_events.py | 7 ++ Lib/asyncio/events.py | 8 +++ Lib/asyncio/tasks.py | 5 +- Lib/test/test_asyncio/test_base_events.py | 6 ++ Lib/test/test_asyncio/test_tasks.py | 28 +++++++++++ 8 files changed, 83 insertions(+), 5 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -1,5 +1,7 @@ .. currentmodule:: asyncio +.. _asyncio-dev: + Develop with asyncio ==================== @@ -81,10 +83,10 @@ When a coroutine function is called but not passed to :func:`async` or to the :class:`Task` constructor, it is not scheduled and it is probably a bug. -To detect such bug, set :data:`asyncio.tasks._DEBUG` to ``True``. When the -coroutine object is destroyed by the garbage collector, a log will be emitted -with the traceback where the coroutine function was called. See the -:ref:`asyncio logger `. +To detect such bug, set the environment variable :envvar:`PYTHONASYNCIODEBUG` +to ``1``. When the coroutine object is destroyed by the garbage collector, a +log will be emitted with the traceback where the coroutine function was called. +See the :ref:`asyncio logger `. The debug flag changes the behaviour of the :func:`coroutine` decorator. The debug flag value is only used when then coroutine function is defined, not when diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -553,6 +553,22 @@ Set the default executor used by :meth:`run_in_executor`. +Debug mode +---------- + +.. method:: BaseEventLoop.get_debug() + + Get the debug mode (:class:`bool`) of the event loop. + +.. method:: BaseEventLoop.set_debug(enabled: bool) + + Set the debug mode of the event loop. + +.. seealso:: + + The :ref:`Develop with asyncio ` section. + + Server ------ diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -614,6 +614,14 @@ .. versionadded:: 3.4 +.. envvar:: PYTHONASYNCIODEBUG + + If this environment variable is set to a non-empty string, enable the debug + mode of the :mod:`asyncio` module. + + .. versionadded:: 3.4 + + Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -123,6 +123,7 @@ self._running = False self._clock_resolution = time.get_clock_info('monotonic').resolution self._exception_handler = None + self._debug = False def _make_socket_transport(self, sock, protocol, waiter=None, *, extra=None, server=None): @@ -795,3 +796,9 @@ if not handle._cancelled: handle._run() handle = None # Needed to break cycles when an exception occurs. + + def get_debug(self): + return self._debug + + def set_debug(self, enabled): + self._debug = enabled diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -345,6 +345,14 @@ def call_exception_handler(self, context): raise NotImplementedError + # Debug flag management. + + def get_debug(self): + raise NotImplementedError + + def set_debug(self, enabled): + raise NotImplementedError + class AbstractEventLoopPolicy: """Abstract policy for accessing the event loop.""" diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -12,6 +12,8 @@ import functools import inspect import linecache +import os +import sys import traceback import weakref @@ -28,7 +30,8 @@ # before you define your coroutines. A downside of using this feature # is that tracebacks show entries for the CoroWrapper.__next__ method # when _DEBUG is true. -_DEBUG = False +_DEBUG = (not sys.flags.ignore_environment + and bool(os.environ.get('PYTHONASYNCIODEBUG'))) class CoroWrapper: diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -197,6 +197,12 @@ self.assertEqual([h2], self.loop._scheduled) self.assertTrue(self.loop._process_events.called) + def test_set_debug(self): + self.loop.set_debug(True) + self.assertTrue(self.loop.get_debug()) + self.loop.set_debug(False) + self.assertFalse(self.loop.get_debug()) + @unittest.mock.patch('asyncio.base_events.time') @unittest.mock.patch('asyncio.base_events.logger') def test__run_once_logging(self, m_logger, m_time): diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1,7 +1,9 @@ """Tests for tasks.py.""" import gc +import os.path import unittest +from test.script_helper import assert_python_ok import asyncio from asyncio import test_utils @@ -1461,6 +1463,32 @@ cb.assert_called_once_with(fut) self.assertEqual(fut.result(), [3, 1, exc, exc2]) + def test_env_var_debug(self): + path = os.path.dirname(asyncio.__file__) + path = os.path.normpath(os.path.join(path, '..')) + code = '\n'.join(( + 'import sys', + 'sys.path.insert(0, %r)' % path, + 'import asyncio.tasks', + 'print(asyncio.tasks._DEBUG)')) + + # Test with -E to not fail if the unit test was run with + # PYTHONASYNCIODEBUG set to a non-empty string + sts, stdout, stderr = assert_python_ok('-E', '-c', code) + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='') + self.assertEqual(stdout.rstrip(), b'False') + + sts, stdout, stderr = assert_python_ok('-c', code, + PYTHONASYNCIODEBUG='1') + self.assertEqual(stdout.rstrip(), b'True') + + sts, stdout, stderr = assert_python_ok('-E', '-c', code, + PYTHONASYNCIODEBUG='1') + self.assertEqual(stdout.rstrip(), b'False') + class FutureGatherTests(GatherTestsBase, unittest.TestCase): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:23 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_update_magic_n?= =?utf-8?q?umber_for_=2320625?= Message-ID: <3fnSNl3TCmz7LkL@mail.python.org> http://hg.python.org/cpython/rev/e5cde3fd7c74 changeset: 89742:e5cde3fd7c74 branch: 3.4 user: Benjamin Peterson date: Wed Feb 19 18:05:36 2014 -0500 summary: update magic number for #20625 files: Lib/importlib/_bootstrap.py | 3 +- Python/importlib.h | 320 ++++++++++++------------ 2 files changed, 162 insertions(+), 161 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -418,12 +418,13 @@ # Python 3.4a1 3280 (remove implicit class argument) # Python 3.4a4 3290 (changes to __qualname__ computation) # Python 3.4a4 3300 (more changes to __qualname__ computation) +# Python 3.4rc2 3310 (alter __qualname__ computation) # # MAGIC must change whenever the bytecode emitted by the compiler may no # longer be understood by older implementations of the eval loop (usually # due to the addition of new opcodes). -MAGIC_NUMBER = (3300).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3310).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:24 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_remove_reference_to_=5FDEBUG_=28now_replaced_with_PYTHONASYNCI?= =?utf-8?q?ODEBUG?= Message-ID: <3fnSNm5Gq0z7Lkd@mail.python.org> http://hg.python.org/cpython/rev/3ebffb48c28b changeset: 89743:3ebffb48c28b branch: 3.4 user: Victor Stinner date: Thu Feb 20 01:44:10 2014 +0100 summary: asyncio doc: remove reference to _DEBUG (now replaced with PYTHONASYNCIODEBUG env var), document the default debug mode files: Doc/library/asyncio-dev.rst | 1 - Doc/library/asyncio-eventloop.rst | 2 +- 2 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -98,7 +98,6 @@ Example with the bug:: import asyncio - asyncio.tasks._DEBUG = True @asyncio.coroutine def test(): diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -558,7 +558,7 @@ .. method:: BaseEventLoop.get_debug() - Get the debug mode (:class:`bool`) of the event loop. + Get the debug mode (:class:`bool`) of the event loop, ``False`` by default. .. method:: BaseEventLoop.set_debug(enabled: bool) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:25 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5kb2Nz?= =?utf-8?q?=3A_Document_Error_Handling_API_and_asyncio=2EHandle?= Message-ID: <3fnSNn6hFXz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/55bd293de3a6 changeset: 89744:55bd293de3a6 branch: 3.4 user: Yury Selivanov date: Wed Feb 19 20:58:44 2014 -0500 summary: asyncio.docs: Document Error Handling API and asyncio.Handle files: Doc/library/asyncio-eventloop.rst | 69 ++++++++++++++++++- 1 files changed, 67 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -142,6 +142,8 @@ Any positional arguments after the callback will be passed to the callback when it is called. + An instance of :class:`asyncio.Handle` is returned. + .. method:: BaseEventLoop.call_soon_threadsafe(callback, \*args) Like :meth:`call_soon`, but thread safe. @@ -167,8 +169,7 @@ Arrange for the *callback* to be called after the given *delay* seconds (either an int or float). - A "handle" is returned: an opaque object with a :meth:`cancel` method - that can be used to cancel the call. + An instance of :class:`asyncio.Handle` is returned. *callback* will be called exactly once per call to :meth:`call_later`. If two callbacks are scheduled for exactly the same time, it is @@ -553,6 +554,56 @@ Set the default executor used by :meth:`run_in_executor`. +Error Handling API +------------------ + +Allows to customize how exceptions are handled in the event loop. + +.. method:: BaseEventLoop.set_exception_handler(handler) + + Set *handler* as the new event loop exception handler. + + If *handler* is ``None``, the default exception handler will + be set. + + If *handler* is a callable object, it should have a + matching signature to ``(loop, context)``, where ``loop`` + will be a reference to the active event loop, ``context`` + will be a ``dict`` object (see :meth:`call_exception_handler` + documentation for details about context). + +.. method:: BaseEventLoop.default_exception_handler(context) + + Default exception handler. + + This is called when an exception occurs and no exception + handler is set, and can be called by a custom exception + handler that wants to defer to the default behavior. + + *context* parameter has the same meaning as in + :meth:`call_exception_handler`. + +.. method:: BaseEventLoop.call_exception_handler(context) + + Call the current event loop exception handler. + + *context* is a ``dict`` object containing the following keys + (new keys may be introduced later): + + * 'message': Error message; + * 'exception' (optional): Exception object; + * 'future' (optional): :class:`asyncio.Future` instance; + * 'handle' (optional): :class:`asyncio.Handle` instance; + * 'protocol' (optional): :ref:`Protocol ` instance; + * 'transport' (optional): :ref:`Transport ` instance; + * 'socket' (optional): :class:`socket.socket` instance. + + .. note:: + + Note: this method should not be overloaded in subclassed + event loops. For any custom exception handling, use + :meth:`set_exception_handler()` method. + Debug mode ---------- @@ -585,6 +636,20 @@ Coroutine to wait until service is closed. +Handle +------ + +.. class:: Handle + + A callback wrapper object returned by :func:`BaseEventLoop.call_soon`, + :func:`BaseEventLoop.call_soon_threadsafe`, :func:`BaseEventLoop.call_later`, + and :func:`BaseEventLoop.call_at`. + + .. method:: cancel() + + Cancel the call. + + .. _asyncio-hello-world-callback: Example: Hello World (callback) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:27 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogbWVyZ2UgMy4zICgj?= =?utf-8?q?20695=29?= Message-ID: <3fnSNq13l7z7LkZ@mail.python.org> http://hg.python.org/cpython/rev/a5247ea950d5 changeset: 89745:a5247ea950d5 branch: 3.4 user: Benjamin Peterson date: Wed Feb 19 23:06:41 2014 -0500 summary: merge 3.3 (#20695) files: Lib/test/test_urllibnet.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -103,7 +103,7 @@ # Make sure fd returned by fileno is valid. with self.urlopen("http://www.python.org/", timeout=None) as open_url: fd = open_url.fileno() - with os.fdopen(fd, encoding='utf-8') as f: + with os.fdopen(fd, 'rb') as f: self.assertTrue(f.read(), "reading from file created using fd " "returned by fileno failed") @@ -150,7 +150,7 @@ with self.urlretrieve("http://www.python.org/") as (file_location, info): self.assertTrue(os.path.exists(file_location), "file location returned by" " urlretrieve is not a valid path") - with open(file_location, encoding='utf-8') as f: + with open(file_location, 'rb') as f: self.assertTrue(f.read(), "reading from the file location returned" " by urlretrieve failed") @@ -160,7 +160,7 @@ support.TESTFN) as (file_location, info): self.assertEqual(file_location, support.TESTFN) self.assertTrue(os.path.exists(file_location)) - with open(file_location, encoding='utf-8') as f: + with open(file_location, 'rb') as f: self.assertTrue(f.read(), "reading from temporary file failed") def test_header(self): @@ -169,7 +169,7 @@ self.assertIsInstance(info, email.message.Message, "info is not an instance of email.message.Message") - logo = "http://www.python.org/community/logos/python-logo-master-v3-TM.png" + logo = "http://www.python.org/static/community_logos/python-logo-master-v3-TM.png" def test_data_header(self): with self.urlretrieve(self.logo) as (file_location, fileheaders): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:28 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5zdWJw?= =?utf-8?q?rocess=3A_Fix_a_race_condition_in_communicate=28=29?= Message-ID: <3fnSNr2Mdtz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/aa0043256457 changeset: 89746:aa0043256457 branch: 3.4 user: Victor Stinner date: Thu Feb 20 10:12:59 2014 +0100 summary: asyncio.subprocess: Fix a race condition in communicate() Use self._loop instead of self._transport._loop, because transport._loop is set to None at process exit. files: Lib/asyncio/subprocess.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -146,7 +146,6 @@ @tasks.coroutine def communicate(self, input=None): - loop = self._transport._loop if input: stdin = self._feed_stdin(input) else: @@ -160,7 +159,7 @@ else: stderr = self._noop() stdin, stdout, stderr = yield from tasks.gather(stdin, stdout, stderr, - loop=loop) + loop=self._loop) yield from self.wait() return (stdout, stderr) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:29 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_=5FProactorWritePipeTransport=2E=5Fpipe=5Fclosed=28=29?= Message-ID: <3fnSNs3rl7z7LjV@mail.python.org> http://hg.python.org/cpython/rev/2d1bb330a4d2 changeset: 89747:2d1bb330a4d2 branch: 3.4 user: Victor Stinner date: Thu Feb 20 10:33:01 2014 +0100 summary: asyncio: Fix _ProactorWritePipeTransport._pipe_closed() The "exc" variable was not defined, pass a BrokenPipeError exception instead. files: Lib/asyncio/proactor_events.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -275,7 +275,7 @@ assert fut is self._read_fut, (fut, self._read_fut) self._read_fut = None if self._write_fut is not None: - self._force_close(exc) + self._force_close(BrokenPipeError()) else: self.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:30 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogcmVt?= =?utf-8?q?ove_unused_imports_and_unused_variables_noticed_by_pyflakes?= Message-ID: <3fnSNt5t4tz7LjY@mail.python.org> http://hg.python.org/cpython/rev/9ac04fa2a354 changeset: 89748:9ac04fa2a354 branch: 3.4 user: Victor Stinner date: Thu Feb 20 10:37:27 2014 +0100 summary: asyncio: remove unused imports and unused variables noticed by pyflakes files: Lib/asyncio/events.py | 3 --- Lib/asyncio/futures.py | 1 - Lib/asyncio/tasks.py | 2 -- Lib/asyncio/test_utils.py | 2 +- Lib/asyncio/unix_events.py | 1 - Lib/asyncio/windows_events.py | 1 - Lib/selectors.py | 3 +-- 7 files changed, 2 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -9,12 +9,9 @@ ] import subprocess -import sys import threading import socket -from .log import logger - class Handle: """Object returned by callback registration methods.""" diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -11,7 +11,6 @@ import traceback from . import events -from .log import logger # States for Future. _PENDING = 'PENDING' diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -7,7 +7,6 @@ 'gather', 'shield', ] -import collections import concurrent.futures import functools import inspect @@ -486,7 +485,6 @@ if isinstance(fs, futures.Future) or iscoroutine(fs): raise TypeError("expect a list of futures, not %s" % type(fs).__name__) loop = loop if loop is not None else events.get_event_loop() - deadline = None if timeout is None else loop.time() + timeout todo = {async(f, loop=loop) for f in set(fs)} from .queues import Queue # Import here to avoid circular import problem. done = Queue(loop=loop) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -15,7 +15,7 @@ import unittest.mock from http.server import HTTPServer -from wsgiref.simple_server import make_server, WSGIRequestHandler, WSGIServer +from wsgiref.simple_server import WSGIRequestHandler, WSGIServer try: import ssl diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -15,7 +15,6 @@ from . import base_subprocess from . import constants from . import events -from . import protocols from . import selector_events from . import tasks from . import transports diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -5,7 +5,6 @@ import math import socket import struct -import subprocess import weakref from . import events diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -5,9 +5,8 @@ """ -from abc import ABCMeta, abstractmethod, abstractproperty +from abc import ABCMeta, abstractmethod from collections import namedtuple, Mapping -import functools import math import select import sys -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:32 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_=5Fcheck=5Fresolved=5Faddress=28=29_for_IPv6_address?= Message-ID: <3fnSNw0RKGz7LkM@mail.python.org> http://hg.python.org/cpython/rev/4b326b3c307c changeset: 89749:4b326b3c307c branch: 3.4 user: Victor Stinner date: Thu Feb 20 16:43:09 2014 +0100 summary: asyncio: Fix _check_resolved_address() for IPv6 address files: Lib/asyncio/base_events.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -45,10 +45,13 @@ # Ensure that the address is already resolved to avoid the trap of hanging # the entire event loop when the address requires doing a DNS lookup. family = sock.family - if family not in (socket.AF_INET, socket.AF_INET6): + if family == socket.AF_INET: + host, port = address + elif family == socket.AF_INET6: + host, port, flow_info, scope_id = address + else: return - host, port = address type_mask = 0 if hasattr(socket, 'SOCK_NONBLOCK'): type_mask |= socket.SOCK_NONBLOCK -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:34 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogb3Bz?= =?utf-8?q?=2C_and_now_fix_also_the_unit_test_for_IPv6_address=3A?= Message-ID: <3fnSNy2z4Zz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/5e5dccf845aa changeset: 89750:5e5dccf845aa branch: 3.4 user: Victor Stinner date: Thu Feb 20 17:01:11 2014 +0100 summary: asyncio: ops, and now fix also the unit test for IPv6 address: test_sock_connect_address() files: Lib/test/test_asyncio/test_events.py | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1335,12 +1335,11 @@ 'selector': self.loop._selector.__class__.__name__}) def test_sock_connect_address(self): - families = [socket.AF_INET] + families = [(socket.AF_INET, ('www.python.org', 80))] if support.IPV6_ENABLED: - families.append(socket.AF_INET6) + families.append((socket.AF_INET6, ('www.python.org', 80, 0, 0))) - address = ('www.python.org', 80) - for family in families: + for family, address in families: for sock_type in (socket.SOCK_STREAM, socket.SOCK_DGRAM): sock = socket.socket(family, sock_type) with sock: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:35 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogTWlzYy9ORVdTOiBB?= =?utf-8?q?dd_some_missing_news_items_re_asyncio=2E?= Message-ID: <3fnSNz4m45z7LjY@mail.python.org> http://hg.python.org/cpython/rev/cb88ed0e2e1d changeset: 89751:cb88ed0e2e1d branch: 3.4 user: Yury Selivanov date: Thu Feb 20 13:59:14 2014 -0500 summary: Misc/NEWS: Add some missing news items re asyncio. files: Misc/NEWS | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,25 @@ Library ------- +- Issue #20566: Change asyncio.as_completed() to use a Queue, to + avoid O(N**2) behavior. + +- Issue #20704: Implement new debug API in asyncio. Add new methods + BaseEventLoop.set_debug() and BaseEventLoop.get_debug(). + Add support for setting 'asyncio.tasks._DEBUG' variable with + 'PYTHONASYNCIODEBUG' environment variable. + +- asyncio: Refactoring and fixes: BaseEventLoop.sock_connect() raises an + error if the address is not resolved; use __slots__ in Handle and + TimerHandle; as_completed() and wait() raise TypeError if the passed + list of Futures is a single Future; call_soon() and other 'call_*()' + functions raise TypeError if the passed callback is a coroutine + function; _ProactorBasePipeTransport uses _FlowControlMixin; + WriteTransport.set_write_buffer_size() calls _maybe_pause_protocol() + to consider pausing receiving if the watermark limits have changed; + fix _check_resolved_address() for IPv6 address; and other minor + improvements, along with multiple documentation updates. + - Issue #20684: Fix inspect.getfullargspec() to not to follow __wrapped__ chains. Make its behaviour consistent with bound methods first argument. Patch by Nick Coghlan and Yury Selivanov. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:36 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5kb2Nz?= =?utf-8?q?=3A_Improve_documentation_of_Streams=2E_Issue_=2320696=2E?= Message-ID: <3fnSP067CWz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/dc8331171f14 changeset: 89752:dc8331171f14 branch: 3.4 user: Yury Selivanov date: Thu Feb 20 14:10:02 2014 -0500 summary: asyncio.docs: Improve documentation of Streams. Issue #20696. files: Doc/library/asyncio-stream.rst | 72 +++++++++++++-------- 1 files changed, 43 insertions(+), 29 deletions(-) diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -34,7 +34,7 @@ .. function:: start_server(client_connected_cb, host=None, port=None, \*, loop=None, limit=None, **kwds) - Start a socket server, call back for each client connected. + Start a socket server, with a callback for each client connected. The first parameter, *client_connected_cb*, takes two parameters: *client_reader*, *client_writer*. *client_reader* is a @@ -58,6 +58,29 @@ This function returns a :ref:`coroutine object `. +.. function:: open_unix_connection(path=None, \*, loop=None, limit=None, **kwds) + + A wrapper for :meth:`~BaseEventLoop.create_unix_connection()` returning + a (reader, writer) pair. + + See :func:`open_connection` for information about return value and other + details. + + This function returns a :ref:`coroutine object `. + + Availability: UNIX. + +.. function:: start_unix_server(client_connected_cb, path=None, \*, loop=None, limit=None, **kwds) + + Start a UNIX Domain Socket server, with a callback for each client connected. + + See :func:`start_server` for information about return value and other + details. + + This function returns a :ref:`coroutine object `. + + Availability: UNIX. + StreamReader ============ @@ -70,11 +93,12 @@ .. method:: feed_eof() - XXX + Acknowledge the EOF. .. method:: feed_data(data) - XXX + Feed *data* bytes in the internal buffer. Any operations waiting + for the data will be resumed. .. method:: set_exception(exc) @@ -86,13 +110,23 @@ .. method:: read(n=-1) - XXX + Read up to *n* bytes. If *n* is not provided, or set to ``-1``, + read until EOF and return all read bytes. + + If the EOF was received and the internal buffer is empty, + return an empty ``bytes`` object. This method returns a :ref:`coroutine object `. .. method:: readline() - XXX + Read one line, where "line" is a sequence of bytes ending with ``\n``. + + If EOF is received, and ``\n`` was not found, the method will + return the partial read bytes. + + If the EOF was received and the internal buffer is empty, + return an empty ``bytes`` object. This method returns a :ref:`coroutine object `. @@ -105,6 +139,10 @@ This method returns a :ref:`coroutine object `. + .. method:: at_eof() + + Return ``True`` if the buffer is empty and :meth:`feed_eof` was called. + StreamWriter ============ @@ -186,30 +224,6 @@ potential uses, and to prevent the user of the :class:`StreamReader` to accidentally call inappropriate methods of the protocol.) - .. method:: connection_made(transport) - - XXX - - .. method:: connection_lost(exc) - - XXX - - .. method:: data_received(data) - - XXX - - .. method:: eof_received() - - XXX - - .. method:: pause_writing() - - XXX - - .. method:: resume_writing() - - XXX - IncompleteReadError =================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:38 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogX2No?= =?utf-8?q?eck=5Fresolved=5Faddress=28=29_must_also_accept_IPv6_without_fl?= =?utf-8?q?ow=5Finfo_and?= Message-ID: <3fnSP20WV0z7LkJ@mail.python.org> http://hg.python.org/cpython/rev/19d361106e92 changeset: 89753:19d361106e92 branch: 3.4 user: Victor Stinner date: Thu Feb 20 21:59:38 2014 +0100 summary: asyncio: _check_resolved_address() must also accept IPv6 without flow_info and scope_id: (host, port). files: Lib/asyncio/base_events.py | 2 +- Lib/test/test_asyncio/test_events.py | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -48,7 +48,7 @@ if family == socket.AF_INET: host, port = address elif family == socket.AF_INET6: - host, port, flow_info, scope_id = address + host, port = address[:2] else: return diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1335,11 +1335,14 @@ 'selector': self.loop._selector.__class__.__name__}) def test_sock_connect_address(self): - families = [(socket.AF_INET, ('www.python.org', 80))] + addresses = [(socket.AF_INET, ('www.python.org', 80))] if support.IPV6_ENABLED: - families.append((socket.AF_INET6, ('www.python.org', 80, 0, 0))) + addresses.extend(( + (socket.AF_INET6, ('www.python.org', 80)), + (socket.AF_INET6, ('www.python.org', 80, 0, 0)), + )) - for family, address in families: + for family, address in addresses: for sock_type in (socket.SOCK_STREAM, socket.SOCK_DGRAM): sock = socket.socket(family, sock_type) with sock: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:39 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5kb2Nz?= =?utf-8?q?=3A_Improve_wordings=3B_add_a_note_to_the_Coroutines_section=2E?= =?utf-8?q?_Issue?= Message-ID: <3fnSP33JM3z7LkM@mail.python.org> http://hg.python.org/cpython/rev/0f26daac85bb changeset: 89754:0f26daac85bb branch: 3.4 user: Yury Selivanov date: Thu Feb 20 16:20:44 2014 -0500 summary: asyncio.docs: Improve wordings; add a note to the Coroutines section. Issue #20706 files: Doc/library/asyncio-eventloop.rst | 45 +++++++++--------- Doc/library/asyncio-stream.rst | 14 ++-- Doc/library/asyncio-sync.rst | 14 ++-- Doc/library/asyncio-task.rst | 14 +++++- 4 files changed, 49 insertions(+), 38 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -102,7 +102,8 @@ Run until the :class:`Future` is done. - If the argument is a coroutine, it is wrapped in a :class:`Task`. + If the argument is a :ref:`coroutine `, it is wrapped + in a :class:`Task`. Return the Future's result, or raise its exception. @@ -207,7 +208,7 @@ socket type :py:data:`~socket.SOCK_STREAM`. *protocol_factory* must be a callable returning a :ref:`protocol ` instance. - This method returns a :ref:`coroutine object ` which will try to + This method is a :ref:`coroutine ` which will try to establish the connection in the background. When successful, the coroutine returns a ``(transport, protocol)`` pair. @@ -274,7 +275,7 @@ :py:data:`~socket.AF_INET6` depending on *host* (or *family* if specified), socket type :py:data:`~socket.SOCK_DGRAM`. - This method returns a :ref:`coroutine object ` which will try to + This method is a :ref:`coroutine ` which will try to establish the connection in the background. When successful, the coroutine returns a ``(transport, protocol)`` pair. @@ -288,7 +289,7 @@ family is used to communicate between processes on the same machine efficiently. - This method returns a :ref:`coroutine object ` which will try to + This method is a :ref:`coroutine ` which will try to establish the connection in the background. When successful, the coroutine returns a ``(transport, protocol)`` pair. @@ -302,8 +303,8 @@ .. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) - A :ref:`coroutine function ` which creates a TCP server bound to host and - port. + A :ref:`coroutine ` method which creates a TCP server bound to + host and port. The return value is a :class:`AbstractServer` object which can be used to stop the service. @@ -332,8 +333,6 @@ expire. If not specified will automatically be set to True on UNIX. - This method returns a :ref:`coroutine object `. - .. seealso:: The function :func:`start_server` creates a (:class:`StreamReader`, @@ -380,7 +379,7 @@ representing the data received. The maximum amount of data to be received at once is specified by *nbytes*. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. seealso:: @@ -392,9 +391,9 @@ This method continues to send data from *data* until either all data has been sent or an error occurs. ``None`` is returned on success. On error, an exception is raised, and there is no way to determine how much data, if - any, was successfully sent. + any, was successfully processed by the receiving end of the connection. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. seealso:: @@ -410,7 +409,7 @@ :py:data:`~socket.AF_INET` and :py:data:`~socket.AF_INET6` address families. Use :meth:`getaddrinfo` to resolve the hostname asynchronously. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. seealso:: @@ -427,7 +426,7 @@ and *address* is the address bound to the socket on the other end of the connection. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. seealso:: @@ -440,13 +439,13 @@ .. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) - Similar to the :meth:`socket.getaddrinfo` function, but return a - :ref:`coroutine object `. + This method is a :ref:`coroutine `, similar to + :meth:`socket.getaddrinfo` function but non-blocking. .. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) - Similar to the :meth:`socket.getnameinfo` function, but return a - :ref:`coroutine object `. + This method is a :ref:`coroutine `, similar to + :meth:`socket.getnameinfo` function but non-blocking. Running subprocesses @@ -472,7 +471,7 @@ XXX - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. See the constructor of the :class:`subprocess.Popen` class for parameters. @@ -480,7 +479,7 @@ XXX - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. See the constructor of the :class:`subprocess.Popen` class for parameters. @@ -493,7 +492,7 @@ Return pair (transport, protocol), where transport support :class:`ReadTransport` interface. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) @@ -504,7 +503,7 @@ Return pair (transport, protocol), where transport support :class:`WriteTransport` interface. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. seealso:: @@ -549,6 +548,8 @@ *executor* is a :class:`~concurrent.futures.Executor` instance, the default executor is used if *executor* is ``None``. + This method is a :ref:`coroutine `. + .. method:: BaseEventLoop.set_default_executor(executor) Set the default executor used by :meth:`run_in_executor`. @@ -633,7 +634,7 @@ .. method:: wait_closed() - Coroutine to wait until service is closed. + A :ref:`coroutine ` to wait until service is closed. Handle diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -30,7 +30,7 @@ :class:`StreamReaderProtocol` classes, just copy the code -- there's really nothing special here except some convenience.) - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. .. function:: start_server(client_connected_cb, host=None, port=None, \*, loop=None, limit=None, **kwds) @@ -56,7 +56,7 @@ The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. a :class:`AbstractServer` object which can be used to stop the service. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. .. function:: open_unix_connection(path=None, \*, loop=None, limit=None, **kwds) @@ -66,7 +66,7 @@ See :func:`open_connection` for information about return value and other details. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. Availability: UNIX. @@ -77,7 +77,7 @@ See :func:`start_server` for information about return value and other details. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. Availability: UNIX. @@ -116,7 +116,7 @@ If the EOF was received and the internal buffer is empty, return an empty ``bytes`` object. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: readline() @@ -128,7 +128,7 @@ If the EOF was received and the internal buffer is empty, return an empty ``bytes`` object. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: readexactly(n) @@ -137,7 +137,7 @@ :attr:`IncompleteReadError.partial` attribute of the exception contains the partial read bytes. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: at_eof() diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -124,7 +124,7 @@ Otherwise, block until another coroutine calls :meth:`set` to set the flag to true, then return ``True``. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. Condition @@ -175,7 +175,7 @@ condition variable in another coroutine. Once awakened, it re-acquires the lock and returns ``True``. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: wait_for(predicate) @@ -184,7 +184,7 @@ The predicate should be a callable which result will be interpreted as a boolean value. The final predicate value is the return value. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. Semaphores @@ -217,7 +217,7 @@ until some other coroutine has called :meth:`release` to make it larger than ``0``, and then return ``True``. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: locked() @@ -279,7 +279,7 @@ If you yield from :meth:`get()`, wait until a item is available. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: get_nowait() @@ -295,7 +295,7 @@ If you yield from ``put()``, wait until a free slot is available before adding item. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: put_nowait(item) @@ -350,7 +350,7 @@ it is complete. When the count of unfinished tasks drops to zero, :meth:`join` unblocks. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: task_done() diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -64,6 +64,14 @@ message is logged. See :ref:`Detect coroutines never scheduled `. +.. note:: + + In this documentation, some methods are documented as coroutines, + even if they are plain Python functions returning a :class:`Future`. + This is intentional to have a freedom of tweaking the implementation + of these functions in the future. If such a function is needed to be + used in a callback-style code, wrap its result with :func:`async`. + .. _asyncio-hello-world-coroutine: @@ -440,7 +448,7 @@ .. function:: sleep(delay, result=None, \*, loop=None) - Create a :ref:`coroutine object ` that completes after a given + Create a :ref:`coroutine ` that completes after a given time (in seconds). If *result* is provided, it is produced to the caller when the coroutine completes. @@ -505,7 +513,7 @@ | | futures finish or are cancelled. | +-----------------------------+----------------------------------------+ - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. Usage:: @@ -529,6 +537,8 @@ cancels the task and raises :exc:`TimeoutError`. To avoid the task cancellation, wrap it in :func:`shield`. + This function is a :ref:`coroutine `. + Usage:: result = yield from asyncio.wait_for(fut, 60.0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:40 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwMjIx?= =?utf-8?q?=3A_Removed_conflicting_=28or_circular=29_hypot_definition?= Message-ID: <3fnSP459w9z7Ljx@mail.python.org> http://hg.python.org/cpython/rev/033d686af4c1 changeset: 89755:033d686af4c1 branch: 3.4 user: Zachary Ware date: Thu Feb 20 15:39:29 2014 -0600 summary: Issue #20221: Removed conflicting (or circular) hypot definition when compiled with VS 2010 or above. Initial patch by Tabrez Mohammed. files: Misc/NEWS | 3 +++ PC/pyconfig.h | 4 ++++ 2 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -68,6 +68,9 @@ Build ----- +- Issue #20221: Removed conflicting (or circular) hypot definition when + compiled with VS 2010 or above. Initial patch by Tabrez Mohammed. + - Issue #20609: Restored the ability to build 64-bit Windows binaries on 32-bit Windows, which was broken by the change in issue #19788. diff --git a/PC/pyconfig.h b/PC/pyconfig.h --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -207,7 +207,11 @@ #define Py_IS_INFINITY(X) (!_finite(X) && !_isnan(X)) #define Py_IS_FINITE(X) _finite(X) #define copysign _copysign + +/* VS 2010 and above already defines hypot as _hypot */ +#if _MSC_VER < 1600 #define hypot _hypot +#endif /* Side by Side assemblies supported in VS 2005 and VS 2008 but not 2010*/ #if _MSC_VER >= 1400 && _MSC_VER < 1600 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:42 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby5kb2Nz?= =?utf-8?q?=3A_Document_subprocess=5Fexec_and_subprocess=5Fshell=2E_Issue_?= =?utf-8?q?=2320694=2E?= Message-ID: <3fnSP60vKCz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/2dfc383b4907 changeset: 89756:2dfc383b4907 branch: 3.4 user: Yury Selivanov date: Thu Feb 20 20:10:28 2014 -0500 summary: asyncio.docs: Document subprocess_exec and subprocess_shell. Issue #20694. files: Doc/library/asyncio-eventloop.rst | 52 +++++++++++++++++- 1 files changed, 48 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -467,17 +467,61 @@ :class:`SelectSelector` or :class:`PollSelector` to handle character devices on Mac OS X 10.6 (Snow Leopard) and later. -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - XXX + Create a subprocess from one or more string arguments, where the first string + specifies the program to execute, and the remaining strings specify the + program's arguments. (Thus, together the string arguments form the + ``sys.argv`` value of the program, assuming it is a Python script.) This is + similar to the standard library :class:`subprocess.Popen` class called with + shell=False and the list of strings passed as the first argument; + however, where :class:`~subprocess.Popen` takes a single argument which is + list of strings, :func:`subprocess_exec` takes multiple string arguments. + + Other parameters: + + * *stdin*: Either a file-like object representing the pipe to be connected + to the subprocess's standard input stream using + :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stdout*: Either a file-like object representing the pipe to be connected + to the subprocess's standard output stream using + :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stderr*: Either a file-like object representing the pipe to be connected + to the subprocess's standard error stream using + :meth:`~BaseEventLoop.connect_read_pipe`, or one of the constants + :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. + By default a new pipe will be created and connected. When + :const:`subprocess.STDOUT` is specified, the subprocess's standard error + stream will be connected to the same pipe as the standard output stream. + + * All other keyword arguments are passed to :class:`subprocess.Popen` + without interpretation, except for *bufsize*, *universal_newlines* and + *shell*, which should not be specified at all. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. This method is a :ref:`coroutine `. See the constructor of the :class:`subprocess.Popen` class for parameters. -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - XXX + Create a subprocess from *cmd*, which is a string using the platform's + "shell" syntax. This is similar to the standard library + :class:`subprocess.Popen` class called with ``shell=True``. + + See :meth:`~BaseEventLoop.subprocess_exec` for more details about + the remaining arguments. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. This method is a :ref:`coroutine `. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:43 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Upgrade_pip_fr?= =?utf-8?b?b20gMS41LjIgdG8gMS41LjM=?= Message-ID: <3fnSP73VyBz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/445a44d3aebc changeset: 89757:445a44d3aebc branch: 3.4 user: Donald Stufft date: Thu Feb 20 20:53:27 2014 -0500 summary: Upgrade pip from 1.5.2 to 1.5.3 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/pip-1.5.2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,7 +10,7 @@ _SETUPTOOLS_VERSION = "2.1" -_PIP_VERSION = "1.5.2" +_PIP_VERSION = "1.5.3" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-1.5.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.2-py2.py3-none-any.whl deleted file mode 100644 index 9133eed9718ca6849514fa8a0b874fcc05050913..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..31cee0aea7ef59dc070f3a7845c09699e95a97e8 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:44 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdDogRml4?= =?utf-8?q?_getfullargspec_to_support_builtin_module-level_functions=2E_Is?= =?utf-8?q?sue?= Message-ID: <3fnSP84vszz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/ed1059f5507b changeset: 89758:ed1059f5507b branch: 3.4 user: Yury Selivanov date: Fri Feb 21 01:32:42 2014 -0500 summary: inspect: Fix getfullargspec to support builtin module-level functions. Issue #20711 files: Lib/inspect.py | 11 +++++++++-- Lib/test/test_inspect.py | 7 +++++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1827,9 +1827,16 @@ p(f.args.kwarg, empty) if self_parameter is not None: + # Possibly strip the bound argument: + # - We *always* strip first bound argument if + # it is a module. + # - We don't strip first bound argument if + # skip_bound_arg is False. assert parameters - if getattr(obj, '__self__', None) and skip_bound_arg: - # strip off self, it's already been bound + _self = getattr(obj, '__self__', None) + self_isbound = _self is not None + self_ismodule = ismodule(_self) + if self_isbound and (self_ismodule or skip_bound_arg): parameters.pop(0) else: # for builtins, self parameter is always positional-only! diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -643,6 +643,13 @@ self.assertFullArgSpecEquals(_pickle.Pickler(io.BytesIO()).dump, args_e=['self', 'obj'], formatted='(self, obj)') + self.assertFullArgSpecEquals( + os.stat, + args_e=['path'], + kwonlyargs_e=['dir_fd', 'follow_symlinks'], + kwonlydefaults_e={'dir_fd': None, 'follow_symlinks': True}, + formatted='(path, *, dir_fd=None, follow_symlinks=True)') + @cpython_only @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:46 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNzEw?= =?utf-8?q?=3A_The_pydoc_summary_line_no_longer_displays_the_=22self=22_pa?= =?utf-8?q?rameter?= Message-ID: <3fnSPB0ZfBz7LkF@mail.python.org> http://hg.python.org/cpython/rev/b2ee3fe195e2 changeset: 89759:b2ee3fe195e2 branch: 3.4 user: Larry Hastings date: Thu Feb 20 23:34:46 2014 -0800 summary: Issue #20710: The pydoc summary line no longer displays the "self" parameter for bound methods. Previous to this change, it displayed "self" for methods implemented in Python but not methods implemented in C; it is now both internally consistent and consistent with inspect.Signature. files: Lib/pydoc.py | 21 ++++++++++++--- Lib/test/test_pydoc.py | 40 ++++++++++++++++++++++++++--- Misc/NEWS | 3 ++ 3 files changed, 54 insertions(+), 10 deletions(-) diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -137,6 +137,19 @@ inspect.isbuiltin(obj) or inspect.ismethoddescriptor(obj)) +def _is_bound_method(fn): + """ + Returns True if fn is a bound method, regardless of whether + fn was implemented in Python or in C. + """ + if inspect.ismethod(fn): + return True + if inspect.isbuiltin(fn): + self = getattr(fn, '__self__', None) + return not (inspect.ismodule(self) or (self is None)) + return False + + def allmethods(cl): methods = {} for key, value in inspect.getmembers(cl, _is_some_method): @@ -898,7 +911,7 @@ anchor = (cl and cl.__name__ or '') + '-' + name note = '' skipdocs = 0 - if inspect.ismethod(object): + if _is_bound_method(object): imclass = object.__self__.__class__ if cl: if imclass is not cl: @@ -909,7 +922,6 @@ object.__self__.__class__, mod) else: note = ' unbound %s method' % self.classlink(imclass,mod) - object = object.__func__ if name == realname: title = '%s' % (anchor, realname) @@ -924,7 +936,7 @@ title = '%s = %s' % ( anchor, name, reallink) argspec = None - if inspect.isfunction(object) or inspect.isbuiltin(object): + if inspect.isroutine(object): try: signature = inspect.signature(object) except (ValueError, TypeError): @@ -1301,7 +1313,7 @@ name = name or realname note = '' skipdocs = 0 - if inspect.ismethod(object): + if _is_bound_method(object): imclass = object.__self__.__class__ if cl: if imclass is not cl: @@ -1312,7 +1324,6 @@ object.__self__.__class__, mod) else: note = ' unbound %s method' % classname(imclass,mod) - object = object.__func__ if name == realname: title = self.bold(realname) diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -6,6 +6,7 @@ import inspect import pydoc import keyword +import _pickle import pkgutil import re import string @@ -689,12 +690,41 @@ self.assertIsNone(pydoc.locate(name)) self.assertRaises(ImportError, pydoc.render_doc, name) + @staticmethod + def _get_summary_line(o): + text = pydoc.plain(pydoc.render_doc(o)) + lines = text.split('\n') + assert len(lines) >= 2 + return lines[2] + + # these should include "self" + def test_unbound_python_method(self): + self.assertEqual(self._get_summary_line(textwrap.TextWrapper.wrap), + "wrap(self, text)") + @requires_docstrings - def test_builtin_signatures(self): - # test producing signatures from builtins - stat_sig = pydoc.render_doc(os.stat) - self.assertEqual(pydoc.plain(stat_sig).splitlines()[2], - 'stat(path, *, dir_fd=None, follow_symlinks=True)') + def test_unbound_builtin_method(self): + self.assertEqual(self._get_summary_line(_pickle.Pickler.dump), + "dump(self, obj, /)") + + # these no longer include "self" + def test_bound_python_method(self): + t = textwrap.TextWrapper() + self.assertEqual(self._get_summary_line(t.wrap), + "wrap(text) method of textwrap.TextWrapper instance") + + @requires_docstrings + def test_bound_builtin_method(self): + s = StringIO() + p = _pickle.Pickler(s) + self.assertEqual(self._get_summary_line(p.dump), + "dump(obj, /) method of _pickle.Pickler instance") + + # this should *never* include self! + @requires_docstrings + def test_module_level_callable(self): + self.assertEqual(self._get_summary_line(os.stat), + "stat(path, *, dir_fd=None, follow_symlinks=True)") @unittest.skipUnless(threading, 'Threading required for this test.') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,9 @@ Library ------- +- Issue #20710: The pydoc summary line no longer displays the "self" parameter + for bound methods. + - Issue #20566: Change asyncio.as_completed() to use a Queue, to avoid O(N**2) behavior. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:47 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Upgrade_pip_fr?= =?utf-8?b?b20gMS41LjMgdG8gMS41LjQ=?= Message-ID: <3fnSPC3WDpz7LkF@mail.python.org> http://hg.python.org/cpython/rev/1597d5e3cf2e changeset: 89760:1597d5e3cf2e branch: 3.4 user: Donald Stufft date: Fri Feb 21 07:42:39 2014 -0500 summary: Upgrade pip from 1.5.3 to 1.5.4 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,7 +10,7 @@ _SETUPTOOLS_VERSION = "2.1" -_PIP_VERSION = "1.5.3" +_PIP_VERSION = "1.5.4" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.3-py2.py3-none-any.whl deleted file mode 100644 index 31cee0aea7ef59dc070f3a7845c09699e95a97e8..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e07d4767a0cc5d6367948bd18149602b2e050fd4 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:48 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdC5zaWdu?= =?utf-8?q?ature=3A_Check_for_function-like_objects_before_builtins=2E_Iss?= =?utf-8?q?ue_=2317159?= Message-ID: <3fnSPD5Kfyz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/fa5127cdfe9d changeset: 89761:fa5127cdfe9d branch: 3.4 user: Yury Selivanov date: Fri Feb 21 18:30:53 2014 -0500 summary: inspect.signature: Check for function-like objects before builtins. Issue #17159 files: Lib/inspect.py | 8 ++++---- Lib/test/test_inspect.py | 16 ++++++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1911,15 +1911,15 @@ return sig.replace(parameters=new_params) - if _signature_is_builtin(obj): - return _signature_from_builtin(Signature, obj, - skip_bound_arg=skip_bound_arg) - if isfunction(obj) or _signature_is_functionlike(obj): # If it's a pure Python function, or an object that is duck type # of a Python function (Cython functions, for instance), then: return Signature.from_function(obj) + if _signature_is_builtin(obj): + return _signature_from_builtin(Signature, obj, + skip_bound_arg=skip_bound_arg) + if isinstance(obj, functools.partial): wrapped_sig = _signature_internal(obj.func, follow_wrapper_chains, diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -14,6 +14,7 @@ import types import unicodedata import unittest +import unittest.mock try: from concurrent.futures import ThreadPoolExecutor @@ -1836,6 +1837,21 @@ ('kwargs', ..., ..., "var_keyword")), ...)) + # Test with cython-like builtins: + _orig_isdesc = inspect.ismethoddescriptor + def _isdesc(obj): + if hasattr(obj, '_builtinmock'): + return True + return _orig_isdesc(obj) + + with unittest.mock.patch('inspect.ismethoddescriptor', _isdesc): + builtin_func = funclike(func) + # Make sure that our mock setup is working + self.assertFalse(inspect.ismethoddescriptor(builtin_func)) + builtin_func._builtinmock = True + self.assertTrue(inspect.ismethoddescriptor(builtin_func)) + self.assertEqual(inspect.signature(builtin_func), sig_func) + def test_signature_functionlike_class(self): # We only want to duck type function-like objects, # not classes. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:49 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogd2hhdHNuZXc6IGlt?= =?utf-8?q?portlib_deprecations=2E?= Message-ID: <3fnSPF72kHz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/1640e3b7def6 changeset: 89762:1640e3b7def6 branch: 3.4 user: R David Murray date: Sat Feb 22 14:28:46 2014 -0500 summary: whatsnew: importlib deprecations. This addresses issue #20199, if I got it right. The deprecation and replacement lists are based on the importlib documentation. files: Doc/whatsnew/3.4.rst | 28 ++++++++++++++++++++++++---- 1 files changed, 24 insertions(+), 4 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1504,10 +1504,30 @@ :meth:`difflib.SequenceMatcher.isbpopular` were removed: use ``x in sm.bjunk`` and ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object. -* :func:`importlib.util.module_for_loader` is pending deprecation. Using - :func:`importlib.util.module_to_load` and - :meth:`importlib.abc.Loader.init_module_attrs` allows subclasses of a loader - to more easily customize module loading. +* As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` + methods and functions are deprecated: :meth:`importlib.find_loader` is + replaced by :func:`importlib.util.find_spec`; + :meth:`importlib.machinery.PathFinder.find_module` is replaced by + :meth:`importlib.machinery.PathFinder.find_spec`; + :meth:`importlib.abc.MetaPathFinder.find_module` is replaced by + :meth:`importlib.abc.MetaPathFinder.find_spec`; + :meth:`importlib.abc.PathEntryFinder.find_loader` and + :meth:`~importlib.abc.PathEntryFinder.find_module` are replaced by + :meth:`importlib.abc.PathEntryFinder.find_spec`; all of the ``xxxLoader`` ABC + ``load_module`` methods (:meth:`importlib.abc.Loader.load_module`, + :meth:`importlib.abc.InspectLoader.load_module`, + :meth:`importlib.abc.FileLoader.load_module`, + :meth:`importlib.abc.SourceLoader.load_module`) should no longer be + implemented, instead loaders should implement an + ``exec_module`` method + (:meth:`importlib.abc.Loader.exec_module`, + :meth:`importlib.abc.InspectLoader.exec_module` + :meth:`importlib.abc.SourceLoader.exec_module`) and let the import system + take care of the rest; and + :meth:`importlib.abc.Loader.module_repr`, + :meth:`importlib.util.module_for_loader`, :meth:`importlib.util.set_loader`, + and :meth:`importlib.util.set_package` are no longer needed because their + functions are now handled automatically by the import system. * The :mod:`imp` module is pending deprecation. To keep compatibility with Python 2/3 code bases, the module's removal is currently not scheduled. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:51 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDg0?= =?utf-8?q?=3A_Disable_the_2_remaining_=22modules=22_tests_in_test=5Fpydoc?= =?utf-8?q?=2E?= Message-ID: <3fnSPH1jXmz7LjV@mail.python.org> http://hg.python.org/cpython/rev/7cfb3db36ec6 changeset: 89763:7cfb3db36ec6 branch: 3.4 user: Eric Snow date: Sat Feb 22 13:57:08 2014 -0700 summary: Issue #20484: Disable the 2 remaining "modules" tests in test_pydoc. I'll look into re-enabling them in issue #20128. files: Lib/test/test_pydoc.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -608,6 +608,7 @@ self.assertEqual(out.getvalue(), '') self.assertEqual(err.getvalue(), '') + @unittest.skip('causes undesireable side-effects (#20128)') def test_modules(self): # See Helper.listmodules(). num_header_lines = 2 @@ -623,6 +624,7 @@ self.assertGreaterEqual(num_lines, expected) + @unittest.skip('causes undesireable side-effects (#20128)') def test_modules_search(self): # See Helper.listmodules(). expected = 'pydoc - ' @@ -635,7 +637,7 @@ self.assertIn(expected, result) - @unittest.skip('some buildbots are not cooperating (#20123)') + @unittest.skip('some buildbots are not cooperating (#20128)') def test_modules_search_builtin(self): expected = 'gc - ' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:52 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjQx?= =?utf-8?q?=3A_Run_custom_actions_with_the_NoImpersonate_flag_to_support_U?= =?utf-8?b?QUMu?= Message-ID: <3fnSPJ36NFz7LkR@mail.python.org> http://hg.python.org/cpython/rev/31c7dc7ccbaa changeset: 89764:31c7dc7ccbaa branch: 3.4 user: Martin v. L?wis date: Sat Feb 22 23:44:20 2014 +0100 summary: Issue #20641: Run custom actions with the NoImpersonate flag to support UAC. files: Misc/NEWS | 3 +++ Tools/msi/msi.py | 20 +++++++++----------- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -71,6 +71,9 @@ Build ----- +- Issue #20641: Run MSI custom actions (pip installation, pyc compilation) + with the NoImpersonate flag, to support elevated execution (UAC). + - Issue #20221: Removed conflicting (or circular) hypot definition when compiled with VS 2010 or above. Initial patch by Tabrez Mohammed. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -435,12 +435,13 @@ ("SetLauncherDirToWindows", 307, "LAUNCHERDIR", "[WindowsFolder]"), # msidbCustomActionTypeExe + msidbCustomActionTypeSourceFile # See "Custom Action Type 18" - ("CompilePyc", 18, "python.exe", compileargs), - ("CompilePyo", 18, "python.exe", "-O "+compileargs), - ("CompileGrammar", 18, "python.exe", lib2to3args), # msidbCustomActionTypeInScript (1024); run during actual installation - ("UpdatePip", 18+1024, "python.exe", updatepipargs), - ("RemovePip", 18, "python.exe", removepipargs), + # msidbCustomActionTypeNoImpersonate (2048); run action in system account, not user account + ("CompilePyc", 18+1024+2048, "python.exe", compileargs), + ("CompilePyo", 18+1024+2048, "python.exe", "-O "+compileargs), + ("CompileGrammar", 18+1024+2048, "python.exe", lib2to3args), + ("UpdatePip", 18+1024+2048, "python.exe", updatepipargs), + ("RemovePip", 18+1024+2048, "python.exe", removepipargs), ]) # UI Sequences, see "InstallUISequence Table", "Using a Sequence Table" @@ -483,17 +484,14 @@ # remove pip when state changes to INSTALLSTATE_ABSENT # run before RemoveFiles ("RemovePip", "&pip_feature=2", 3499), - ("CompilePyc", "COMPILEALL", 6800), - ("CompilePyo", "COMPILEALL", 6801), - ("CompileGrammar", "COMPILEALL", 6802), + ("CompilePyc", "COMPILEALL", 4002), + ("CompilePyo", "COMPILEALL", 4003), + ("CompileGrammar", "COMPILEALL", 4004), ]) add_data(db, "AdminExecuteSequence", [("InitialTargetDir", 'TARGETDIR=""', 750), ("SetDLLDirToTarget", 'DLLDIR=""', 751), ("SetLauncherDirToTarget", 'LAUNCHERDIR=""', 752), - ("CompilePyc", "COMPILEALL", 6800), - ("CompilePyo", "COMPILEALL", 6801), - ("CompileGrammar", "COMPILEALL", 6802), ]) ##################################################################### -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:53 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Regenerated_py?= =?utf-8?q?doc_topics_for_Python_3=2E4=2E0rc2=2E?= Message-ID: <3fnSPK5fBkz7LkF@mail.python.org> http://hg.python.org/cpython/rev/d15c9e903b75 changeset: 89765:d15c9e903b75 branch: 3.4 user: Larry Hastings date: Sun Feb 23 02:16:46 2014 -0600 summary: Regenerated pydoc topics for Python 3.4.0rc2. files: Lib/pydoc_data/topics.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Feb 10 04:20:03 2014 +# Autogenerated by Sphinx on Sun Feb 23 02:12:38 2014 topics = {'assert': '\nThe "assert" statement\n**********************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, "assert expression", is equivalent to\n\n if __debug__:\n if not expression: raise AssertionError\n\nThe extended form, "assert expression1, expression2", is equivalent to\n\n if __debug__:\n if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that "__debug__" and "AssertionError" refer\nto the built-in variables with those names. In the current\nimplementation, the built-in variable "__debug__" is "True" under\nnormal circumstances, "False" when optimization is requested (command\nline option -O). The current code generator emits no code for an\nassert statement when optimization is requested at compile time. Note\nthat it is unnecessary to include the source code for the expression\nthat failed in the error message; it will be displayed as part of the\nstack trace.\n\nAssignments to "__debug__" are illegal. The value for the built-in\nvariable is determined when the interpreter starts.\n', 'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n target_list ::= target ("," target)* [","]\n target ::= identifier\n | "(" target_list ")"\n | "[" target_list "]"\n | attributeref\n | subscription\n | slicing\n | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable. The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n that target.\n\n* If the target list is a comma-separated list of targets: The\n object must be an iterable with the same number of items as there\n are targets in the target list, and the items are assigned, from\n left to right, to the corresponding targets.\n\n * If the target list contains one target prefixed with an\n asterisk, called a "starred" target: The object must be a sequence\n with at least as many items as there are targets in the target\n list, minus one. The first items of the sequence are assigned,\n from left to right, to the targets before the starred target. The\n final items of the sequence are assigned to the targets after the\n starred target. A list of the remaining items in the sequence is\n then assigned to the starred target (the list can be empty).\n\n * Else: The object must be a sequence with the same number of\n items as there are targets in the target list, and the items are\n assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n * If the name does not occur in a "global" or "nonlocal" statement\n in the current code block: the name is bound to the object in the\n current local namespace.\n\n * Otherwise: the name is bound to the object in the global\n namespace or the outer namespace determined by "nonlocal",\n respectively.\n\n The name is rebound if it was already bound. This may cause the\n reference count for the object previously bound to the name to reach\n zero, causing the object to be deallocated and its destructor (if it\n has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in\n square brackets: The object must be an iterable with the same number\n of items as there are targets in the target list, and its items are\n assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n the reference is evaluated. It should yield an object with\n assignable attributes; if this is not the case, "TypeError" is\n raised. That object is then asked to assign the assigned object to\n the given attribute; if it cannot perform the assignment, it raises\n an exception (usually but not necessarily "AttributeError").\n\n Note: If the object is a class instance and the attribute reference\n occurs on both sides of the assignment operator, the RHS expression,\n "a.x" can access either an instance attribute or (if no instance\n attribute exists) a class attribute. The LHS target "a.x" is always\n set as an instance attribute, creating it if necessary. Thus, the\n two occurrences of "a.x" do not necessarily refer to the same\n attribute: if the RHS expression refers to a class attribute, the\n LHS creates a new instance attribute as the target of the\n assignment:\n\n class Cls:\n x = 3 # class variable\n inst = Cls()\n inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3\n\n This description does not necessarily apply to descriptor\n attributes, such as properties created with "property()".\n\n* If the target is a subscription: The primary expression in the\n reference is evaluated. It should yield either a mutable sequence\n object (such as a list) or a mapping object (such as a dictionary).\n Next, the subscript expression is evaluated.\n\n If the primary is a mutable sequence object (such as a list), the\n subscript must yield an integer. If it is negative, the sequence\'s\n length is added to it. The resulting value must be a nonnegative\n integer less than the sequence\'s length, and the sequence is asked\n to assign the assigned object to its item with that index. If the\n index is out of range, "IndexError" is raised (assignment to a\n subscripted sequence cannot add new items to a list).\n\n If the primary is a mapping object (such as a dictionary), the\n subscript must have a type compatible with the mapping\'s key type,\n and the mapping is then asked to create a key/datum pair which maps\n the subscript to the assigned object. This can either replace an\n existing key/value pair with the same key value, or insert a new\n key/value pair (if no key with the same value existed).\n\n For user-defined objects, the "__setitem__()" method is called with\n appropriate arguments.\n\n* If the target is a slicing: The primary expression in the\n reference is evaluated. It should yield a mutable sequence object\n (such as a list). The assigned object should be a sequence object\n of the same type. Next, the lower and upper bound expressions are\n evaluated, insofar they are present; defaults are zero and the\n sequence\'s length. The bounds should evaluate to integers. If\n either bound is negative, the sequence\'s length is added to it. The\n resulting bounds are clipped to lie between zero and the sequence\'s\n length, inclusive. Finally, the sequence object is asked to replace\n the slice with the items of the assigned sequence. The length of\n the slice may be different from the length of the assigned sequence,\n thus changing the length of the target sequence, if the object\n allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample "a, b = b, a" swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe! For instance, the\nfollowing program prints "[0, 2]":\n\n x = [0, 1]\n i = 0\n i, x[i] = 1, 2\n print(x)\n\nSee also: **PEP 3132** - Extended Iterable Unpacking\n\n The specification for the "*target" feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n augtarget ::= identifier | attributeref | subscription | slicing\n augop ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like "x += 1" can be rewritten as\n"x = x + 1" to achieve a similar, but not exactly equal effect. In the\naugmented version, "x" is only evaluated once. Also, when possible,\nthe actual operation is performed *in-place*, meaning that rather than\ncreating a new object and assigning that to the target, the old object\nis modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n', 'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name. See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a "NameError" exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them. The transformation inserts the\nclass name, with leading underscores removed and a single underscore\ninserted, in front of the name. For example, the identifier "__spam"\noccurring in a class named "Ham" will be transformed to "_Ham__spam".\nThis transformation is independent of the syntactical context in which\nthe identifier is used. If the transformed name is extremely long\n(longer than 255 characters), implementation defined truncation may\nhappen. If the class name consists only of underscores, no\ntransformation is done.\n', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:54 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Version_bump_f?= =?utf-8?q?or_Python_3=2E4=2E0rc2=2E?= Message-ID: <3fnSPL6zrBz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/a300712ed38c changeset: 89766:a300712ed38c branch: 3.4 tag: v3.4.0rc2 user: Larry Hastings date: Sun Feb 23 02:18:24 2014 -0600 summary: Version bump for Python 3.4.0rc2. files: Include/patchlevel.h | 4 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/RPM/python-3.4.spec | 2 +- README | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 4 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_SERIAL 2 /* Version as a string */ -#define PY_VERSION "3.4.0rc1+" +#define PY_VERSION "3.4.0rc2" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.4.0rc1" +__version__ = "3.4.0rc2" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.4.0rc1" +IDLE_VERSION = "3.4.0rc2" diff --git a/Misc/RPM/python-3.4.spec b/Misc/RPM/python-3.4.spec --- a/Misc/RPM/python-3.4.spec +++ b/Misc/RPM/python-3.4.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.4.0rc1 +%define version 3.4.0rc2 %define libvers 3.4 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,4 +1,4 @@ -This is Python version 3.4.0 release candidate 1 +This is Python version 3.4.0 release candidate 2 ================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:56 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Added_tag_v3?= =?utf-8?q?=2E4=2E0rc2_for_changeset_a300712ed38c?= Message-ID: <3fnSPN1CvBz7LkB@mail.python.org> http://hg.python.org/cpython/rev/c1f9fbefb237 changeset: 89767:c1f9fbefb237 branch: 3.4 user: Larry Hastings date: Sun Feb 23 02:18:41 2014 -0600 summary: Added tag v3.4.0rc2 for changeset a300712ed38c files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -128,3 +128,4 @@ ba32913eb13ec545a46dd0ce18035b6c416f0d78 v3.4.0b2 a97ce3ecc96af79bd2e1ac66ce48d9138e0ca749 v3.4.0b3 5e088cea8660677969113741c1313d570d977e02 v3.4.0rc1 +a300712ed38c9a242b736c44e806caea25a6dc05 v3.4.0rc2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:57 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5MjU1?= =?utf-8?q?=3A_The_builtins_module_is_restored_to_initial_value_before?= Message-ID: <3fnSPP4GPhz7LkW@mail.python.org> http://hg.python.org/cpython/rev/5d3b9862f1bc changeset: 89768:5d3b9862f1bc branch: 3.4 user: Serhiy Storchaka date: Mon Feb 10 18:21:34 2014 +0200 summary: Issue #19255: The builtins module is restored to initial value before cleaning other modules. The sys and builtins modules are cleaned last. files: Include/moduleobject.h | 1 + Include/pystate.h | 2 +- Lib/test/test_builtin.py | 29 +++++++++ Misc/NEWS | 3 + Objects/moduleobject.c | 13 ++- Python/import.c | 85 +++++++++++++++++---------- Python/pystate.c | 2 + 7 files changed, 96 insertions(+), 39 deletions(-) diff --git a/Include/moduleobject.h b/Include/moduleobject.h --- a/Include/moduleobject.h +++ b/Include/moduleobject.h @@ -25,6 +25,7 @@ PyAPI_FUNC(PyObject *) PyModule_GetFilenameObject(PyObject *); #ifndef Py_LIMITED_API PyAPI_FUNC(void) _PyModule_Clear(PyObject *); +PyAPI_FUNC(void) _PyModule_ClearDict(PyObject *); #endif PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*); PyAPI_FUNC(void*) PyModule_GetState(PyObject*); diff --git a/Include/pystate.h b/Include/pystate.h --- a/Include/pystate.h +++ b/Include/pystate.h @@ -33,7 +33,6 @@ int codecs_initialized; int fscodec_initialized; - #ifdef HAVE_DLOPEN int dlopenflags; #endif @@ -41,6 +40,7 @@ int tscdump; #endif + PyObject *builtins_copy; } PyInterpreterState; #endif diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -16,6 +16,7 @@ import warnings from operator import neg from test.support import TESTFN, unlink, run_unittest, check_warnings +from test.script_helper import assert_python_ok try: import pty, signal except ImportError: @@ -1592,6 +1593,34 @@ data = 'The quick Brown fox Jumped over The lazy Dog'.split() self.assertRaises(TypeError, sorted, data, None, lambda x,y: 0) + +class ShutdownTest(unittest.TestCase): + + def test_cleanup(self): + # Issue #19255: builtins are still available at shutdown + code = """if 1: + import builtins + import sys + + class C: + def __del__(self): + print("before") + # Check that builtins still exist + len(()) + print("after") + + c = C() + # Make this module survive until builtins and sys are cleaned + builtins.here = sys.modules[__name__] + sys.here = sys.modules[__name__] + # Create a reference loop so that this module needs to go + # through a GC phase. + here = sys.modules[__name__] + """ + rc, out, err = assert_python_ok("-c", code) + self.assertEqual(["before", "after"], out.decode().splitlines()) + + def load_tests(loader, tests, pattern): from doctest import DocTestSuite tests.addTest(DocTestSuite(builtins)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -89,6 +89,9 @@ Core and Builtins ----------------- +- Issue #19255: The builtins module is restored to initial value before + cleaning other modules. The sys and builtins modules are cleaned last. + - Issue #20437: Fixed 22 potential bugs when deleting objects references. - Issue #20500: Displaying an exception at interpreter shutdown no longer diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -299,6 +299,14 @@ void _PyModule_Clear(PyObject *m) { + PyObject *d = ((PyModuleObject *)m)->md_dict; + if (d != NULL) + _PyModule_ClearDict(d); +} + +void +_PyModule_ClearDict(PyObject *d) +{ /* To make the execution order of destructors for global objects a bit more predictable, we first zap all objects whose name starts with a single underscore, before we clear @@ -308,11 +316,6 @@ Py_ssize_t pos; PyObject *key, *value; - PyObject *d; - - d = ((PyModuleObject *)m)->md_dict; - if (d == NULL) - return; /* First, clear only names starting with a single underscore */ pos = 0; diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -49,9 +49,13 @@ void _PyImport_Init(void) { + PyInterpreterState *interp = PyThreadState_Get()->interp; initstr = PyUnicode_InternFromString("__init__"); if (initstr == NULL) Py_FatalError("Can't initialize import variables"); + interp->builtins_copy = PyDict_Copy(interp->builtins); + if (interp->builtins_copy == NULL) + Py_FatalError("Can't backup builtins dict"); } void @@ -397,8 +401,10 @@ PyObject *key, *value, *dict; PyInterpreterState *interp = PyThreadState_GET()->interp; PyObject *modules = interp->modules; - PyObject *builtins = interp->builtins; + PyObject *builtins_mod = NULL; + PyObject *sys_mod = NULL; PyObject *weaklist = NULL; + char **p; if (modules == NULL) return; /* Already done */ @@ -411,31 +417,22 @@ /* XXX Perhaps these precautions are obsolete. Who knows? */ - value = PyDict_GetItemString(modules, "builtins"); - if (value != NULL && PyModule_Check(value)) { - dict = PyModule_GetDict(value); + if (Py_VerboseFlag) + PySys_WriteStderr("# clear builtins._\n"); + PyDict_SetItemString(interp->builtins, "_", Py_None); + + for (p = sys_deletes; *p != NULL; p++) { if (Py_VerboseFlag) - PySys_WriteStderr("# clear builtins._\n"); - PyDict_SetItemString(dict, "_", Py_None); + PySys_WriteStderr("# clear sys.%s\n", *p); + PyDict_SetItemString(interp->sysdict, *p, Py_None); } - value = PyDict_GetItemString(modules, "sys"); - if (value != NULL && PyModule_Check(value)) { - char **p; - PyObject *v; - dict = PyModule_GetDict(value); - for (p = sys_deletes; *p != NULL; p++) { - if (Py_VerboseFlag) - PySys_WriteStderr("# clear sys.%s\n", *p); - PyDict_SetItemString(dict, *p, Py_None); - } - for (p = sys_files; *p != NULL; p+=2) { - if (Py_VerboseFlag) - PySys_WriteStderr("# restore sys.%s\n", *p); - v = PyDict_GetItemString(dict, *(p+1)); - if (v == NULL) - v = Py_None; - PyDict_SetItemString(dict, *p, v); - } + for (p = sys_files; *p != NULL; p+=2) { + if (Py_VerboseFlag) + PySys_WriteStderr("# restore sys.%s\n", *p); + value = PyDict_GetItemString(interp->sysdict, *(p+1)); + if (value == NULL) + value = Py_None; + PyDict_SetItemString(interp->sysdict, *p, value); } /* We prepare a list which will receive (name, weakref) tuples of @@ -473,11 +470,15 @@ /* Clear the modules dict. */ PyDict_Clear(modules); - /* Replace the interpreter's reference to builtins with an empty dict - (module globals still have a reference to the original builtins). */ - builtins = interp->builtins; - interp->builtins = PyDict_New(); - Py_DECREF(builtins); + /* Restore the original builtins dict, to ensure that any + user data gets cleared. */ + dict = PyDict_Copy(interp->builtins); + if (dict == NULL) + PyErr_Clear(); + PyDict_Clear(interp->builtins); + if (PyDict_Update(interp->builtins, interp->builtins_copy)) + PyErr_Clear(); + Py_XDECREF(dict); /* Clear module dict copies stored in the interpreter state */ _PyState_ClearModules(); /* Collect references */ @@ -488,7 +489,15 @@ /* Now, if there are any modules left alive, clear their globals to minimize potential leaks. All C extension modules actually end - up here, since they are kept alive in the interpreter state. */ + up here, since they are kept alive in the interpreter state. + + The special treatment of "builtins" here is because even + when it's not referenced as a module, its dictionary is + referenced by almost every module's __builtins__. Since + deleting a module clears its dictionary (even if there are + references left to it), we need to delete the "builtins" + module last. Likewise, we don't delete sys until the very + end because it is implicitly referenced (e.g. by print). */ if (weaklist != NULL) { Py_ssize_t i, n; n = PyList_GET_SIZE(weaklist); @@ -498,17 +507,27 @@ PyObject *mod = PyWeakref_GET_OBJECT(PyTuple_GET_ITEM(tup, 1)); if (mod == Py_None) continue; + assert(PyModule_Check(mod)); + dict = PyModule_GetDict(mod); + if (dict == interp->builtins || dict == interp->sysdict) + continue; Py_INCREF(mod); - assert(PyModule_Check(mod)); if (Py_VerboseFlag && PyUnicode_Check(name)) - PySys_FormatStderr("# cleanup[3] wiping %U\n", - name, mod); + PySys_FormatStderr("# cleanup[3] wiping %U\n", name); _PyModule_Clear(mod); Py_DECREF(mod); } Py_DECREF(weaklist); } + /* Next, delete sys and builtins (in that order) */ + if (Py_VerboseFlag) + PySys_FormatStderr("# cleanup[3] wiping sys\n"); + _PyModule_ClearDict(interp->sysdict); + if (Py_VerboseFlag) + PySys_FormatStderr("# cleanup[3] wiping builtins\n"); + _PyModule_ClearDict(interp->builtins); + /* Clear and delete the modules directory. Actual modules will still be there only if imported during the execution of some destructor. */ diff --git a/Python/pystate.c b/Python/pystate.c --- a/Python/pystate.c +++ b/Python/pystate.c @@ -72,6 +72,7 @@ interp->modules_by_index = NULL; interp->sysdict = NULL; interp->builtins = NULL; + interp->builtins_copy = NULL; interp->tstate_head = NULL; interp->codec_search_path = NULL; interp->codec_search_cache = NULL; @@ -115,6 +116,7 @@ Py_CLEAR(interp->modules_by_index); Py_CLEAR(interp->sysdict); Py_CLEAR(interp->builtins); + Py_CLEAR(interp->builtins_copy); Py_CLEAR(interp->importlib); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:32:58 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:32:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Temporary_sile?= =?utf-8?q?nce_test_broken_by_issue19255=2E?= Message-ID: <3fnSPQ6D3Sz7LkX@mail.python.org> http://hg.python.org/cpython/rev/a0bc45a50e43 changeset: 89769:a0bc45a50e43 branch: 3.4 user: Serhiy Storchaka date: Mon Feb 10 19:09:19 2014 +0200 summary: Temporary silence test broken by issue19255. Remove unused variables. files: Lib/test/test_io.py | 3 ++- Python/import.c | 2 -- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -2691,7 +2691,8 @@ class PyTextIOWrapperTest(TextIOWrapperTest): io = pyio - shutdown_error = "LookupError: unknown encoding: ascii" + #shutdown_error = "LookupError: unknown encoding: ascii" + shutdown_error = "TypeError: 'NoneType' object is not iterable" class IncrementalNewlineDecoderTest(unittest.TestCase): diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -401,8 +401,6 @@ PyObject *key, *value, *dict; PyInterpreterState *interp = PyThreadState_GET()->interp; PyObject *modules = interp->modules; - PyObject *builtins_mod = NULL; - PyObject *sys_mod = NULL; PyObject *weaklist = NULL; char **p; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:00 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5MjU1?= =?utf-8?q?=3A_Clear_error_after_failed_PyDict=5FSetItem=28=29_on_shutdown?= =?utf-8?q?=2E?= Message-ID: <3fnSPS0Xbgz7Lkd@mail.python.org> http://hg.python.org/cpython/rev/af8f856db136 changeset: 89770:af8f856db136 branch: 3.4 user: Serhiy Storchaka date: Wed Feb 12 09:55:45 2014 +0200 summary: Issue #19255: Clear error after failed PyDict_SetItem() on shutdown. This silences a Coverity complain. files: Objects/moduleobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -330,7 +330,8 @@ else PyErr_Clear(); } - PyDict_SetItem(d, key, Py_None); + if (PyDict_SetItem(d, key, Py_None) != 0) + PyErr_Clear(); } } } @@ -349,7 +350,8 @@ else PyErr_Clear(); } - PyDict_SetItem(d, key, Py_None); + if (PyDict_SetItem(d, key, Py_None) != 0) + PyErr_Clear(); } } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:01 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Try_to_fix_tes?= =?utf-8?q?t=5Fcleanup_=28issue_=2320599=29=2E?= Message-ID: <3fnSPT2Qlyz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/3d5154fa8413 changeset: 89771:3d5154fa8413 branch: 3.4 user: Serhiy Storchaka date: Wed Feb 12 12:40:22 2014 +0200 summary: Try to fix test_cleanup (issue #20599). files: Lib/test/test_builtin.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1604,10 +1604,10 @@ class C: def __del__(self): - print("before") + print("before", flush=True) # Check that builtins still exist len(()) - print("after") + print("after", flush=True) c = C() # Make this module survive until builtins and sys are cleaned -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:02 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTk5?= =?utf-8?q?=3A_Force_ASCII_encoding_for_stdout_in_test=5Fcleanup=28=29_of_?= =?utf-8?q?test=5Fbuiltin?= Message-ID: <3fnSPV4sQtz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/c978dffb95ac changeset: 89772:c978dffb95ac branch: 3.4 user: Victor Stinner date: Wed Feb 12 18:27:55 2014 +0100 summary: Issue #20599: Force ASCII encoding for stdout in test_cleanup() of test_builtin On Windows, the codec of sys.stdout is implemented in Python. At exit, the codec may be unloaded before the destructor tries to write something to sys.stdout. files: Lib/test/test_builtin.py | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1604,10 +1604,10 @@ class C: def __del__(self): - print("before", flush=True) + print("before") # Check that builtins still exist len(()) - print("after", flush=True) + print("after") c = C() # Make this module survive until builtins and sys are cleaned @@ -1617,7 +1617,15 @@ # through a GC phase. here = sys.modules[__name__] """ - rc, out, err = assert_python_ok("-c", code) + # Issue #20599: Force ASCII encoding to get a codec implemented in C, + # otherwise the codec may be unloaded before C.__del__() is called, and + # so print("before") fails because the codec cannot be used to encode + # "before" to sys.stdout.encoding. For example, on Windows, + # sys.stdout.encoding is the OEM code page and these code pages are + # implemented in Python + rc, out, err = assert_python_ok("-c", code, + PYTHONIOENCODING="ascii", + __cleanenv=True) self.assertEqual(["before", "after"], out.decode().splitlines()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:03 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTk5?= =?utf-8?q?=3A_Don=27t_clear_environment_in_test=5Fcleanup=28=29_of_test?= =?utf-8?q?=5Fbuiltin?= Message-ID: <3fnSPW6vBnz7LkL@mail.python.org> http://hg.python.org/cpython/rev/1fb37d2d7d9d changeset: 89773:1fb37d2d7d9d branch: 3.4 user: Victor Stinner date: Thu Feb 13 10:54:32 2014 +0100 summary: Issue #20599: Don't clear environment in test_cleanup() of test_builtin files: Lib/test/test_builtin.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1624,8 +1624,7 @@ # sys.stdout.encoding is the OEM code page and these code pages are # implemented in Python rc, out, err = assert_python_ok("-c", code, - PYTHONIOENCODING="ascii", - __cleanenv=True) + PYTHONIOENCODING="ascii") self.assertEqual(["before", "after"], out.decode().splitlines()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:05 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNzYz?= =?utf-8?q?=3A_Fix_importlib=2Emachinery=2EPathFinder_to_support?= Message-ID: <3fnSPY1lRFz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/149be7000c11 changeset: 89774:149be7000c11 branch: 3.4 user: Brett Cannon date: Wed Feb 26 18:26:49 2014 -0500 summary: Issue #20763: Fix importlib.machinery.PathFinder to support PathEntryFinder instances which only define find_module(). Reported by Yukihiro Nakadaira. files: Lib/importlib/_bootstrap.py | 2 +- Lib/test/test_importlib/import_/test_path.py | 24 ++++++++++ Python/importlib.h | 2 +- 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1869,7 +1869,7 @@ loader, portions = finder.find_loader(fullname) else: loader = finder.find_module(fullname) - portions = None + portions = [] if loader is not None: return spec_from_loader(fullname, loader) spec = ModuleSpec(fullname, None) diff --git a/Lib/test/test_importlib/import_/test_path.py b/Lib/test/test_importlib/import_/test_path.py --- a/Lib/test/test_importlib/import_/test_path.py +++ b/Lib/test/test_importlib/import_/test_path.py @@ -116,5 +116,29 @@ FinderTests, importlib=importlib, machinery=machinery) +class PathEntryFinderTests: + + def test_finder_with_failing_find_module(self): + # PathEntryFinder with find_module() defined should work. + # Issue #20763. + class Finder: + path_location = 'test_finder_with_find_module' + def __init__(self, path): + if path != self.path_location: + raise ImportError + + @staticmethod + def find_module(fullname): + return None + + + with util.import_state(path=[Finder.path_location]+sys.path[:], + path_hooks=[Finder]): + self.machinery.PathFinder.find_spec('importlib') + +Frozen_PEFTests, Source_PEFTests = util.test_both( + PathEntryFinderTests, machinery=machinery) + + if __name__ == '__main__': unittest.main() diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:06 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwNzU3?= =?utf-8?q?=3A_return_success_for_skipped_pip_uninstall?= Message-ID: <3fnSPZ3Ynwz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/6d0994805e18 changeset: 89775:6d0994805e18 branch: 3.4 user: Nick Coghlan date: Fri Feb 28 23:35:05 2014 +1000 summary: Close #20757: return success for skipped pip uninstall The 3.4rc2 Windows uninstaller would fail if pip had been updated to a version that didn't match the version installed by ensurepip. This skip is no longer treated as an error, so an updated pip ends up being handled like any other pip installed package and is left alone by the CPython uninstaller. files: Lib/ensurepip/__init__.py | 5 +++-- Lib/test/test_ensurepip.py | 6 ++++-- Misc/NEWS | 14 ++++++++++++++ 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -128,9 +128,10 @@ # If the pip version doesn't match the bundled one, leave it alone if pip.__version__ != _PIP_VERSION: - msg = ("ensurepip will only uninstall a matching pip " + msg = ("ensurepip will only uninstall a matching version " "({!r} installed, {!r} bundled)") - raise RuntimeError(msg.format(pip.__version__, _PIP_VERSION)) + print(msg.format(pip.__version__, _PIP_VERSION), file=sys.stderr) + return _require_ssl_for_pip() _disable_pip_configuration_settings() diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -196,10 +196,12 @@ ensurepip._uninstall_helper() self.run_pip.assert_not_called() - def test_uninstall_fails_with_wrong_version(self): + def test_uninstall_skipped_with_warning_for_wrong_version(self): with fake_pip("not a valid version"): - with self.assertRaises(RuntimeError): + with test.support.captured_stderr() as stderr: ensurepip._uninstall_helper() + warning = stderr.getvalue().strip() + self.assertIn("only uninstall a matching version", warning) self.run_pip.assert_not_called() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,20 @@ Python News +++++++++++ +What's New in Python 3.4.0 release candidate 3? +=============================================== + +Release date: 2014-03-09 + +Build +----- + +- Issue #20757: The ensurepip helper for the Windows uninstaller now skips + uninstalling pip (rather than failing) if the user has updated pip to a + different version from the one bundled with ensurepip. + + + What's New in Python 3.4.0 release candidate 2? =============================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:07 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwNTY4?= =?utf-8?q?=3A_install_unversioned_pip_command_on_Windows?= Message-ID: <3fnSPb5377z7LkP@mail.python.org> http://hg.python.org/cpython/rev/5d190cca2f47 changeset: 89776:5d190cca2f47 branch: 3.4 user: Nick Coghlan date: Fri Feb 28 23:37:35 2014 +1000 summary: Close #20568: install unversioned pip command on Windows files: Misc/NEWS | 3 +++ Tools/msi/msi.py | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Build ----- +- Issue #20568: The Windows installer now installs the unversioned ``pip`` + command in addition to the versioned ``pip3`` and ``pip3.4`` commands. + - Issue #20757: The ensurepip helper for the Windows uninstaller now skips uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -420,7 +420,7 @@ compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"' lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"' - updatepipargs = r'-m ensurepip -U' + updatepipargs = r'-m ensurepip -U --default-install' removepipargs = r'-m ensurepip._uninstall' # See "CustomAction Table" add_data(db, "CustomAction", [ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:08 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDY1?= =?utf-8?q?=3A_Update_OS_X_installer_build_to_use_SQLite_3=2E8=2E0=2E1=2E?= Message-ID: <3fnSPc6Sq7z7LkF@mail.python.org> http://hg.python.org/cpython/rev/7260dffad495 changeset: 89777:7260dffad495 branch: 3.4 user: Ned Deily date: Sat Mar 01 14:00:46 2014 -0800 summary: Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. files: Mac/BuildScript/build-installer.py | 6 +++--- Misc/NEWS | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -283,9 +283,9 @@ ), ), dict( - name="SQLite 3.8.3", - url="http://www.sqlite.org/2014/sqlite-autoconf-3080300.tar.gz", - checksum='11572878dc0ac74ae370367a464ab5cf', + name="SQLite 3.8.3.1", + url="http://www.sqlite.org/2014/sqlite-autoconf-3080301.tar.gz", + checksum='509ff98d8dc9729b618b7e96612079c6', extra_cflags=('-Os ' '-DSQLITE_ENABLE_FTS4 ' '-DSQLITE_ENABLE_FTS3_PARENTHESIS ' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -17,6 +17,7 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. +- Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. What's New in Python 3.4.0 release candidate 2? -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:10 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDY1?= =?utf-8?b?OiBmaXggTkVXUyB0eXBvIChpdCdzIDMuOC4zLjEp?= Message-ID: <3fnSPf0txcz7LjV@mail.python.org> http://hg.python.org/cpython/rev/1f922e9742b7 changeset: 89778:1f922e9742b7 branch: 3.4 user: Ned Deily date: Sat Mar 01 14:04:48 2014 -0800 summary: Issue #20465: fix NEWS typo (it's 3.8.3.1) files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -17,7 +17,7 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X installer build to use SQLite 3.8.0.1. +- Issue #20465: Update OS X installer build to use SQLite 3.8.3.1. What's New in Python 3.4.0 release candidate 2? -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:11 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNzg2?= =?utf-8?q?=3A_Fix_signatures_for_dict=2E=5F=5Fdelitem=5F=5F_and_property?= =?utf-8?b?Ll9fZGVsZXRlX18=?= Message-ID: <3fnSPg2SYxz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/7ad0e19cc682 changeset: 89779:7ad0e19cc682 branch: 3.4 user: Yury Selivanov date: Sun Mar 02 12:25:27 2014 -0500 summary: Issue #20786: Fix signatures for dict.__delitem__ and property.__delete__ files: Lib/test/test_inspect.py | 5 +++++ Misc/NEWS | 7 ++++++- Objects/typeobject.c | 4 ++-- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1764,6 +1764,11 @@ __call__ = type test_callable(ThisWorksNow()) + # Regression test for issue #20786 + test_unbound_method(dict.__delitem__) + test_unbound_method(property.__delete__) + + @cpython_only @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -4,9 +4,14 @@ What's New in Python 3.4.0 release candidate 3? =============================================== - Release date: 2014-03-09 +Core and Builtins +----------------- + +- Issue #20786: Fix signatures for dict.__delitem__ and + property.__delete__ builtins. + Build ----- diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -6189,7 +6189,7 @@ "__set__($self, instance, value, /)\n--\n\nSet an attribute of instance to value."), TPSLOT("__delete__", tp_descr_set, slot_tp_descr_set, wrap_descr_delete, - "__delete__(instance, /)\n--\n\nDelete an attribute of instance."), + "__delete__($self, instance, /)\n--\n\nDelete an attribute of instance."), FLSLOT("__init__", tp_init, slot_tp_init, (wrapperfunc)wrap_init, "__init__($self, /, *args, **kwargs)\n--\n\n" "Initialize self. See help(type(self)) for accurate signature.", @@ -6286,7 +6286,7 @@ "__setitem__($self, key, value, /)\n--\n\nSet self[key] to value."), MPSLOT("__delitem__", mp_ass_subscript, slot_mp_ass_subscript, wrap_delitem, - "__delitem__(key)\n--\n\nDelete self[key]."), + "__delitem__($self, key, /)\n--\n\nDelete self[key]."), SQSLOT("__len__", sq_length, slot_sq_length, wrap_lenfunc, "__len__($self, /)\n--\n\nReturn len(self)."), -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:12 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTY4?= =?utf-8?q?=3A_Fix_typo_in_pip_option=2E?= Message-ID: <3fnSPh4Ctpz7LkL@mail.python.org> http://hg.python.org/cpython/rev/3f8ca67bde4d changeset: 89780:3f8ca67bde4d branch: 3.4 user: Martin v. L?wis date: Sun Mar 02 19:15:47 2014 +0100 summary: Issue #20568: Fix typo in pip option. files: Tools/msi/msi.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -420,7 +420,7 @@ compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"' lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"' - updatepipargs = r'-m ensurepip -U --default-install' + updatepipargs = r'-m ensurepip -U --default-pip' removepipargs = r'-m ensurepip._uninstall' # See "CustomAction Table" add_data(db, "CustomAction", [ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:13 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIElzc3VlICMyMDc0?= =?utf-8?q?8=3A_Uninstalling_pip_does_not_leave_behind_the_pyc_of?= Message-ID: <3fnSPj6KF6z7Lk8@mail.python.org> http://hg.python.org/cpython/rev/b018a33c7db4 changeset: 89781:b018a33c7db4 branch: 3.4 user: Martin v. L?wis date: Sun Mar 02 19:29:19 2014 +0100 summary: Issue #20748: Uninstalling pip does not leave behind the pyc of the uninstaller anymore. files: Misc/NEWS | 3 +++ Tools/msi/msi.py | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Build ----- +- Issue #20748: Uninstalling pip does not leave behind the pyc of + the uninstaller anymore. + - Issue #20568: The Windows installer now installs the unversioned ``pip`` command in addition to the versioned ``pip3`` and ``pip3.4`` commands. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -421,7 +421,7 @@ compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"' lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"' updatepipargs = r'-m ensurepip -U --default-pip' - removepipargs = r'-m ensurepip._uninstall' + removepipargs = r'-B -m ensurepip._uninstall' # See "CustomAction Table" add_data(db, "CustomAction", [ # msidbCustomActionTypeFirstSequence + msidbCustomActionTypeTextData + msidbCustomActionTypeProperty -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:15 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDY1?= =?utf-8?q?=3A_Update_Windows_installer_to_SQLite_3=2E8=2E3=2E1=2E?= Message-ID: <3fnSPl16xLz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/9be3bf7b5b8d changeset: 89782:9be3bf7b5b8d branch: 3.4 user: Martin v. L?wis date: Sun Mar 02 19:42:50 2014 +0100 summary: Issue #20465: Update Windows installer to SQLite 3.8.3.1. files: Misc/NEWS | 3 ++- PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/pyproject.props | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 6 +++--- 5 files changed, 8 insertions(+), 7 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -25,7 +25,8 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X installer build to use SQLite 3.8.3.1. +- Issue #20465: Update OS X and Windows installer builds to use + SQLite 3.8.3.1. What's New in Python 3.4.0 release candidate 2? diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -50,7 +50,7 @@ /> $(OutDir)python$(PyDebugExt).exe $(OutDir)kill_python$(PyDebugExt).exe ..\.. - $(externalsDir)\sqlite-3.8.1 + $(externalsDir)\sqlite-3.8.3.1 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.5 $(externalsDir)\openssl-1.0.1e diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -206,7 +206,7 @@ The ssl sub-project does not have the ability to clean the OpenSSL build; if you need to rebuild, you'll have to clean it by hand. _sqlite3 - Wraps SQLite 3.8.1, which is itself built by sqlite3.vcxproj + Wraps SQLite 3.8.3.1, which is itself built by sqlite3.vcxproj Homepage: http://www.sqlite.org/ _tkinter diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -37,9 +37,9 @@ if not exist tk-8.6.1.0 svn export http://svn.python.org/projects/external/tk-8.6.1.0 @rem sqlite3 -if not exist sqlite-3.8.1 ( - rd /s/q sqlite-source-3.7.12 - svn export http://svn.python.org/projects/external/sqlite-3.8.1 +if not exist sqlite-3.8.3.1 ( + rd /s/q sqlite-source-3.8.1 + svn export http://svn.python.org/projects/external/sqlite-3.8.3.1 ) @rem lzma -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:16 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE0NTEy?= =?utf-8?q?=3A_Launch_pydoc_-b_instead_of_pydocgui=2Epyw_on_Windows=2E?= Message-ID: <3fnSPm2rntz7LkV@mail.python.org> http://hg.python.org/cpython/rev/fa34ea4e9e47 changeset: 89783:fa34ea4e9e47 branch: 3.4 user: Martin v. L?wis date: Sun Mar 02 20:29:18 2014 +0100 summary: Issue #14512: Launch pydoc -b instead of pydocgui.pyw on Windows. files: Misc/NEWS | 2 ++ Tools/msi/msi.py | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Build ----- +- Issue #14512: Launch pydoc -b instead of pydocgui.pyw on Windows. + - Issue #20748: Uninstalling pip does not leave behind the pyc of the uninstaller anymore. diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -1324,8 +1324,6 @@ tcltkshortcuts = [ ("IDLE", "MenuDir", "IDLE|IDLE (Python GUI)", "pythonw.exe", tcltk.id, r'"[TARGETDIR]Lib\idlelib\idle.pyw"', None, None, "python_icon.exe", 0, None, "TARGETDIR"), - ("PyDoc", "MenuDir", "MODDOCS|Module Docs", "pythonw.exe", - tcltk.id, r'"[TARGETDIR]Tools\scripts\pydocgui.pyw"', None, None, "python_icon.exe", 0, None, "TARGETDIR"), ] add_data(db, "Shortcut", tcltkshortcuts + @@ -1340,6 +1338,8 @@ ("Manual", "MenuDir", "MANUAL|Python Manuals", "REGISTRY.doc", "[#%s]" % docfile, None, None, None, None, None, None, None), + ("PyDoc", "MenuDir", "MODDOCS|Module Docs", "python.exe", + default_feature.id, r'-m pydoc -b', None, None, "python_icon.exe", 0, None, "TARGETDIR"), ("Uninstall", "MenuDir", "UNINST|Uninstall Python", "REGISTRY", SystemFolderName+"msiexec", "/x%s" % product_code, None, None, None, None, None, None), -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:17 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwODE0?= =?utf-8?q?=3A_doc=3A_Fix_=22Pretty_top=22_example_of_tracemalloc?= Message-ID: <3fnSPn47lZz7LkL@mail.python.org> http://hg.python.org/cpython/rev/027ca483a7e5 changeset: 89784:027ca483a7e5 branch: 3.4 user: Victor Stinner date: Mon Mar 03 11:57:57 2014 +0100 summary: Close #20814: doc: Fix "Pretty top" example of tracemalloc files: Doc/library/tracemalloc.rst | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -200,8 +200,7 @@ # replace "/path/to/module/file.py" with "module/file.py" filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" - % (index, filename, frame.lineno, - stat.size / 1024)) + % (index, filename, frame.lineno, stat.size / 1024)) other = top_stats[limit:] if other: @@ -215,7 +214,7 @@ # ... run your application ... snapshot = tracemalloc.take_snapshot() - display_top(snapshot, 10) + display_top(snapshot) Example of output of the Python test suite:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:18 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Close_issue206?= =?utf-8?q?53=3A_improve_functional_API_docs=3B_minor_code_changes?= Message-ID: <3fnSPp6YY9z7LkZ@mail.python.org> http://hg.python.org/cpython/rev/737f2be5e80c changeset: 89785:737f2be5e80c branch: 3.4 user: Ethan Furman date: Mon Mar 03 12:42:52 2014 -0800 summary: Close issue20653: improve functional API docs; minor code changes files: Doc/library/enum.rst | 34 +++++++++++++++++++++++++++ Lib/enum.py | 39 ++++++++++++++++++++++--------- 2 files changed, 61 insertions(+), 12 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -374,6 +374,9 @@ With pickle protocol version 4 it is possible to easily pickle enums nested in other classes. +It is possible to modify how Enum members are pickled/unpickled by defining +:meth:`__reduce_ex__` in the enumeration class. + Functional API -------------- @@ -420,6 +423,12 @@ >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__) +.. warning:: + + If :param module: is not supplied, and Enum cannot determine what it is, + the new Enum members will not be unpicklable; to keep errors closer to + the source, pickling will be disabled. + The new pickle protocol 4 also, in some circumstances, relies on :attr:`__qualname__` being set to the location where pickle will be able to find the class. For example, if the class was made available in class @@ -427,6 +436,31 @@ >>> Animals = Enum('Animals', 'ant bee cat dog', qualname='SomeData.Animals') +The complete signature is:: + + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=) + +:param value: What the new Enum class will record as its name. + +:param names: The Enum members. This can be a whitespace or comma seperated +string:: + + 'red green blue', 'red,green,blue', 'red, green, blue' + +(values will start at 1), or an iterator of name, value pairs:: + + [('cyan', 4), ('magenta', 5), ('yellow', 6)] + +or a mapping:: + + {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + +:param module: name of module where new Enum class can be found. + +:param qualname: where in module new Enum class can be found. + +:param type: type to mix in to new Enum class. + Derived Enumerations -------------------- diff --git a/Lib/enum.py b/Lib/enum.py --- a/Lib/enum.py +++ b/Lib/enum.py @@ -115,14 +115,21 @@ # Reverse value->name map for hashable values. enum_class._value2member_map_ = {} - # check for a supported pickle protocols, and if not present sabotage - # pickling, since it won't work anyway. - # if new class implements its own __reduce_ex__, do not sabotage - if classdict.get('__reduce_ex__') is None: + # If a custom type is mixed into the Enum, and it does not know how + # to pickle itself, pickle.dumps will succeed but pickle.loads will + # fail. Rather than have the error show up later and possibly far + # from the source, sabotage the pickle protocol for this class so + # that pickle.dumps also fails. + # + # However, if the new class implements its own __reduce_ex__, do not + # sabotage -- it's on them to make sure it works correctly. We use + # __reduce_ex__ instead of any of the others as it is preferred by + # pickle over __reduce__, and it handles all pickle protocols. + if '__reduce_ex__' not in classdict: if member_type is not object: methods = ('__getnewargs_ex__', '__getnewargs__', '__reduce_ex__', '__reduce__') - if not any(map(member_type.__dict__.get, methods)): + if not any(m in member_type.__dict__ for m in methods): _make_class_unpicklable(enum_class) # instantiate them, checking for duplicates as we go @@ -193,14 +200,22 @@ to an enumeration member (i.e. Color(3)) and for the functional API (i.e. Color = Enum('Color', names='red green blue')). - When used for the functional API: `module`, if set, will be stored in - the new class' __module__ attribute; `qualname`, if set, will be stored - in the new class' __qualname__ attribute; `type`, if set, will be mixed - in as the first base class. + When used for the functional API: - Note: if `module` is not set this routine will attempt to discover the - calling module by walking the frame stack; if this is unsuccessful - the resulting class will not be pickleable. + `value` will be the name of the new class. + + `names` should be either a string of white-space/comma delimited names + (values will start at 1), or an iterator/mapping of name, value pairs. + + `module` should be set to the module this class is being created in; + if it is not set, an attempt to find that module will be made, but if + it fails the class will not be picklable. + + `qualname` should be set to the actual location this class can be found + at in its module; by default it is set to the global scope. If this is + not correct, unpickling will fail in some circumstances. + + `type`, if set, will be mixed in as the first base class. """ if names is None: # simple value lookup -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:20 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUyMDY1Mzog?= =?utf-8?q?fix_ReST_for_Enum?= Message-ID: <3fnSPr15Qjz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/2c5a5fa0692c changeset: 89786:2c5a5fa0692c branch: 3.4 user: Ethan Furman date: Mon Mar 03 15:02:04 2014 -0800 summary: Issue20653: fix ReST for Enum files: Doc/library/enum.rst | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -425,7 +425,7 @@ .. warning:: - If :param module: is not supplied, and Enum cannot determine what it is, + If ``module`` is not supplied, and Enum cannot determine what it is, the new Enum members will not be unpicklable; to keep errors closer to the source, pickling will be disabled. @@ -440,26 +440,26 @@ Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=) -:param value: What the new Enum class will record as its name. +:value: What the new Enum class will record as its name. -:param names: The Enum members. This can be a whitespace or comma seperated -string:: +:names: The Enum members. This can be a whitespace or comma seperated string + (values will start at 1):: - 'red green blue', 'red,green,blue', 'red, green, blue' + 'red green blue' | 'red,green,blue' | 'red, green, blue' -(values will start at 1), or an iterator of name, value pairs:: + or an iterator of (name, value) pairs:: [('cyan', 4), ('magenta', 5), ('yellow', 6)] -or a mapping:: + or a mapping:: {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} -:param module: name of module where new Enum class can be found. +:module: name of module where new Enum class can be found. -:param qualname: where in module new Enum class can be found. +:qualname: where in module new Enum class can be found. -:param type: type to mix in to new Enum class. +:type: type to mix in to new Enum class. Derived Enumerations -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:21 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIwODM5?= =?utf-8?q?=3A_pkgutil=2Efind=5Floader_now_uses_importlib=2Eutil=2Efind=5F?= =?utf-8?q?spec?= Message-ID: <3fnSPs47Wcz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/a8b30fd6ee4f changeset: 89787:a8b30fd6ee4f branch: 3.4 user: Nick Coghlan date: Tue Mar 04 20:39:42 2014 +1000 summary: Close #20839: pkgutil.find_loader now uses importlib.util.find_spec files: Doc/library/pkgutil.rst | 17 +++++++++-------- Lib/pkgutil.py | 17 +++++------------ Lib/test/test_pkgutil.py | 19 +++++++++++++++++++ Misc/NEWS | 9 ++++++++- 4 files changed, 41 insertions(+), 21 deletions(-) diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -74,15 +74,17 @@ Retrieve a :pep:`302` module loader for the given *fullname*. - This is a convenience wrapper around :func:`importlib.find_loader` that - sets the *path* argument correctly when searching for submodules, and - also ensures parent packages (if any) are imported before searching for - submodules. + This is a backwards compatibility wrapper around + :func:`importlib.util.find_spec` that converts most failures to + :exc:`ImportError` and only returns the loader rather than the full + :class:`ModuleSpec`. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. + .. versionchanged:: 3.4 + Updated to be based on :pep:`451` .. function:: get_importer(path_item) @@ -109,14 +111,13 @@ not already imported, its containing package (if any) is imported, in order to establish the package ``__path__``. - This function uses :func:`iter_importers`, and is thus subject to the same - limitations regarding platform-specific special import locations such as the - Windows registry. - .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. + .. versionchanged:: 3.4 + Updated to be based on :pep:`451` + .. function:: iter_importers(fullname='') diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py --- a/Lib/pkgutil.py +++ b/Lib/pkgutil.py @@ -470,29 +470,22 @@ def find_loader(fullname): """Find a PEP 302 "loader" object for fullname - This is s convenience wrapper around :func:`importlib.find_loader` that - sets the *path* argument correctly when searching for submodules, and - also ensures parent packages (if any) are imported before searching for - submodules. + This is a backwards compatibility wrapper around + importlib.util.find_spec that converts most failures to ImportError + and only returns the loader rather than the full spec """ if fullname.startswith('.'): msg = "Relative module name {!r} not supported".format(fullname) raise ImportError(msg) - path = None - pkg_name = fullname.rpartition(".")[0] - if pkg_name: - pkg = importlib.import_module(pkg_name) - path = getattr(pkg, "__path__", None) - if path is None: - return None try: - return importlib.find_loader(fullname, path) + spec = importlib.util.find_spec(fullname) except (ImportError, AttributeError, TypeError, ValueError) as ex: # This hack fixes an impedance mismatch between pkgutil and # importlib, where the latter raises other errors for cases where # pkgutil previously raised ImportError msg = "Error while finding loader for {!r} ({}: {})" raise ImportError(msg.format(fullname, type(ex), ex)) from ex + return spec.loader def extend_path(path, name): diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -334,6 +334,25 @@ self.assertIsNotNone(pkgutil.get_loader("test.support")) self.assertEqual(len(w.warnings), 0) + def test_get_loader_handles_missing_loader_attribute(self): + global __loader__ + this_loader = __loader__ + del __loader__ + try: + with check_warnings() as w: + self.assertIsNotNone(pkgutil.get_loader(__name__)) + self.assertEqual(len(w.warnings), 0) + finally: + __loader__ = this_loader + + + def test_find_loader_avoids_emulation(self): + with check_warnings() as w: + self.assertIsNotNone(pkgutil.find_loader("sys")) + self.assertIsNotNone(pkgutil.find_loader("os")) + self.assertIsNotNone(pkgutil.find_loader("test.support")) + self.assertEqual(len(w.warnings), 0) + def test_get_importer_avoids_emulation(self): # We use an illegal path so *none* of the path hooks should fire with check_warnings() as w: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,13 @@ - Issue #20786: Fix signatures for dict.__delitem__ and property.__delete__ builtins. +Library +------- + +- Issue #20839: Don't trigger a DeprecationWarning in the still supported + pkgutil.get_loader() API when __loader__ isn't set on a module (nor + when pkgutil.find_loader() is called directly). + Build ----- @@ -27,7 +34,7 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X and Windows installer builds to use +- Issue #20465: Update OS X and Windows installer builds to use SQLite 3.8.3.1. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:22 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_pydoc_t?= =?utf-8?q?opics_and_suspicious_filters=2E?= Message-ID: <3fnSPt6dJLz7LjV@mail.python.org> http://hg.python.org/cpython/rev/c97d80985b71 changeset: 89788:c97d80985b71 branch: 3.4 user: Larry Hastings date: Sun Mar 09 04:12:12 2014 -0700 summary: Update pydoc topics and suspicious filters. files: Doc/tools/sphinxext/susp-ignored.csv | 1 - Lib/pydoc_data/topics.py | 2 +- 2 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -282,4 +282,3 @@ whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" whatsnew/changelog,,:password,user:password -whatsnew/changelog,,:gz,w:gz diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Sun Feb 23 02:12:38 2014 +# Autogenerated by Sphinx on Sun Mar 9 03:59:56 2014 topics = {'assert': '\nThe "assert" statement\n**********************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, "assert expression", is equivalent to\n\n if __debug__:\n if not expression: raise AssertionError\n\nThe extended form, "assert expression1, expression2", is equivalent to\n\n if __debug__:\n if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that "__debug__" and "AssertionError" refer\nto the built-in variables with those names. In the current\nimplementation, the built-in variable "__debug__" is "True" under\nnormal circumstances, "False" when optimization is requested (command\nline option -O). The current code generator emits no code for an\nassert statement when optimization is requested at compile time. Note\nthat it is unnecessary to include the source code for the expression\nthat failed in the error message; it will be displayed as part of the\nstack trace.\n\nAssignments to "__debug__" are illegal. The value for the built-in\nvariable is determined when the interpreter starts.\n', 'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n target_list ::= target ("," target)* [","]\n target ::= identifier\n | "(" target_list ")"\n | "[" target_list "]"\n | attributeref\n | subscription\n | slicing\n | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable. The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n that target.\n\n* If the target list is a comma-separated list of targets: The\n object must be an iterable with the same number of items as there\n are targets in the target list, and the items are assigned, from\n left to right, to the corresponding targets.\n\n * If the target list contains one target prefixed with an\n asterisk, called a "starred" target: The object must be a sequence\n with at least as many items as there are targets in the target\n list, minus one. The first items of the sequence are assigned,\n from left to right, to the targets before the starred target. The\n final items of the sequence are assigned to the targets after the\n starred target. A list of the remaining items in the sequence is\n then assigned to the starred target (the list can be empty).\n\n * Else: The object must be a sequence with the same number of\n items as there are targets in the target list, and the items are\n assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n * If the name does not occur in a "global" or "nonlocal" statement\n in the current code block: the name is bound to the object in the\n current local namespace.\n\n * Otherwise: the name is bound to the object in the global\n namespace or the outer namespace determined by "nonlocal",\n respectively.\n\n The name is rebound if it was already bound. This may cause the\n reference count for the object previously bound to the name to reach\n zero, causing the object to be deallocated and its destructor (if it\n has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in\n square brackets: The object must be an iterable with the same number\n of items as there are targets in the target list, and its items are\n assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n the reference is evaluated. It should yield an object with\n assignable attributes; if this is not the case, "TypeError" is\n raised. That object is then asked to assign the assigned object to\n the given attribute; if it cannot perform the assignment, it raises\n an exception (usually but not necessarily "AttributeError").\n\n Note: If the object is a class instance and the attribute reference\n occurs on both sides of the assignment operator, the RHS expression,\n "a.x" can access either an instance attribute or (if no instance\n attribute exists) a class attribute. The LHS target "a.x" is always\n set as an instance attribute, creating it if necessary. Thus, the\n two occurrences of "a.x" do not necessarily refer to the same\n attribute: if the RHS expression refers to a class attribute, the\n LHS creates a new instance attribute as the target of the\n assignment:\n\n class Cls:\n x = 3 # class variable\n inst = Cls()\n inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3\n\n This description does not necessarily apply to descriptor\n attributes, such as properties created with "property()".\n\n* If the target is a subscription: The primary expression in the\n reference is evaluated. It should yield either a mutable sequence\n object (such as a list) or a mapping object (such as a dictionary).\n Next, the subscript expression is evaluated.\n\n If the primary is a mutable sequence object (such as a list), the\n subscript must yield an integer. If it is negative, the sequence\'s\n length is added to it. The resulting value must be a nonnegative\n integer less than the sequence\'s length, and the sequence is asked\n to assign the assigned object to its item with that index. If the\n index is out of range, "IndexError" is raised (assignment to a\n subscripted sequence cannot add new items to a list).\n\n If the primary is a mapping object (such as a dictionary), the\n subscript must have a type compatible with the mapping\'s key type,\n and the mapping is then asked to create a key/datum pair which maps\n the subscript to the assigned object. This can either replace an\n existing key/value pair with the same key value, or insert a new\n key/value pair (if no key with the same value existed).\n\n For user-defined objects, the "__setitem__()" method is called with\n appropriate arguments.\n\n* If the target is a slicing: The primary expression in the\n reference is evaluated. It should yield a mutable sequence object\n (such as a list). The assigned object should be a sequence object\n of the same type. Next, the lower and upper bound expressions are\n evaluated, insofar they are present; defaults are zero and the\n sequence\'s length. The bounds should evaluate to integers. If\n either bound is negative, the sequence\'s length is added to it. The\n resulting bounds are clipped to lie between zero and the sequence\'s\n length, inclusive. Finally, the sequence object is asked to replace\n the slice with the items of the assigned sequence. The length of\n the slice may be different from the length of the assigned sequence,\n thus changing the length of the target sequence, if the object\n allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample "a, b = b, a" swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe! For instance, the\nfollowing program prints "[0, 2]":\n\n x = [0, 1]\n i = 0\n i, x[i] = 1, 2\n print(x)\n\nSee also: **PEP 3132** - Extended Iterable Unpacking\n\n The specification for the "*target" feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n augtarget ::= identifier | attributeref | subscription | slicing\n augop ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like "x += 1" can be rewritten as\n"x = x + 1" to achieve a similar, but not exactly equal effect. In the\naugmented version, "x" is only evaluated once. Also, when possible,\nthe actual operation is performed *in-place*, meaning that rather than\ncreating a new object and assigning that to the target, the old object\nis modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n', 'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name. See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a "NameError" exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them. The transformation inserts the\nclass name, with leading underscores removed and a single underscore\ninserted, in front of the name. For example, the identifier "__spam"\noccurring in a class named "Ham" will be transformed to "_Ham__spam".\nThis transformation is independent of the syntactical context in which\nthe identifier is used. If the transformed name is extremely long\n(longer than 255 characters), implementation defined truncation may\nhappen. If the class name consists only of underscores, no\ntransformation is done.\n', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:24 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Version_bump_f?= =?utf-8?b?b3IgMy40LjByYzMu?= Message-ID: <3fnSPw0sGVz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/8a81cdab3e9d changeset: 89789:8a81cdab3e9d branch: 3.4 tag: v3.4.0rc3 user: Larry Hastings date: Sun Mar 09 04:13:05 2014 -0700 summary: Version bump for 3.4.0rc3. files: Include/patchlevel.h | 4 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/RPM/python-3.4.spec | 2 +- README | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 4 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 2 +#define PY_RELEASE_SERIAL 3 /* Version as a string */ -#define PY_VERSION "3.4.0rc2" +#define PY_VERSION "3.4.0rc3" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.4.0rc2" +__version__ = "3.4.0rc3" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.4.0rc2" +IDLE_VERSION = "3.4.0rc3" diff --git a/Misc/RPM/python-3.4.spec b/Misc/RPM/python-3.4.spec --- a/Misc/RPM/python-3.4.spec +++ b/Misc/RPM/python-3.4.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.4.0rc2 +%define version 3.4.0rc3 %define libvers 3.4 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,4 +1,4 @@ -This is Python version 3.4.0 release candidate 2 +This is Python version 3.4.0 release candidate 3 ================================================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:25 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Added_tag_v3?= =?utf-8?q?=2E4=2E0rc3_for_changeset_8a81cdab3e9d?= Message-ID: <3fnSPx2GPhz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/370f5258e3bc changeset: 89790:370f5258e3bc branch: 3.4 user: Larry Hastings date: Sun Mar 09 04:13:25 2014 -0700 summary: Added tag v3.4.0rc3 for changeset 8a81cdab3e9d files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -129,3 +129,4 @@ a97ce3ecc96af79bd2e1ac66ce48d9138e0ca749 v3.4.0b3 5e088cea8660677969113741c1313d570d977e02 v3.4.0rc1 a300712ed38c9a242b736c44e806caea25a6dc05 v3.4.0rc2 +8a81cdab3e9d521daaef989fade94b16455fc3b8 v3.4.0rc3 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:26 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Mark_branch_as?= =?utf-8?q?_being_after_Python_3=2E4=2E0rc3=2E?= Message-ID: <3fnSPy4Gsxz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/c3f13d542e40 changeset: 89791:c3f13d542e40 branch: 3.4 user: Larry Hastings date: Sat Mar 15 20:57:42 2014 -0700 summary: Mark branch as being after Python 3.4.0rc3. files: Include/patchlevel.h | 2 +- Misc/NEWS | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 3 /* Version as a string */ -#define PY_VERSION "3.4.0rc3" +#define PY_VERSION "3.4.0rc3+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,8 +2,21 @@ Python News +++++++++++ +What's New in Python 3.4.0? +=========================== + +Release date: 2014-03-16 + +Core and Builtins +----------------- + +Library +------- + + What's New in Python 3.4.0 release candidate 3? =============================================== + Release date: 2014-03-09 Core and Builtins -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:27 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogd2hhdHNuZXc6IGNw?= =?utf-8?q?273_codec_=28=2310907797=29?= Message-ID: <3fnSPz63xbz7LkX@mail.python.org> http://hg.python.org/cpython/rev/f7d093a4ebf6 changeset: 89792:f7d093a4ebf6 branch: 3.4 user: R David Murray date: Fri Mar 07 21:00:34 2014 -0500 summary: whatsnew: cp273 codec (#10907797) Also updated the docs and added the aliases mentioned by the references. files: Doc/library/codecs.rst | 4 ++++ Doc/whatsnew/3.4.rst | 3 +++ Lib/encodings/aliases.py | 5 +++++ 3 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -971,6 +971,10 @@ +-----------------+--------------------------------+--------------------------------+ | cp037 | IBM037, IBM039 | English | +-----------------+--------------------------------+--------------------------------+ +| cp273 | 273, IBM273, csIBM273 | German | +| | | | +| | | .. versionadded:: 3.4 | ++-----------------+--------------------------------+--------------------------------+ | cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | +-----------------+--------------------------------+--------------------------------+ | cp437 | 437, IBM437 | English | diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -364,6 +364,9 @@ Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. +* New EBCDIC :ref:`codec ` ``cp273``. (Contributed by + Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) + New Modules diff --git a/Lib/encodings/aliases.py b/Lib/encodings/aliases.py --- a/Lib/encodings/aliases.py +++ b/Lib/encodings/aliases.py @@ -109,6 +109,11 @@ '1258' : 'cp1258', 'windows_1258' : 'cp1258', + # cp273 codec + '273' : 'cp273', + 'ibm273' : 'cp273', + 'csibm273' : 'cp273', + # cp424 codec '424' : 'cp424', 'csibm424' : 'cp424', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:29 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogbWVyZ2UgMy4zICgj?= =?utf-8?q?20896=29?= Message-ID: <3fnSQ10lsVz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/9b91c23f071c changeset: 89793:9b91c23f071c branch: 3.4 user: Benjamin Peterson date: Wed Mar 12 18:10:57 2014 -0500 summary: merge 3.3 (#20896) files: Lib/test/test_ssl.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1351,12 +1351,15 @@ def test_get_server_certificate(self): def _test_get_server_certificate(host, port, cert=None): with support.transient_internet(host): - pem = ssl.get_server_certificate((host, port)) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) try: - pem = ssl.get_server_certificate((host, port), ca_certs=CERTFILE) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=CERTFILE) except ssl.SSLError as x: #should fail if support.verbose: @@ -1364,7 +1367,9 @@ else: self.fail("Got server certificate %s for %s:%s!" % (pem, host, port)) - pem = ssl.get_server_certificate((host, port), ca_certs=cert) + pem = ssl.get_server_certificate((host, port), + ssl.PROTOCOL_SSLv23, + ca_certs=cert) if not pem: self.fail("No server certificate on %s:%s!" % (host, port)) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:30 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogbWVyZ2UgMy4zICgj?= =?utf-8?q?19060=29?= Message-ID: <3fnSQ22MqKz7Lkd@mail.python.org> http://hg.python.org/cpython/rev/d2ad2e04dc9d changeset: 89794:d2ad2e04dc9d branch: 3.4 user: Benjamin Peterson date: Wed Mar 12 21:42:04 2014 -0500 summary: merge 3.3 (#19060) files: Doc/library/subprocess.rst | 2 +- Lib/subprocess.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -9,7 +9,7 @@ The :mod:`subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to -replace several other, older modules and functions, such as:: +replace several older modules and functions:: os.system os.spawn* diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -11,7 +11,7 @@ This module allows you to spawn processes, connect to their input/output/error pipes, and obtain their return codes. This module -intends to replace several other, older modules and functions, like: +intends to replace several older modules and functions: os.system os.spawn* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:31 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogd2hhdHNuZXc6IGRp?= =?utf-8?q?fflib=2Eisbjunk_=26c_were_removed=2C_not_deprecated=2E?= Message-ID: <3fnSQ34DVrz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/45988af92990 changeset: 89795:45988af92990 branch: 3.4 user: R David Murray date: Thu Mar 13 21:34:54 2014 -0400 summary: whatsnew: difflib.isbjunk &c were removed, not deprecated. Also move NEWS item to correct position (it was in 3.3). files: Doc/whatsnew/3.4.rst | 10 ++++++---- Misc/NEWS | 5 +++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1503,10 +1503,6 @@ Deprecated Python Modules, Functions and Methods ------------------------------------------------ -* :meth:`difflib.SequenceMatcher.isbjunk` and - :meth:`difflib.SequenceMatcher.isbpopular` were removed: use ``x in sm.bjunk`` and - ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object. - * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` methods and functions are deprecated: :meth:`importlib.find_loader` is replaced by :func:`importlib.util.find_spec`; @@ -1612,6 +1608,12 @@ * :class:`inspect.Signature`: positional-only parameters are now required to have a valid name. +* :meth:`difflib.SequenceMatcher.isbjunk` and + :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have + now been removed: use ``x in sm.bjunk`` and + ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object + (:issue:`13248`). + Code Cleanups ------------- diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2987,6 +2987,9 @@ - Issue #17467: add readline and readlines support to mock_open in unittest.mock. +- Issue #13248: removed deprecated and undocumented difflib.isbjunk, + isbpopular. + - Issue #17192: Update the ctypes module's libffi to v3.0.13. This specifically addresses a stack misalignment issue on x86 and issues on some more recent platforms. @@ -4412,8 +4415,6 @@ Library ------- -- Issue #13248: removed deprecated and undocumented difflib.isbjunk, isbpopular. - - Issue #13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:32 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_merge_from_3=2E3?= Message-ID: <3fnSQ467kQz7Lkc@mail.python.org> http://hg.python.org/cpython/rev/7f4fba171a40 changeset: 89796:7f4fba171a40 branch: 3.4 user: Ned Deily date: Sat Mar 15 13:19:20 2014 -0700 summary: Issue #20939: merge from 3.3 files: Lib/test/test_urllibnet.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -80,7 +80,7 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "http://www.python.org/" + URL = "https://www.python.org/" with self.urlopen(URL) as open_url: gotten_url = open_url.geturl() self.assertEqual(gotten_url, URL) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ Library ------- +- Issue #20939: Fix test_geturl failure in test_urllibnet due to + new redirect of http://www.python.org/ to https://www.python.org. + What's New in Python 3.4.0 release candidate 3? =============================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:36 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Merge_in_all_d?= =?utf-8?q?ocumentation_changes_since_branching_3=2E4=2E0rc1=2E?= Message-ID: <3fnSQ81v9yz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/cc8eee3520bf changeset: 89797:cc8eee3520bf branch: 3.4 user: Larry Hastings date: Sat Mar 15 21:13:56 2014 -0700 summary: Merge in all documentation changes since branching 3.4.0rc1. files: Doc/c-api/arg.rst | 1 + Doc/c-api/exceptions.rst | 8 +- Doc/c-api/init.rst | 1 + Doc/c-api/module.rst | 2 +- Doc/c-api/object.rst | 6 +- Doc/c-api/tuple.rst | 8 + Doc/c-api/typeobj.rst | 5 +- Doc/conf.py | 6 +- Doc/contents.rst | 12 +- Doc/distributing/index.rst | 143 + Doc/distutils/apiref.rst | 28 +- Doc/distutils/configfile.rst | 2 + Doc/distutils/index.rst | 6 +- Doc/distutils/setupscript.rst | 9 +- Doc/extending/index.rst | 42 +- Doc/faq/general.rst | 3 +- Doc/faq/library.rst | 1 + Doc/faq/programming.rst | 1 + Doc/glossary.rst | 8 + Doc/howto/pyporting.rst | 33 +- Doc/includes/email-alternative-new-api.py | 2 +- Doc/install/index.rst | 9 +- Doc/installing/index.rst | 210 + Doc/library/aifc.rst | 4 +- Doc/library/asyncio-eventloop.rst | 6 +- Doc/library/asyncio-protocol.rst | 15 +- Doc/library/asyncio-subprocess.rst | 8 +- Doc/library/asyncio-sync.rst | 29 +- Doc/library/audioop.rst | 8 +- Doc/library/base64.rst | 32 +- Doc/library/csv.rst | 57 +- Doc/library/curses.rst | 2 +- Doc/library/distutils.rst | 19 +- Doc/library/email-examples.rst | 6 +- Doc/library/email.contentmanager.rst | 19 +- Doc/library/email.message.rst | 4 +- Doc/library/email.policy.rst | 2 +- Doc/library/ensurepip.rst | 2 +- Doc/library/enum.rst | 2 +- Doc/library/exceptions.rst | 1 - Doc/library/fractions.rst | 8 +- Doc/library/functions.rst | 35 +- Doc/library/getopt.rst | 1 + Doc/library/hashlib.rst | 12 +- Doc/library/hmac.rst | 5 +- Doc/library/http.server.rst | 14 +- Doc/library/importlib.rst | 2 +- Doc/library/inspect.rst | 5 + Doc/library/io.rst | 1 + Doc/library/ipaddress.rst | 20 +- Doc/library/logging.config.rst | 8 +- Doc/library/math.rst | 6 +- Doc/library/multiprocessing.rst | 37 +- Doc/library/operator.rst | 4 +- Doc/library/os.path.rst | 18 +- Doc/library/os.rst | 12 +- Doc/library/pathlib.rst | 30 + Doc/library/pkgutil.rst | 2 + Doc/library/plistlib.rst | 13 +- Doc/library/pydoc.rst | 5 + Doc/library/site.rst | 10 +- Doc/library/smtplib.rst | 3 + Doc/library/socket.rst | 13 +- Doc/library/ssl.rst | 5 +- Doc/library/stringprep.rst | 1 - Doc/library/subprocess.rst | 23 +- Doc/library/sunau.rst | 4 +- Doc/library/sys.rst | 13 +- Doc/library/test.rst | 1 + Doc/library/textwrap.rst | 29 +- Doc/library/tracemalloc.rst | 39 +- Doc/library/types.rst | 21 + Doc/library/undoc.rst | 2 +- Doc/library/unittest.mock-examples.rst | 2 +- Doc/library/unittest.mock.rst | 4 +- Doc/library/unittest.rst | 31 +- Doc/library/urllib.error.rst | 2 +- Doc/library/urllib.request.rst | 17 +- Doc/library/venv.rst | 2 + Doc/library/wave.rst | 35 +- Doc/library/xml.etree.elementtree.rst | 3 +- Doc/library/xml.rst | 73 +- Doc/library/zipfile.rst | 26 +- Doc/reference/datamodel.rst | 18 +- Doc/tools/sphinxext/indexcontent.html | 8 +- Doc/tools/sphinxext/susp-ignored.csv | 1 + Doc/tutorial/introduction.rst | 8 +- Doc/tutorial/whatnow.rst | 4 +- Doc/using/unix.rst | 2 + Doc/using/venv-create.inc | 10 +- Doc/using/windows.rst | 4 + Doc/whatsnew/3.4.rst | 1196 ++++++++- Misc/NEWS | 8 +- 93 files changed, 2040 insertions(+), 578 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -45,6 +45,7 @@ Unless otherwise stated, buffers are not NUL-terminated. .. note:: + For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of the length argument (int or :c:type:`Py_ssize_t`) is controlled by defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -525,11 +525,11 @@ reference, as accessible from Python through :attr:`__cause__`. -.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *ctx) +.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause) - Set the cause associated with the exception to *ctx*. Use *NULL* to clear - it. There is no type check to make sure that *ctx* is either an exception - instance or :const:`None`. This steals a reference to *ctx*. + Set the cause associated with the exception to *cause*. Use *NULL* to clear + it. There is no type check to make sure that *cause* is either an exception + instance or :const:`None`. This steals a reference to *cause*. :attr:`__suppress_context__` is implicitly set to ``True`` by this function. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -582,6 +582,7 @@ .. index:: module: _thread .. note:: + When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -120,7 +120,7 @@ Return a pointer to the :c:type:`PyModuleDef` struct from which the module was created, or *NULL* if the module wasn't created with - :c:func:`PyModule_Create`.i + :c:func:`PyModule_Create`. .. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -357,9 +357,9 @@ .. c:function:: Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t default) - Return an estimated length for the object *o*. First trying to return its - actual length, then an estimate using ``__length_hint__``, and finally - returning the default value. On error ``-1`` is returned. This is the + Return an estimated length for the object *o*. First try to return its + actual length, then an estimate using :meth:`~object.__length_hint__`, and + finally return the default value. On error return ``-1``. This is the equivalent to the Python expression ``operator.length_hint(o, default)``. .. versionadded:: 3.4 diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -129,6 +129,14 @@ Initializes a struct sequence type *type* from *desc* in place. +.. c:function:: int PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc *desc) + + The same as ``PyStructSequence_InitType``, but returns ``0`` on success and ``-1`` on + failure. + + .. versionadded:: 3.4 + + .. c:type:: PyStructSequence_Desc Contains the meta information of a struct sequence type to create. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -205,9 +205,8 @@ bit currently defined is :const:`Py_PRINT_RAW`. When the :const:`Py_PRINT_RAW` flag bit is set, the instance should be printed the same way as :c:member:`~PyTypeObject.tp_str` would format it; when the :const:`Py_PRINT_RAW` flag bit is clear, the instance - should be printed the same was as :c:member:`~PyTypeObject.tp_repr` would format it. It should - return ``-1`` and set an exception condition when an error occurred during the - comparison. + should be printed the same way as :c:member:`~PyTypeObject.tp_repr` would format it. It should + return ``-1`` and set an exception condition when an error occurs. It is possible that the :c:member:`~PyTypeObject.tp_print` field will be deprecated. In any case, it is recommended not to define :c:member:`~PyTypeObject.tp_print`, but instead to rely on diff --git a/Doc/conf.py b/Doc/conf.py --- a/Doc/conf.py +++ b/Doc/conf.py @@ -61,6 +61,8 @@ # By default, highlight as Python 3. highlight_language = 'python3' +needs_sphinx = '1.1' + # Options for HTML output # ----------------------- @@ -118,11 +120,11 @@ latex_documents = [ ('c-api/index', 'c-api.tex', 'The Python/C API', _stdauthor, 'manual'), - ('distutils/index', 'distutils.tex', + ('distributing/index', 'distributing.tex', 'Distributing Python Modules', _stdauthor, 'manual'), ('extending/index', 'extending.tex', 'Extending and Embedding Python', _stdauthor, 'manual'), - ('install/index', 'install.tex', + ('installing/index', 'installing.tex', 'Installing Python Modules', _stdauthor, 'manual'), ('library/index', 'library.tex', 'The Python Library Reference', _stdauthor, 'manual'), diff --git a/Doc/contents.rst b/Doc/contents.rst --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -11,8 +11,8 @@ library/index.rst extending/index.rst c-api/index.rst - distutils/index.rst - install/index.rst + distributing/index.rst + installing/index.rst howto/index.rst faq/index.rst glossary.rst @@ -21,3 +21,11 @@ bugs.rst copyright.rst license.rst + +.. include legacy packaging docs in build + +.. toctree:: + :hidden: + + distutils/index.rst + install/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst new file mode 100644 --- /dev/null +++ b/Doc/distributing/index.rst @@ -0,0 +1,143 @@ +.. _distributing-index: + +############################### + Distributing Python Modules +############################### + +:Email: distutils-sig at python.org + + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the distribution part of the process. For a guide to +installing other Python projects, refer to the +:ref:`installation guide `. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* the `Python Package Index `__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + `__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub `__ and + `BitBucket `__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Open source licensing and collaboration +======================================= + +In most parts of the world, software is automatically covered by copyright. +This means that other developers require explicit permission to copy, use, +modify and redistribute the software. + +Open source licensing is a way of explicitly granting such permission in a +relatively consistent way, allowing developers to share and collaborate +efficiently by making common solutions to various problems freely available. +This leaves many developers free to spend more time focusing on the problems +that are relatively unique to their specific situation. + +The distribution tools provided with Python are designed to make it +reasonably straightforward for developers to make their own contributions +back to that common pool of software if they choose to do so. + +The same distribution tools can also be used to distribute software within +an organisation, regardless of whether that software is published as open +source software or not. + + +Installing the tools +==================== + +The standard library does not include build tools that support modern +Python packaging standards, as the core development team has found that it +is important to have standard tools that work consistently, even on older +versions of Python. + +The currently recommended build and distribution tools can be installed +using ``pip``:: + + pip install setuptools wheel twine + + +Reading the guide +================= + +The Python Packaging User Guide covers the various key steps and elements +involved in creating a project + +* `Project structure`_ +* `Building and packaging the project`_ +* `Uploading the project to the Python Package Index`_ + +.. _Project structure: \ + http://packaging.python.org/en/latest/tutorial.html#creating-your-own-project +.. _Building and packaging the project: \ + http://packaging.python.org/en/latest/tutorial.html#building-packaging-your-project +.. _Uploading the project to the Python Package Index: \ + http://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi + + +How do I...? +============ + +These are quick answers or links for some common tasks. + +... choose a name for my project? +--------------------------------- + +This isn't an easy topic, but here are a few tips: + +* check the Python Package Index to see if the name is already in use +* check popular hosting sites like GitHub, BitBucket, etc to see if there + is already a project with that name +* check what comes up in a web search for the name you're considering +* avoid particularly common words, especially ones with multiple meanings, + as they can make it difficult for users to find your software when + searching for it + + +... create and distribute binary extensions? +-------------------------------------------- + +This is actually quite a complex topic, with a variety of alternatives +available depending on exactly what you're aiming to achieve. See the +Python Packaging User Guide for more information and recommendations. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + `__ + +.. other topics: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -853,17 +853,6 @@ port of GCC (same as cygwin in no-cygwin mode). -:mod:`distutils.emxccompiler` --- OS/2 EMX Compiler -=================================================== - -.. module:: distutils.emxccompiler - :synopsis: OS/2 EMX Compiler support - - -This module provides the EMXCCompiler class, a subclass of -:class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2. - - :mod:`distutils.archive_util` --- Archiving utilities ====================================================== @@ -1171,15 +1160,6 @@ underscore. No { } or ( ) style quoting is available. -.. function:: grok_environment_error(exc[, prefix='error: ']) - - Generate a useful error message from an :exc:`OSError` exception object. - Handles Python 1.5.1 and later styles, and does what it can to deal with - exception objects that don't have a filename (which happens when the error - is due to a two-file operation, such as :func:`~os.rename` or :func:`~os.link`). - Returns the error message as a string prefixed with *prefix*. - - .. function:: split_quoted(s) Split a string up according to Unix shell-like rules for quotes and backslashes. @@ -1943,8 +1923,12 @@ .. module:: distutils.command.clean :synopsis: Clean a package build area - -.. % todo +This command removes the temporary files created by :command:`build` +and its subcommands, like intermediary compiled object files. With +the ``--all`` option, the complete build directory will be removed. + +Extension modules built :ref:`in place ` +will not be cleaned, as they are not in the build directory. :mod:`distutils.command.config` --- Perform package configuration diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -69,6 +69,8 @@ Note that an option spelled :option:`--foo-bar` on the command-line is spelled :option:`foo_bar` in configuration files. +.. _distutils-build-ext-inplace: + For example, say you want your extensions to be built "in-place"---that is, you have an extension :mod:`pkg.ext`, and you want the compiled extension file (:file:`ext.so` on Unix, say) to be put in the same source directory as your diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -1,8 +1,8 @@ .. _distutils-index: -############################### - Distributing Python Modules -############################### +############################################## + Distributing Python Modules (Legacy version) +############################################## :Authors: Greg Ward, Anthony Baxter :Email: distutils-sig at python.org diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -685,6 +685,8 @@ DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -700,7 +702,8 @@ and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,14 +21,31 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. -.. note:: - This guide only covers the basic tools for creating extensions provided - as part of this version of CPython. Third party tools may offer simpler - alternatives. Refer to the `binary extensions section - `__ - in the Python Packaging User Guide for more information. +Recommended third party tools +============================= +This guide only covers the basic tools for creating extensions provided +as part of this version of CPython. Third party tools like Cython, +``cffi``, SWIG and Numba offer both simpler and more sophisticated +approaches to creating C and C++ extensions for Python. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions `_ + The Python Packaging User Guide not only covers several available + tools that simplify the creation of binary extensions, but also + discusses the various reasons why creating an extension module may be + desirable in the first place. + + +Creating extensions without third party tools +============================================= + +This section of the guide covers creating C and C++ extensions without +assistance from third party tools. It is intended primarily for creators +of those tools, rather than being a recommended way to create your own +C extensions. .. toctree:: :maxdepth: 2 @@ -38,4 +55,17 @@ newtypes.rst building.rst windows.rst + +Embedding the CPython runtime in a larger application +===================================================== + +Sometimes, rather than creating an extension that runs inside the Python +interpreter as the main application, it is desirable to instead embed +the CPython runtime inside a larger application. This section covers +some of the details involved in doing that successfully. + +.. toctree:: + :maxdepth: 2 + :numbered: + embedding.rst diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -471,7 +471,8 @@ Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult -http://www.python.org/editors/ for a full list of Python editing environments. +`the Python wiki `_ for a full list +of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -513,6 +513,7 @@ you can also use the :mod:`array` module. .. note:: + To read and write binary data, it is mandatory to open the file in binary mode (here, passing ``"rb"`` to :func:`open`). If you use ``"r"`` instead (the default), the file will be open in text mode diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1103,6 +1103,7 @@ result = [obj.method() for obj in mylist] +.. _faq-augmented-assignment-tuple-error: Why does a_tuple[i] += ['item'] raise an exception when the addition works? --------------------------------------------------------------------------- diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -783,6 +783,14 @@ mapping rather than a sequence because the lookups use arbitrary :term:`immutable` keys rather than integers. + The :class:`collections.abc.Sequence` abstract base class + defines a much richer interface that goes beyond just + :meth:`__getitem__` and :meth:`__len__`, adding :meth:`count`, + :meth:`index`, :meth:`__contains__`, and + :meth:`__reversed__`. Types that implement this expanded + interface can be registered explicitly using + :func:`~abc.register`. + single dispatch A form of :term:`generic function` dispatch where the implementation is chosen based on the type of a single argument. diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -26,6 +26,32 @@ For help with porting, you can email the python-porting_ mailing list with questions. +The Short Version +================= + +* Decide what's the oldest version of Python 2 you want to support (if at all) +* Make sure you have a thorough test suite and use continuous integration + testing to make sure you stay compatible with the versions of Python you care + about +* If you have dependencies, check their Python 3 status using caniusepython3 + (`command-line tool `__, + `web app `__) + +With that done, your options are: + +* If you are dropping Python 2 support, use 2to3_ to port to Python 3 +* If you are keeping Python 2 support, then start writing Python 2/3-compatible + code starting **TODAY** + + + If you have dependencies that have not been ported, reach out to them to port + their project while working to make your code compatible with Python 3 so + you're ready when your dependencies are all ported + + If all your dependencies have been ported (or you have none), go ahead and + port to Python 3 + +* If you are creating a new project that wants to have 2/3 compatibility, + code in Python 3 and then backport to Python 2 + Before You Begin ================ @@ -85,7 +111,7 @@ Projects to Consider -------------------- -The lowest level library for suppoting Python 2 & 3 simultaneously is six_. +The lowest level library for supporting Python 2 & 3 simultaneously is six_. Reading through its documentation will give you an idea of where exactly the Python language changed between versions 2 & 3 and thus what you will want the library to help you continue to support. @@ -548,7 +574,10 @@ If you have Python 3 code and have little interest in supporting Python 2 you can use 3to2_ to translate from Python 3 code to Python 2 code. This is only -recommended if you don't plan to heavily support Python 2 users. +recommended if you don't plan to heavily support Python 2 users. Otherwise +write your code for Python 3 and then backport as far back as you want. This +is typically easier than going from Python 2 to 3 as you will have worked out +any difficulties with e.g. bytes/strings, etc. Other Resources diff --git a/Doc/includes/email-alternative-new-api.py b/Doc/includes/email-alternative-new-api.py --- a/Doc/includes/email-alternative-new-api.py +++ b/Doc/includes/email-alternative-new-api.py @@ -19,7 +19,7 @@ [1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718 ---?ric +--Pep? """) # Add the html version. This converts the message into a multipart/alternative diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -2,9 +2,9 @@ .. _install-index: -***************************** - Installing Python Modules -***************************** +******************************************** + Installing Python Modules (Legacy version) +******************************************** :Author: Greg Ward @@ -58,7 +58,8 @@ document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. .. _inst-trivial-install: diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst new file mode 100644 --- /dev/null +++ b/Doc/installing/index.rst @@ -0,0 +1,210 @@ +.. highlightlang:: none + +.. _installing-index: + +***************************** + Installing Python Modules +***************************** + +:Email: distutils-sig at python.org + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the installation part of the process. For a guide to +creating and sharing your own Python projects, refer to the +:ref:`distribution guide `. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* ``pip`` is the preferred installer program. Starting with Python 3.4, it + is included by default with the Python binary installers. +* a virtual environment is a semi-isolated Python environment that allows + packages to be installed for use by a particular application, rather than + being installed system wide +* ``pyvenv`` is the standard tool for creating virtual environments, and has + been part of Python since Python 3.3. Starting with Python 3.4, it + defaults to installing ``pip`` into all created virtual environments +* the `Python Package Index `__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + `__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub `__ and + `BitBucket `__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Basic usage +=========== + +The standard packaging tools are all designed to be used from the command +line. For Windows users, the examples below assume that the option to +adjust the system PATH environment variable was selected when installing +Python. For Linux users, the command to install into the system version of +Python 3 is likely to be ``pip3`` rather than ``pip``. + +The following command will install the latest version of a module and its +dependencies from the Python Package Index:: + + pip install SomePackage + +It's also possible to specify an exact or minimum version directly on the +command line:: + + pip install SomePackage==1.0.4 # specific version + pip install 'SomePackage>=1.0.4' # minimum version + +Normally, if a suitable module is already installed, attempting to install +it again will have no effect. Upgrading existing modules must be requested +explicitly:: + + pip install --upgrade SomePackage + +More information and resources regarding ``pip`` and its capabilities can be +found in the `Python Packaging User Guide `__. + +``pyvenv`` has its own documentation at :ref:`scripts-pyvenv`. Installing +into an active virtual environment uses the commands shown above. + +.. seealso:: + + `Python Packaging User Guide: Installing Python packages + `__ + + +How do I ...? +============= + +These are quick answers or links for some common tasks. + +... install ``pip`` in versions of Python prior to Python 3.4? +-------------------------------------------------------------- + +Python only started bundling ``pip`` with Python 3.4. For earlier versions, +``pip`` needs to be "bootstrapped" as described in the Python Packaging +User Guide. + +.. seealso:: + + `Python Packaging User Guide: Installing the Tools + `__ + + +.. installing-per-user-installation: + +... install packages just for the current user? +----------------------------------------------- + +Passing the ``--user`` option to ``pip install`` will install a package +just for the current user, rather than for all users of the system. + + +... install scientific Python packages? +--------------------------------------- + +A number of scientific Python packages have complex binary dependencies, and +aren't currently easy to install using ``pip`` directly. At this point in +time, it will often be easier for users to install these packages by +`other means +`__ +rather than attempting to install them with ``pip``. + +.. seealso:: + + `Python Packaging User Guide: Installing Scientific Packages + `__ + + +... work with multiple versions of Python installed in parallel? +---------------------------------------------------------------- + +On Linux, Mac OS X and other POSIX systems, use the versioned Python commands +in combination with the ``-m`` switch to run the appropriate copy of +``pip``:: + + python2 -m pip install SomePackage # default Python 2 + python2.7 -m pip install SomePackage # specifically Python 2.7 + python3 -m pip install SomePackage # default Python 3 + python3.4 -m pip install SomePackage # specifically Python 3.4 + +(appropriately versioned ``pip`` commands may also be available) + +On Windows, use the ``py`` Python launcher in combination with the ``-m`` +switch:: + + py -2 -m pip install SomePackage # default Python 2 + py -2.7 -m pip install SomePackage # specifically Python 2.7 + py -3 -m pip install SomePackage # default Python 3 + py -3.4 -m pip install SomePackage # specifically Python 3.4 + +.. other questions: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) + + +Common installation issues +========================== + +Installing into the system Python on Linux +------------------------------------------ + +On Linux systems, a Python installation will typically be included as part +of the distribution. Installing into this Python installation requires +root access to the system, and may interfere with the operation of the +system package manager and other components of the system if a component +is unexpectedly upgraded using ``pip``. + +On such systems, it is often better to use a virtual environment or a +per-user installation when installing packages with ``pip``. + + +Installing binary extensions +---------------------------- + +Python has typically relied heavily on source based distribution, with end +users being expected to compile extension modules from source as part of +the installation process. + +With the introduction of support for the binary ``wheel`` format, and the +ability to publish wheels for at least Windows and Mac OS X through the +Python Package Index, this problem is expected to diminish over time, +as users are more regularly able to install pre-built extensions rather +than needing to build them themselves. + +Some of the solutions for installing `scientific software +`__ +that is not yet available as pre-built ``wheel`` files may also help with +obtaining other binary extensions without needing to build them locally. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + `__ diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -226,7 +226,7 @@ file parameters have been set. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.writeframesraw(data) @@ -235,7 +235,7 @@ updated. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.close() diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -488,7 +488,7 @@ * *stdout*: Either a file-like object representing the pipe to be connected to the subprocess's standard output stream using - :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :meth:`~BaseEventLoop.connect_read_pipe`, or the constant :const:`subprocess.PIPE` (the default). By default a new pipe will be created and connected. @@ -589,8 +589,8 @@ Arrange for a callback to be called in the specified executor. - *executor* is a :class:`~concurrent.futures.Executor` instance, - the default executor is used if *executor* is ``None``. + The *executor* argument should be an :class:`~concurrent.futures.Executor` + instance. The default executor is used if *executor* is ``None``. This method is a :ref:`coroutine `. diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -376,8 +376,8 @@ Flow control callbacks ---------------------- -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: +These callbacks may be called on :class:`Protocol`, +:class:`DatagramProtocol` and :class:`SubprocessProtocol` instances: .. method:: BaseProtocol.pause_writing() @@ -402,6 +402,15 @@ are important to ensure that things go as expected when either mark is zero. +.. note:: + On BSD systems (OS X, FreeBSD, etc.) flow control is not supported + for :class:`DatagramProtocol`, because send failures caused by + writing too many packets cannot be detected easily. The socket + always appears 'ready' and excess packets are dropped; an + :class:`OSError` with errno set to :const:`errno.ENOBUFS` may or + may not be raised; if it is raised, it will be reported to + :meth:`DatagramProtocol.error_received` but otherwise ignored. + Coroutines and protocols ------------------------ @@ -488,6 +497,6 @@ :meth:`Transport.close` can be called immediately after :meth:`WriteTransport.write` even if data are not sent yet on the socket: both methods are asynchronous. ``yield from`` is not needed because these transport -methods don't return coroutines. +methods are not coroutines. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -8,16 +8,16 @@ .. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - Run the shell command *cmd* given as a string. Return a :class:`Process` + Run the shell command *cmd* given as a string. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. .. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - Create a subprocess. Return a :class:`Process` instance. + Create a subprocess. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object `. + This function is a :ref:`coroutine `. Use the :meth:`BaseEventLoop.connect_read_pipe` and :meth:`BaseEventLoop.connect_write_pipe` methods to connect pipes. diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -64,7 +64,7 @@ .. method:: locked() - Return ``True`` if lock is acquired. + Return ``True`` if the lock is acquired. .. method:: acquire() @@ -73,7 +73,7 @@ This method blocks until the lock is unlocked, then sets it to locked and returns ``True``. - This method returns a :ref:`coroutine object `. + This method is a :ref:`coroutine `. .. method:: release() @@ -141,6 +141,15 @@ A new :class:`Lock` object is created and used as the underlying lock. + .. method:: acquire() + + Acquire the underlying lock. + + This method blocks until the lock is unlocked, then sets it to locked and + returns ``True``. + + This method is a :ref:`coroutine `. + .. method:: notify(n=1) By default, wake up one coroutine waiting on this condition, if any. @@ -156,6 +165,10 @@ call until it can reacquire the lock. Since :meth:`notify` does not release the lock, its caller should. + .. method:: locked() + + Return ``True`` if the underlying lock is acquired. + .. method:: notify_all() Wake up all threads waiting on this condition. This method acts like @@ -163,6 +176,18 @@ calling thread has not acquired the lock when this method is called, a :exc:`RuntimeError` is raised. + .. method:: release() + + Release the underlying lock. + + When the lock is locked, reset it to unlocked, and return. If any other + coroutines are blocked waiting for the lock to become unlocked, allow + exactly one of them to proceed. + + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. + + There is no return value. + .. method:: wait() Wait until notified. diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -12,10 +12,8 @@ .. versionchanged:: 3.4 Support for 24-bit samples was added. - -.. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted by all functions in this - module. Strings no more supported. + All functions now accept any :term:`bytes-like object`. + String input now results in an immediate error. .. index:: single: Intel/DVI ADPCM @@ -82,7 +80,7 @@ "Byteswap" all samples in a fragment and returns the modified fragment. Converts big-endian samples to little-endian and vice versa. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: cross(fragment, width) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -1,27 +1,33 @@ -:mod:`base64` --- RFC 3548: Base16, Base32, Base64 Data Encodings -================================================================= +:mod:`base64` --- Base16, Base32, Base64, Base85 Data Encodings +=============================================================== .. module:: base64 - :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings + :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings; + Base85 and Ascii85 .. index:: pair: base64; encoding single: MIME; base64 encoding -This module provides data encoding and decoding as specified in :rfc:`3548`. -This standard defines the Base16, Base32, and Base64 algorithms for encoding -and decoding arbitrary binary strings into ASCII-only byte strings that can be +This module provides functions for encoding binary data to printable +ASCII characters and decoding such encodings back to binary data. +It provides encoding and decoding functions for the encodings specified in +in :rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, +and for the de-facto standard Ascii85 and Base85 encodings. + +The :rfc:`3548` encodings are suitable for encoding binary data so that it can safely sent by email, used as parts of URLs, or included as part of an HTTP POST request. The encoding algorithm is not the same as the :program:`uuencode` program. -There are two interfaces provided by this module. The modern interface -supports encoding and decoding ASCII byte string objects using all three -alphabets. Additionally, the decoding functions of the modern interface also -accept Unicode strings containing only ASCII characters. The legacy interface -provides for encoding and decoding to and from file-like objects as well as -byte strings, but only using the Base64 standard alphabet. +There are two :rfc:`3548` interfaces provided by this module. The modern +interface supports encoding and decoding ASCII byte string objects using all +three :rfc:`3548` defined alphabets (normal, URL-safe, and filesystem-safe). +Additionally, the decoding functions of the modern interface also accept +Unicode strings containing only ASCII characters. The legacy interface provides +for encoding and decoding to and from file-like objects as well as byte +strings, but only using the Base64 standard alphabet. .. versionchanged:: 3.3 ASCII-only Unicode strings are now accepted by the decoding functions of @@ -29,7 +35,7 @@ .. versionchanged:: 3.4 Any :term:`bytes-like object`\ s are now accepted by all - encoding and decoding functions in this module. + encoding and decoding functions in this module. Ascii85/Base85 support added. The modern interface provides: diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -142,36 +142,43 @@ The :mod:`csv` module defines the following classes: -.. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds) +.. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, \ + dialect='excel', *args, **kwds) - Create an object which operates like a regular reader but maps the information - read into a dict whose keys are given by the optional *fieldnames* parameter. - If the *fieldnames* parameter is omitted, the values in the first row of the - *csvfile* will be used as the fieldnames. If the row read has more fields - than the fieldnames sequence, the remaining data is added as a sequence - keyed by the value of *restkey*. If the row read has fewer fields than the - fieldnames sequence, the remaining keys take the value of the optional - *restval* parameter. Any other optional or keyword arguments are passed to - the underlying :class:`reader` instance. + Create an object which operates like a regular reader but maps the + information read into a dict whose keys are given by the optional + *fieldnames* parameter. The *fieldnames* parameter is a :mod:`sequence + ` whose elements are associated with the fields of the + input data in order. These elements become the keys of the resulting + dictionary. If the *fieldnames* parameter is omitted, the values in the + first row of the *csvfile* will be used as the fieldnames. If the row read + has more fields than the fieldnames sequence, the remaining data is added as + a sequence keyed by the value of *restkey*. If the row read has fewer + fields than the fieldnames sequence, the remaining keys take the value of + the optional *restval* parameter. Any other optional or keyword arguments + are passed to the underlying :class:`reader` instance. -.. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds) +.. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', \ + dialect='excel', *args, **kwds) - Create an object which operates like a regular writer but maps dictionaries onto - output rows. The *fieldnames* parameter identifies the order in which values in - the dictionary passed to the :meth:`writerow` method are written to the - *csvfile*. The optional *restval* parameter specifies the value to be written - if the dictionary is missing a key in *fieldnames*. If the dictionary passed to - the :meth:`writerow` method contains a key not found in *fieldnames*, the - optional *extrasaction* parameter indicates what action to take. If it is set - to ``'raise'`` a :exc:`ValueError` is raised. If it is set to ``'ignore'``, - extra values in the dictionary are ignored. Any other optional or keyword - arguments are passed to the underlying :class:`writer` instance. + Create an object which operates like a regular writer but maps dictionaries + onto output rows. The *fieldnames* parameter is a :mod:`sequence + ` of keys that identify the order in which values in the + dictionary passed to the :meth:`writerow` method are written to the + *csvfile*. The optional *restval* parameter specifies the value to be + written if the dictionary is missing a key in *fieldnames*. If the + dictionary passed to the :meth:`writerow` method contains a key not found in + *fieldnames*, the optional *extrasaction* parameter indicates what action to + take. If it is set to ``'raise'`` a :exc:`ValueError` is raised. If it is + set to ``'ignore'``, extra values in the dictionary are ignored. Any other + optional or keyword arguments are passed to the underlying :class:`writer` + instance. - Note that unlike the :class:`DictReader` class, the *fieldnames* parameter of - the :class:`DictWriter` is not optional. Since Python's :class:`dict` objects - are not ordered, there is not enough information available to deduce the order - in which the row should be written to the *csvfile*. + Note that unlike the :class:`DictReader` class, the *fieldnames* parameter + of the :class:`DictWriter` is not optional. Since Python's :class:`dict` + objects are not ordered, there is not enough information available to deduce + the order in which the row should be written to the *csvfile*. .. class:: Dialect diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -12,7 +12,7 @@ de-facto standard for portable advanced terminal handling. While curses is most widely used in the Unix environment, versions are available -for DOS, OS/2, and possibly other systems as well. This extension module is +for Windows, DOS, and possibly other systems as well. This extension module is designed to match the API of ncurses, an open-source curses library hosted on Linux and the BSD variants of Unix. diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -12,14 +12,15 @@ 100%-pure Python, or may be extension modules written in C, or may be collections of Python packages which include modules coded in both Python and C. +Most Python users will *not* want to use this module directly, but instead +use the cross-version tools maintained by the Python Packaging Authority. +Refer to the `Python Packaging User Guide `_ +for more information. -User documentation and API reference are provided in another document: +For the benefits of packaging tool authors and users seeking a deeper +understanding of the details of the current packaging and distribution +system, the legacy :mod:`distutils` based user documentation and API +reference remain available: -.. seealso:: - - :ref:`distutils-index` - The manual for developers and packagers of Python modules. This describes - how to prepare :mod:`distutils`\ -based packages so that they may be - easily installed into an existing Python installation. It also contains - instructions for end-users wanting to install a distutils-based package, - :ref:`install-index`. +* :ref:`install-index` +* :ref:`distutils-index` diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -40,8 +40,10 @@ .. literalinclude:: ../includes/email-alternative.py -Examples using the Provision API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _email-contentmanager-api-examples: + +Examples using the Provisional API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is a reworking of the last example using the provisional API. To make things a bit more interesting, we include a related image in the html part, and we save a copy of what we are going to send to disk, as well as sending it. diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -54,6 +54,7 @@ documented in this module because of the provisional nature of the code, the implementation lives in the :mod:`email.message` module. +.. currentmodule:: email.message .. class:: EmailMessage(policy=default) @@ -235,6 +236,16 @@ all other headers intact and in their original order. +.. class:: MIMEPart(policy=default) + + This class represents a subpart of a MIME message. It is identical to + :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are + added when :meth:`~EmailMessage.set_content` is called, since sub-parts do + not need their own :mailheader:`MIME-Version` headers. + + +.. currentmodule:: email.contentmanager + .. class:: ContentManager() Base class for content managers. Provides the standard registry mechanisms @@ -305,14 +316,6 @@ values of *typekey*, see :meth:`set_content`. -.. class:: MIMEPart(policy=default) - - This class represents a subpart of a MIME message. It is identical to - :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are - added when :meth:`~EmailMessage.set_content` is called, since sub-parts do - not need their own :mailheader:`MIME-Version` headers. - - Content Manager Instances ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -466,7 +466,7 @@ to ``False``. - .. method:: set_param(param, value, header='Content-Type', requote=True, + .. method:: set_param(param, value, header='Content-Type', requote=True, \ charset=None, language='', replace=False) Set a parameter in the :mailheader:`Content-Type` header. If the @@ -488,7 +488,7 @@ end of the list of headers. If *replace* is ``True``, the header will be updated in place. - .. versionchanged: 3.4 ``replace`` keyword was added. + .. versionchanged:: 3.4 ``replace`` keyword was added. .. method:: del_param(param, header='content-type', requote=True) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -419,7 +419,7 @@ additional arguments. By default ``content_manager`` is set to :data:`~email.contentmanager.raw_data_manager`. - .. versionadded 3.4 + .. versionadded:: 3.4 The class provides the following concrete implementations of the abstract diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -28,7 +28,7 @@ .. seealso:: - :ref:`install-index` + :ref:`installing-index` The end user guide for installing Python packages :pep:`453`: Explicit bootstrapping of pip in Python installations diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -139,7 +139,7 @@ >>> Color['green'] -If have an enum member and need its :attr:`name` or :attr:`value`:: +If you have an enum member and need its :attr:`name` or :attr:`value`:: >>> member = Color.red >>> member.name diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -265,7 +265,6 @@ :exc:`mmap.error` have been merged into :exc:`OSError`. .. versionchanged:: 3.4 - The :attr:`filename` attribute is now the original file name passed to the function, instead of the name encoded to or decoded from the filesystem encoding. Also, the :attr:`filename2` attribute was added. diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -99,7 +99,9 @@ value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`float`. @@ -108,7 +110,9 @@ This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`decimal.Decimal` instance. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -540,12 +540,13 @@ A call to ``format(value, format_spec)`` is translated to ``type(value).__format__(format_spec)`` which bypasses the instance dictionary when searching for the value's :meth:`__format__` method. A - :exc:`TypeError` exception is raised if the method is not found or if either - the *format_spec* or the return value are not strings. + :exc:`TypeError` exception is raised if the method search reaches + :mod:`object` and the *format_spec* is non-empty, or if either the + *format_spec* or the return value are not strings. - .. versionadded:: 3.4 + .. versionchanged:: 3.4 ``object().__format__(format_spec)`` raises :exc:`TypeError` - if *format_spec* is not empty string. + if *format_spec* is not an empty string. .. _func-frozenset: @@ -609,12 +610,26 @@ This function is added to the built-in namespace by the :mod:`site` module. + .. versionchanged:: 3.4 + Changes to :mod:`pydoc` and :mod:`inspect` mean that the reported + signatures for callables are now more comprehensive and consistent. + .. function:: hex(x) - Convert an integer number to a hexadecimal string. The result is a valid Python - expression. If *x* is not a Python :class:`int` object, it has to define an - :meth:`__index__` method that returns an integer. + Convert an integer number to a lowercase hexadecimal string + prefixed with "0x", for example: + + >>> hex(255) + '0xff' + >>> hex(-42) + '-0x2a' + + If x is not a Python :class:`int` object, it has to define an __index__() + method that returns an integer. + + See also :func:`int` for converting a hexadecimal string to an + integer using a base of 16. .. note:: @@ -780,6 +795,9 @@ such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and ``heapq.nlargest(1, iterable, key=keyfunc)``. + .. versionadded:: 3.4 + The *default* keyword-only argument. + .. _func-memoryview: .. function:: memoryview(obj) @@ -811,6 +829,9 @@ such as ``sorted(iterable, key=keyfunc)[0]`` and ``heapq.nsmallest(1, iterable, key=keyfunc)``. + .. versionadded:: 3.4 + The *default* keyword-only argument. + .. function:: next(iterator[, default]) Retrieve the next item from the *iterator* by calling its diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -10,6 +10,7 @@ -------------- .. note:: + The :mod:`getopt` module is a parser for command line options whose API is designed to be familiar to users of the C :c:func:`getopt` function. Users who are unfamiliar with the C :c:func:`getopt` function or who would like to write diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -101,18 +101,18 @@ .. data:: algorithms_guaranteed - Contains the names of the hash algorithms guaranteed to be supported + A set containing the names of the hash algorithms guaranteed to be supported by this module on all platforms. .. versionadded:: 3.2 .. data:: algorithms_available - Contains the names of the hash algorithms that are available - in the running Python interpreter. These names will be recognized - when passed to :func:`new`. :attr:`algorithms_guaranteed` - will always be a subset. Duplicate algorithms with different - name formats may appear in this set (thanks to OpenSSL). + A set containing the names of the hash algorithms that are available in the + running Python interpreter. These names will be recognized when passed to + :func:`new`. :attr:`algorithms_guaranteed` will always be a subset. The + same algorithm may appear multiple times in this set under different names + (thanks to OpenSSL). .. versionadded:: 3.2 diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -23,9 +23,8 @@ defaults to the :data:`hashlib.md5` constructor. .. versionchanged:: 3.4 - Parameter *key* can be a bytes or bytearray object. Parameter *msg* can - be of any type supported by :mod:`hashlib`. - + Parameter *key* can be a bytes or bytearray object. + Parameter *msg* can be of any type supported by :mod:`hashlib`. Paramter *digestmod* can be the name of a hash algorithm. .. deprecated:: 3.4 diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -351,7 +351,7 @@ The :class:`SimpleHTTPRequestHandler` class can be used in the following manner in order to create a very basic webserver serving files relative to -the current directory. :: +the current directory:: import http.server import socketserver @@ -365,15 +365,17 @@ print("serving at port", PORT) httpd.serve_forever() +.. _http-server-cli: + :mod:`http.server` can also be invoked directly using the :option:`-m` switch of the interpreter with a ``port number`` argument. Similar to -the previous example, this serves files relative to the current directory. :: +the previous example, this serves files relative to the current directory:: python -m http.server 8000 -By default, server binds itself to all interfaces. To restrict it to bind to a -particular interface only, ``--bind ADDRESS`` argument can be used. For e.g, to -restrict the server to bind only to localhost. :: +By default, server binds itself to all interfaces. The option ``-b/--bind`` +specifies a specific address to which it should bind. For example, the +following command causes the server to bind to localhost only:: python -m http.server 8000 --bind 127.0.0.1 @@ -422,7 +424,7 @@ reasons. Problems with the CGI script will be translated to error 403. :class:`CGIHTTPRequestHandler` can be enabled in the command line by passing -the ``--cgi`` option.:: +the ``--cgi`` option:: python -m http.server --cgi 8000 diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -134,7 +134,7 @@ When :func:`reload` is executed: - * Python modules' code is recompiled and the module-level code re-executed, + * Python module's code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module's dictionary by reusing the :term:`loader` which originally loaded the module. The ``init`` function of extension modules is not called a second diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -729,6 +729,11 @@ Consider using the new :ref:`Signature Object ` interface, which provides a better way of introspecting functions. + .. versionchanged:: 3.4 + This function is now based on :func:`signature`, but still ignores + ``__wrapped__`` attributes and includes the already bound first + parameter in the signature output for bound methods. + .. function:: getargvalues(frame) diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -686,6 +686,7 @@ :exc:`UnsupportedOperation`. .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -154,20 +154,20 @@ .. attribute:: is_private ``True`` if the address is allocated for private networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). .. attribute:: is_global ``True`` if the address is allocated for public networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). - .. versionadded:: 3.4 + .. versionadded:: 3.4 .. attribute:: is_unspecified - ``True`` if the address is unspecified. See :RFC:`5375` (for IPv4) + ``True`` if the address is unspecified. See :RFC:`5735` (for IPv4) or :RFC:`2373` (for IPv6). .. attribute:: is_reserved @@ -184,6 +184,9 @@ ``True`` if the address is reserved for link-local usage. See :RFC:`3927`. +.. _iana-ipv4-special-registry: http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml +.. _iana-ipv6-special-registry: http://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml + .. class:: IPv6Address(address) @@ -218,18 +221,23 @@ The long form of the address representation, with all leading zeroes and groups consisting entirely of zeroes included. + + For the following attributes, see the corresponding documention of the + :class:`IPv4Address` class: + .. attribute:: packed .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast .. attribute:: is_private + .. attribute:: is_global .. attribute:: is_unspecified .. attribute:: is_reserved .. attribute:: is_loopback .. attribute:: is_link_local - Refer to the corresponding attribute documentation in - :class:`IPv4Address` + .. versionadded:: 3.4 + is_global .. attribute:: is_site_local diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -148,7 +148,9 @@ send it to the socket as a string of bytes preceded by a four-byte length string packed in binary using ``struct.pack('>L', n)``. - .. note:: Because portions of the configuration are passed through + .. note:: + + Because portions of the configuration are passed through :func:`eval`, use of this function may open its users to a security risk. While the function only binds to a socket on ``localhost``, and so does not accept connections from remote machines, there are scenarios where @@ -752,7 +754,9 @@ :class:`~logging.Formatter` can present exception tracebacks in an expanded or condensed format. -.. note:: Due to the use of :func:`eval` as described above, there are +.. note:: + + Due to the use of :func:`eval` as described above, there are potential security risks which result from using the :func:`listen` to send and receive configurations via sockets. The risks are limited to where multiple users with no mutual trust run code on the same machine; see the diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -36,9 +36,9 @@ .. function:: copysign(x, y) - Return *x* with the sign of *y*. On a platform that supports - signed zeros, ``copysign(1.0, -0.0)`` returns *-1.0*. - + Return a float with the magnitude (absolute value) of *x* but the sign of + *y*. On platforms that support signed zeros, ``copysign(1.0, -0.0)`` + returns *-1.0*. .. function:: fabs(x) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -136,9 +136,11 @@ Available on Unix platforms which support passing file descriptors over Unix pipes. -Before Python 3.4 *fork* was the only option available on Unix. Also, -prior to Python 3.4, child processes would inherit all the parents -inheritable handles on Windows. +.. versionchanged:: 3.4 + *spawn* added on all unix platforms, and *forkserver* added for + some unix platforms. + Child processes no longer inherit all of the parents inheritable + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named @@ -1474,7 +1476,7 @@ *exposed* is used to specify a sequence of method names which proxies for this typeid should be allowed to access using - :meth:`BaseProxy._callMethod`. (If *exposed* is ``None`` then + :meth:`BaseProxy._callmethod`. (If *exposed* is ``None`` then :attr:`proxytype._exposed_` is used instead if it exists.) In the case where no exposed list is specified, all "public methods" of the shared object will be accessible. (Here a "public method" means any attribute @@ -1853,25 +1855,30 @@ callbacks and has a parallel map implementation. *processes* is the number of worker processes to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. If - *initializer* is not ``None`` then each worker process will call + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + *context* can be used to specify the context used for starting + the worker processes. Usually a pool is created using the + function :func:`multiprocessing.Pool` or the :meth:`Pool` method + of a context object. In both cases *context* is set + appropriately. + Note that the methods of the pool object should only be called by the process which created the pool. .. versionadded:: 3.2 - *maxtasksperchild* is the number of tasks a worker process can complete - before it will exit and be replaced with a fresh worker process, to enable - unused resources to be freed. The default *maxtasksperchild* is None, which - means worker processes will live as long as the pool. + *maxtasksperchild* .. versionadded:: 3.4 - *context* can be used to specify the context used for starting - the worker processes. Usually a pool is created using the - function :func:`multiprocessing.Pool` or the :meth:`Pool` method - of a context object. In both cases *context* is set - appropriately. + *context* .. note:: diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -240,9 +240,9 @@ .. function:: length_hint(obj, default=0) - Return an estimated length for the object *o*. First trying to return its + Return an estimated length for the object *o*. First try to return its actual length, then an estimate using :meth:`object.__length_hint__`, and - finally returning the default value. + finally return the default value. .. versionadded:: 3.4 diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -193,11 +193,17 @@ .. function:: ismount(path) - Return ``True`` if pathname *path* is a :dfn:`mount point`: a point in a file - system where a different file system has been mounted. The function checks - whether *path*'s parent, :file:`path/..`, is on a different device than *path*, - or whether :file:`path/..` and *path* point to the same i-node on the same - device --- this should detect mount points for all Unix and POSIX variants. + Return ``True`` if pathname *path* is a :dfn:`mount point`: a point in a + file system where a different file system has been mounted. On POSIX, the + function checks whether *path*'s parent, :file:`path/..`, is on a different + device than *path*, or whether :file:`path/..` and *path* point to the same + i-node on the same device --- this should detect mount points for all Unix + and POSIX variants. On Windows, a drive letter root and a share UNC are + always mount points, and for any other path ``GetVolumePathName`` is called + to see if it is different from the input path. + + .. versionadded:: 3.4 + Support for detecting non-root mount points on Windows. .. function:: join(path1[, path2[, ...]]) @@ -251,7 +257,7 @@ .. function:: samefile(path1, path2) Return ``True`` if both pathname arguments refer to the same file or directory. - On Unix, this is determined by the device number and i-node number and raises an + This is determined by the device number and i-node number and raises an exception if a :func:`os.stat` call on either pathname fails. Availability: Unix, Windows. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -260,7 +260,9 @@ Availability: Unix. - .. note:: On Mac OS X, :func:`getgroups` behavior differs somewhat from + .. note:: + + On Mac OS X, :func:`getgroups` behavior differs somewhat from other Unix platforms. If the Python interpreter was built with a deployment target of :const:`10.5` or earlier, :func:`getgroups` returns the list of effective group ids associated with the current user process; @@ -940,8 +942,9 @@ the C library. .. versionchanged:: 3.4 - Add :data:`O_TMPFILE` constant. It's only available on Linux Kernel 3.11 - or newer. + Add :data:`O_PATH` on systems that support it. + Add :data:`O_TMPFILE`, only available on Linux Kernel 3.11 + or newer. .. function:: openpty() @@ -2653,7 +2656,7 @@ Fork a child process. Return ``0`` in the child and the child's process id in the parent. If an error occurs :exc:`OSError` is raised. - Note that some platforms including FreeBSD <= 6.3, Cygwin and OS/2 EMX have + Note that some platforms including FreeBSD <= 6.3 and Cygwin have known issues when using fork() from a thread. .. warning:: @@ -2899,7 +2902,6 @@ :manpage:`times(2)` or the corresponding Windows Platform API documentation. On Windows, only :attr:`user` and :attr:`system` are known; the other attributes are zero. - On OS/2, only :attr:`elapsed` is known; the other attributes are zero. Availability: Unix, Windows. diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -522,6 +522,36 @@ ValueError: '/etc/passwd' does not start with '/usr' +.. method:: PurePath.with_name(name) + + Return a new path with the :attr:`name` changed. If the original path + doesn't have a name, ValueError is raised:: + + >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz') + >>> p.with_name('setup.py') + PureWindowsPath('c:/Downloads/setup.py') + >>> p = PureWindowsPath('c:/') + >>> p.with_name('setup.py') + Traceback (most recent call last): + File "", line 1, in + File "/home/antoine/cpython/default/Lib/pathlib.py", line 751, in with_name + raise ValueError("%r has an empty name" % (self,)) + ValueError: PureWindowsPath('c:/') has an empty name + + +.. method:: PurePath.with_suffix(suffix) + + Return a new path with the :attr:`suffix` changed. If the original path + doesn't have a suffix, the new *suffix* is appended instead:: + + >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz') + >>> p.with_suffix('.bz2') + PureWindowsPath('c:/Downloads/pathlib.tar.bz2') + >>> p = PureWindowsPath('README') + >>> p.with_suffix('.txt') + PureWindowsPath('README.txt') + + .. _concrete-paths: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -147,6 +147,7 @@ *prefix* is a string to output on the front of every module name on output. .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and @@ -185,6 +186,7 @@ walk_packages(ctypes.__path__, ctypes.__name__ + '.') .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -32,6 +32,9 @@ (but only with string keys), :class:`Data`, :class:`bytes`, :class:`bytesarray` or :class:`datetime.datetime` objects. +.. versionchanged:: 3.4 + New API, old API deprecated. Support for binary format plists added. + .. seealso:: `PList manual page `_ @@ -107,7 +110,7 @@ An :exc:`OverflowError` will be raised for integer values that cannot be represented in (binary) plist files. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: dumps(value, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) @@ -116,7 +119,7 @@ the documentation for :func:`dump` for an explanation of the keyword arguments of this function. - .. versionadded: 3.4 + .. versionadded:: 3.4 The following functions are deprecated: @@ -135,7 +138,7 @@ to ``__getitem_``. This means that you can use attribute access to access items of these dictionaries. - .. deprecated: 3.4 Use :func:`load` instead. + .. deprecated:: 3.4 Use :func:`load` instead. .. function:: writePlist(rootObject, pathOrFile) @@ -143,7 +146,7 @@ Write *rootObject* to an XML plist file. *pathOrFile* may be either a file name or a (writable and binary) file object - .. deprecated: 3.4 Use :func:`dump` instead. + .. deprecated:: 3.4 Use :func:`dump` instead. .. function:: readPlistFromBytes(data) @@ -194,7 +197,7 @@ .. deprecated:: 3.4 Use a :class:`bytes` object instead -The following constants are avaiable: +The following constants are available: .. data:: FMT_XML diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -84,3 +84,8 @@ .. versionchanged:: 3.2 Added the ``-b`` option, deprecated the ``-g`` option. + +.. versionchanged:: 3.4 + :mod:`pydoc` now uses :func:`inspect.signature` rather than + :func:`inspect.getfullargspec` to extract signature information from + callables. diff --git a/Doc/library/site.rst b/Doc/library/site.rst --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -123,9 +123,13 @@ configure the :mod:`rlcompleter` module, if Python is started in :ref:`interactive mode ` and without the :option:`-S` option. The default behavior is enable tab-completion and to use -:file:`~/.python_history` as the history save file. To disable it, override -the :data:`sys.__interactivehook__` attribute in your :mod:`sitecustomize` -or :mod:`usercustomize` module or your :envvar:`PYTHONSTARTUP` file. +:file:`~/.python_history` as the history save file. To disable it, delete (or +override) the :data:`sys.__interactivehook__` attribute in your +:mod:`sitecustomize` or :mod:`usercustomize` module or your +:envvar:`PYTHONSTARTUP` file. + +.. versionchanged:: 3.4 + Activation of rlcompleter and history was made automatic. Module contents diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -117,6 +117,9 @@ Subclass of :exc:`OSError` that is the base exception class for all the other exceptions provided by this module. + .. versionchanged:: 3.4 + SMTPException became subclass of :exc:`OSError` + .. exception:: SMTPServerDisconnected diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -6,8 +6,7 @@ This module provides access to the BSD *socket* interface. It is available on -all modern Unix systems, Windows, MacOS, OS/2, and probably additional -platforms. +all modern Unix systems, Windows, MacOS, and probably additional platforms. .. note:: @@ -468,7 +467,7 @@ (10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))] .. versionchanged:: 3.2 - parameters can now be passed as single keyword arguments. + parameters can now be passed using keyword arguments. .. function:: getfqdn([name]) @@ -634,6 +633,9 @@ Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. function:: inet_ntop(address_family, packed_ip) @@ -650,6 +652,9 @@ Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -794,6 +799,7 @@ :keyword:`with` statement around them. .. note:: + :meth:`close()` releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call :meth:`shutdown()` @@ -1512,4 +1518,3 @@ details of socket semantics. For Unix, refer to the manual pages; for Windows, see the WinSock (or Winsock 2) specification. For IPv6-ready APIs, readers may want to refer to :rfc:`3493` titled Basic Socket Interface Extensions for IPv6. - diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -818,6 +818,7 @@ 'version': 3} .. note:: + To validate a certificate for a particular service, you can use the :func:`match_hostname` function. @@ -841,10 +842,8 @@ .. versionchanged:: 3.4 :exc:`ValueError` is raised when the handshake isn't done. - - .. versionchanged:: 3.4 The returned dictionary includes additional X509v3 extension items - such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. + such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. .. method:: SSLSocket.cipher() diff --git a/Doc/library/stringprep.rst b/Doc/library/stringprep.rst --- a/Doc/library/stringprep.rst +++ b/Doc/library/stringprep.rst @@ -3,7 +3,6 @@ .. module:: stringprep :synopsis: String preparation, as per RFC 3453 - :deprecated: .. moduleauthor:: Martin v. L?wis .. sectionauthor:: Martin v. L?wis diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -629,6 +629,12 @@ :exc:`TimeoutExpired` exception. It is safe to catch this exception and retry the wait. + .. note:: + + The function is implemented using a busy loop (non-blocking call and + short sleeps). Use the :mod:`asyncio` module for an asynchronous wait: + see :class:`asyncio.create_subprocess_exec`. + .. warning:: This will deadlock when using ``stdout=PIPE`` and/or @@ -639,6 +645,11 @@ .. versionchanged:: 3.3 *timeout* was added. + .. deprecated:: 3.4 + + Do not use the undocumented *endtime* parameter. It is was + unintentionally exposed in 3.3 but was intended to be private + for internal use. Use *timeout* instead. .. method:: Popen.communicate(input=None, timeout=None) @@ -1080,8 +1091,10 @@ >>> subprocess.getstatusoutput('/bin/junk') (256, 'sh: /bin/junk: not found') - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added .. function:: getoutput(cmd) @@ -1094,8 +1107,10 @@ >>> subprocess.getoutput('ls /bin/ls') '/bin/ls' - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added Notes diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -251,7 +251,7 @@ Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.writeframes(data) @@ -259,7 +259,7 @@ Write audio frames and make sure *nframes* is correct. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.close() diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -227,7 +227,9 @@ installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* is the version number of Python, for example ``3.2``. - .. note:: If a :ref:`virtual environment ` is in effect, this + .. note:: + + If a :ref:`virtual environment ` is in effect, this value will be changed in ``site.py`` to point to the virtual environment. The value for the Python installation will still be available, via :data:`base_exec_prefix`. @@ -692,10 +694,11 @@ .. data:: __interactivehook__ - When present, this function is automatically called (with no arguments) - when the interpreter is launched in :ref:`interactive mode `. - This is done after the :envvar:`PYTHONSTARTUP` file is read, so that you - can set this hook there. + When this attribute exists, its value is automatically called (with no + arguments) when the interpreter is launched in :ref:`interactive mode + `. This is done after the :envvar:`PYTHONSTARTUP` file is + read, so that you can set this hook there. The :mod:`site` module + :ref:`sets this `. .. versionadded:: 3.4 diff --git a/Doc/library/test.rst b/Doc/library/test.rst --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -199,6 +199,7 @@ test suite. .. note:: + :mod:`test.support` is not a public module. It is documented here to help Python developers write tests. The API of this module is subject to change without backwards compatibility concerns between releases. diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -40,13 +40,14 @@ :func:`wrap`. -.. function:: shorten(text, width=70, *, placeholder=" [...]") +.. function:: shorten(text, width, **kwargs) - Collapse and truncate the given text to fit in the given width. + Collapse and truncate the given *text* to fit in the given *width*. - The text first has its whitespace collapsed. If it then fits in - the *width*, it is returned unchanged. Otherwise, as many words - as possible are joined and then the *placeholder* is appended:: + First the whitespace in *text* is collapsed (all whitespace is replaced by + single spaces). If the result fits in the *width*, it is returned. + Otherwise, enough words are dropped from the end so that the remaining words + plus the :attr:`placeholder` fit within :attr:`width`:: >>> textwrap.shorten("Hello world!", width=12) 'Hello world!' @@ -55,6 +56,12 @@ >>> textwrap.shorten("Hello world", width=10, placeholder="...") 'Hello...' + Optional keyword arguments correspond to the instance attributes of + :class:`TextWrapper`, documented below. Note that the whitespace is + collapsed before the text is passed to the :class:`TextWrapper` :meth:`fill` + function, so changing the value of :attr:`.tabsize`, :attr:`.expand_tabs`, + :attr:`.drop_whitespace`, and :attr:`.replace_whitespace` will have no effect. + .. versionadded:: 3.4 @@ -110,8 +117,8 @@ :func:`wrap`, :func:`fill` and :func:`shorten` work by creating a :class:`TextWrapper` instance and calling a single method on it. That instance is not reused, so for applications that process many text -strings, it may be more efficient to create your own -:class:`TextWrapper` object. +strings using :func:`wrap` and/or :func:`fill`, it may be more efficient to +create your own :class:`TextWrapper` object. Text is preferably wrapped on whitespaces and right after the hyphens in hyphenated words; only then will long words be broken if necessary, unless @@ -252,16 +259,16 @@ .. attribute:: max_lines - (default: ``None``) If not ``None``, then the text be will truncated to - *max_lines* lines. + (default: ``None``) If not ``None``, then the output will contain at most + *max_lines* lines, with *placeholder* appearing at the end of the output. .. versionadded:: 3.4 .. attribute:: placeholder - (default: ``' [...]'``) String that will be appended to the last line of - text if it will be truncated. + (default: ``' [...]'``) String that will appear at the end of the output + text if it has been truncated. .. versionadded:: 3.4 diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -184,6 +184,7 @@ Code to display the 10 lines allocating the most memory with a pretty output, ignoring ```` and ```` files:: + import linecache import os import tracemalloc @@ -201,6 +202,9 @@ filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)) + line = linecache.getline(frame.filename, frame.lineno).strip() + if line: + print(' %s' % line) other = top_stats[limit:] if other: @@ -218,19 +222,28 @@ Example of output of the Python test suite:: - 2013-11-08 14:16:58.149320: Top 10 lines - #1: collections/__init__.py:368: 291.9 KiB - #2: Lib/doctest.py:1291: 200.2 KiB - #3: unittest/case.py:571: 160.3 KiB - #4: Lib/abc.py:133: 99.8 KiB - #5: urllib/parse.py:476: 71.8 KiB - #6: :5: 62.7 KiB - #7: Lib/base64.py:140: 59.8 KiB - #8: Lib/_weakrefset.py:37: 51.8 KiB - #9: collections/__init__.py:362: 50.6 KiB - #10: test/test_site.py:56: 48.0 KiB - 7496 other: 4161.9 KiB - Total allocated size: 5258.8 KiB + Top 10 lines + #1: Lib/base64.py:414: 419.8 KiB + _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] + #2: Lib/base64.py:306: 419.8 KiB + _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] + #3: collections/__init__.py:368: 293.6 KiB + exec(class_definition, namespace) + #4: Lib/abc.py:133: 115.2 KiB + cls = super().__new__(mcls, name, bases, namespace) + #5: unittest/case.py:574: 103.1 KiB + testMethod() + #6: Lib/linecache.py:127: 95.4 KiB + lines = fp.readlines() + #7: urllib/parse.py:476: 71.8 KiB + for a in _hexdig for b in _hexdig} + #8: :5: 62.0 KiB + #9: Lib/_weakrefset.py:37: 60.0 KiB + self.data = set() + #10: Lib/base64.py:142: 59.8 KiB + _b32tab2 = [a + b for a in _b32tab for b in _b32tab] + 6220 other: 3602.8 KiB + Total allocated size: 5303.1 KiB See :meth:`Snapshot.statistics` for more options. diff --git a/Doc/library/types.rst b/Doc/library/types.rst --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -15,6 +15,9 @@ Python interpreter, but not exposed as builtins like :class:`int` or :class:`str` are. +Finally, it provides some additional type-related utility classes and functions +that are not fundamental enough to be builtins. + Dynamic Type Creation --------------------- @@ -220,6 +223,9 @@ Return a new view of the underlying mapping's values. +Additional Utility Classes and Functions +---------------------------------------- + .. class:: SimpleNamespace A simple :class:`object` subclass that provides attribute access to its @@ -246,3 +252,18 @@ instead. .. versionadded:: 3.3 + + +.. function:: DynamicClassAttribute(fget=None, fset=None, fdel=None, doc=None) + + Route attribute access on a class to __getattr__. + + This is a descriptor, used to define attributes that act differently when + accessed through an instance and through a class. Instance access remains + normal, but access to an attribute through a class will be routed to the + class's __getattr__ method; this is done by raising AttributeError. + + This allows one to have properties active on an instance, and have virtual + attributes on the class with the same name (see Enum for an example). + + .. versionadded:: 3.4 diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst --- a/Doc/library/undoc.rst +++ b/Doc/library/undoc.rst @@ -20,7 +20,7 @@ documented beyond this mention. There's little need to document these. :mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32, Win64, WinCE, and OS/2 platforms. + --- Implementation of :mod:`os.path` on Win32, Win64, and WinCE platforms. :mod:`posixpath` --- Implementation of :mod:`os.path` on POSIX. diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -426,7 +426,7 @@ As an alternative `patch`, `patch.object` and `patch.dict` can be used as class decorators. When used in this way it is the same as applying the -decorator indvidually to every method whose name starts with "test". +decorator individually to every method whose name starts with "test". .. _further-examples: diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -958,7 +958,7 @@ .. [#] The only exceptions are magic methods and attributes (those that have leading and trailing double underscores). Mock doesn't create these but - instead of raises an ``AttributeError``. This is because the interpreter + instead raises an ``AttributeError``. This is because the interpreter will often implicitly request these methods, and gets *very* confused to get a new Mock object when it expects a magic method. If you need magic method support see :ref:`magic methods `. @@ -1509,7 +1509,7 @@ Both patch_ and patch.object_ correctly patch and restore descriptors: class methods, static methods and properties. You should patch these on the *class* rather than an instance. They also work with *some* objects -that proxy attribute access, like the `django setttings object +that proxy attribute access, like the `django settings object `_. diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -239,9 +239,10 @@ Unittest supports simple test discovery. In order to be compatible with test discovery, all of the test files must be :ref:`modules ` or -:ref:`packages ` importable from the top-level directory of -the project (this means that their filenames must be valid -:ref:`identifiers `). +:ref:`packages ` (including :term:`namespace packages +`) importable from the top-level directory of +the project (this means that their filenames must be valid :ref:`identifiers +`). Test discovery is implemented in :meth:`TestLoader.discover`, but can also be used from the command line. The basic command-line usage is:: @@ -306,6 +307,9 @@ Test modules and packages can customize test loading and discovery by through the `load_tests protocol`_. +.. versionchanged:: 3.4 + Test discovery supports :term:`namespace packages `. + .. _organizing-tests: @@ -1620,11 +1624,11 @@ .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) - Find and return all test modules from the specified start directory, - recursing into subdirectories to find them. Only test files that match - *pattern* will be loaded. (Using shell style pattern matching.) Only - module names that are importable (i.e. are valid Python identifiers) will - be loaded. + Find all the test modules by recursing into subdirectories from the + specified start directory, and return a TestSuite object containing them. + Only test files that match *pattern* will be loaded. (Using shell style + pattern matching.) Only module names that are importable (i.e. are valid + Python identifiers) will be loaded. All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level @@ -1654,12 +1658,11 @@ .. versionchanged:: 3.4 Modules that raise :exc:`SkipTest` on import are recorded as skips, - not errors. - - .. versionchanged:: 3.4 - Paths are sorted before being imported to ensure execution order for a - given test suite is the same even if the underlying file system's ordering - is not dependent on file name like in ext3/4. + not errors. + Discovery works for :term:`namespace packages `. + Paths are sorted before being imported so that execution order is + the same even if the underlying file system's ordering is not + dependent on file name. The following attributes of a :class:`TestLoader` can be configured either by diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -48,7 +48,7 @@ .. attribute:: headers - The HTTP response headers for the HTTP request that cause the + The HTTP response headers for the HTTP request that caused the :exc:`HTTPError`. .. versionadded:: 3.4 diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -218,7 +218,7 @@ fetching of the image, this should be true. *method* should be a string that indicates the HTTP request method that - will be used (e.g. ``'HEAD'``). Its value is stored in the + will be used (e.g. ``'HEAD'``). If provided, its value is stored in the :attr:`~Request.method` attribute and is used by :meth:`get_method()`. Subclasses may indicate a default method by setting the :attr:`~Request.method` attribute in the class itself. @@ -440,13 +440,20 @@ .. attribute:: Request.method - The HTTP request method to use. This value is used by - :meth:`~Request.get_method` to override the computed HTTP request - method that would otherwise be returned. This attribute is initialized with - the value of the *method* argument passed to the constructor. + The HTTP request method to use. By default its value is :const:`None`, + which means that :meth:`~Request.get_method` will do its normal computation + of the method to be used. Its value can be set (thus overriding the default + computation in :meth:`~Request.get_method`) either by providing a default + value by setting it at the class level in a :class:`Request` subclass, or by + passing a value in to the :class:`Request` constructor via the *method* + argument. .. versionadded:: 3.3 + .. versionchanged:: 3.4 + A default value can now be set in subclasses; previously it could only + be set via the constructor argument. + .. method:: Request.get_method() diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -76,6 +76,8 @@ without there needing to be any reference to its venv in ``PATH``. +.. _venv-api: + API --- diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -150,14 +150,30 @@ Wave_write Objects ------------------ +For seekable output streams, the ``wave`` header will automatically be updated +to reflect the number of frames actually written. For unseekable streams, the +*nframes* value must be accurate when the first frame data is written. An +accurate *nframes* value can be achieved either by calling +:meth:`~Wave_write.setnframes` or :meth:`~Wave_write.setparams` with the number +of frames that will be written before :meth:`~Wave_write.close` is called and +then using :meth:`~Wave_write.writeframesraw` to write the frame data, or by +calling :meth:`~Wave_write.writeframes` with all of the frame data to be +written. In the latter case :meth:`~Wave_write.writeframes` will calculate +the number of frames in the data and set *nframes* accordingly before writing +the frame data. + Wave_write objects, as returned by :func:`.open`, have the following methods: +.. versionchanged:: 3.4 + Added support for unseekable files. + .. method:: Wave_write.close() Make sure *nframes* is correct, and close the file if it was opened by - :mod:`wave`. This method is called upon object collection. Can raise an - exception if *nframes* is not correct and a file is not seekable. + :mod:`wave`. This method is called upon object collection. It will raise + an exception if the output stream is not seekable and *nframes* does not + match the number of frames actually written. .. method:: Wave_write.setnchannels(n) @@ -181,8 +197,9 @@ .. method:: Wave_write.setnframes(n) - Set the number of frames to *n*. This will be changed later if more frames are - written. + Set the number of frames to *n*. This will be changed later if the number + of frames actually written is different (this update attempt will + raise an error if the output stream is not seekable). .. method:: Wave_write.setcomptype(type, name) @@ -209,16 +226,18 @@ Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: Wave_write.writeframes(data) - Write audio frames and make sure *nframes* is correct. Can raise an - exception if a file is not seekable. + Write audio frames and make sure *nframes* is correct. It will raise an + error if the output stream is not seekable and the total number of frames + that have been written after *data* has been written does not match the + previously set value for *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. Note that it is invalid to set any parameters after calling :meth:`writeframes` diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -952,7 +952,8 @@ specified in the XML file. .. deprecated:: 3.4 - The *html* argument. + The *html* argument. The remaining arguments should be passed via + keywword to prepare for the removal of the *html* argument. .. method:: close() diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -14,9 +14,9 @@ .. warning:: The XML modules are not secure against erroneous or maliciously - constructed data. If you need to parse untrusted or unauthenticated data see - :ref:`xml-vulnerabilities`. - + constructed data. If you need to parse untrusted or + unauthenticated data see the :ref:`xml-vulnerabilities` and + :ref:`defused-packages` sections. It is important to note that modules in the :mod:`xml` package require that there be at least one SAX-compliant XML parser available. The Expat parser is @@ -46,16 +46,15 @@ .. _xml-vulnerabilities: XML vulnerabilities -=================== +------------------- The XML processing modules are not secure against maliciously constructed data. -An attacker can abuse vulnerabilities for e.g. denial of service attacks, to -access local files, to generate network connections to other machines, or -to or circumvent firewalls. The attacks on XML abuse unfamiliar features -like inline `DTD`_ (document type definition) with entities. +An attacker can abuse XML features to carry out denial of service attacks, +access local files, generate network connections to other machines, or +circumvent firewalls. -The following table gives an overview of the known attacks and if the various -modules are vulnerable to them. +The following table gives an overview of the known attacks and whether +the various modules are vulnerable to them. ========================= ======== ========= ========= ======== ========= kind sax etree minidom pulldom xmlrpc @@ -68,7 +67,7 @@ ========================= ======== ========= ========= ======== ========= 1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a - ParserError when an entity occurs. + :exc:`ParserError` when an entity occurs. 2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns the unexpanded entity verbatim. 3. :mod:`xmlrpclib` doesn't expand external entities and omits them. @@ -77,23 +76,21 @@ billion laughs / exponential entity expansion The `Billion Laughs`_ attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. Each entity refers to another entity - several times, the final entity definition contains a small string. Eventually - the small string is expanded to several gigabytes. The exponential expansion - consumes lots of CPU time, too. + several times, and the final entity definition contains a small string. + The exponential expansion results in several gigabytes of text and + consumes lots of memory and CPU time. quadratic blowup entity expansion A quadratic blowup attack is similar to a `Billion Laughs`_ attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of thousand chars over and over again. The attack isn't as - efficient as the exponential case but it avoids triggering countermeasures of - parsers against heavily nested entities. + efficient as the exponential case but it avoids triggering parser countermeasures + that forbid deeply-nested entities. external entity expansion Entity declarations can contain more than just text for replacement. They can - also point to external resources by public identifiers or system identifiers. - System identifiers are standard URIs or can refer to local files. The XML - parser retrieves the resource with e.g. HTTP or FTP requests and embeds the - content into the XML document. + also point to external resources or local files. The XML + parser accesses the resource and embeds the content into the XML document. DTD retrieval Some XML libraries like Python's :mod:`xml.dom.pulldom` retrieve document type @@ -101,31 +98,32 @@ implications as the external entity expansion issue. decompression bomb - The issue of decompression bombs (aka `ZIP bomb`_) apply to all XML libraries - that can parse compressed XML stream like gzipped HTTP streams or LZMA-ed + Decompression bombs (aka `ZIP bomb`_) apply to all XML libraries + that can parse compressed XML streams such as gzipped HTTP streams or + LZMA-compressed files. For an attacker it can reduce the amount of transmitted data by three magnitudes or more. -The documentation of `defusedxml`_ on PyPI has further information about +The documentation for `defusedxml`_ on PyPI has further information about all known attack vectors with examples and references. -defused packages ----------------- +.. _defused-packages: + +The :mod:`defusedxml` and :mod:`defusedexpat` Packages +------------------------------------------------------ `defusedxml`_ is a pure Python package with modified subclasses of all stdlib -XML parsers that prevent any potentially malicious operation. The courses of -action are recommended for any server code that parses untrusted XML data. The -package also ships with example exploits and an extended documentation on more -XML exploits like xpath injection. +XML parsers that prevent any potentially malicious operation. Use of this +package is recommended for any server code that parses untrusted XML data. The +package also ships with example exploits and extended documentation on more +XML exploits such as XPath injection. -`defusedexpat`_ provides a modified libexpat and patched replacment -:mod:`pyexpat` extension module with countermeasures against entity expansion -DoS attacks. Defusedexpat still allows a sane and configurable amount of entity -expansions. The modifications will be merged into future releases of Python. - -The workarounds and modifications are not included in patch releases as they -break backward compatibility. After all inline DTD and entity expansion are -well-definied XML features. +`defusedexpat`_ provides a modified libexpat and a patched +:mod:`pyexpat` module that have countermeasures against entity expansion +DoS attacks. The :mod:`defusedexpat` module still allows a sane and configurable amount of entity +expansions. The modifications may be included in some future release of Python, +but will not be included in any bugfix releases of +Python because they break backward compatibility. .. _defusedxml: https://pypi.python.org/pypi/defusedxml/ @@ -133,4 +131,3 @@ .. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs .. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb .. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition - diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -401,18 +401,32 @@ ``2``, only files with that optimization level (see :func:`compile`) are added to the archive, compiling if necessary. - If the pathname is a file, the filename must end with :file:`.py`, and + If *pathname* is a file, the filename must end with :file:`.py`, and just the (corresponding :file:`\*.py[co]`) file is added at the top level - (no path information). If the pathname is a file that does not end with + (no path information). If *pathname* is a file that does not end with :file:`.py`, a :exc:`RuntimeError` will be raised. If it is a directory, and the directory is not a package directory, then all the files :file:`\*.py[co]` are added at the top level. If the directory is a package directory, then all :file:`\*.py[co]` are added under the package name as a file path, and if any subdirectories are package directories, - all of these are added recursively. *basename* is intended for internal - use only. When *filterfunc(pathname)* is given, it will be called for every - invocation. When it returns a false value, that path and its subpaths will - be ignored. + all of these are added recursively. + + *basename* is intended for internal use only. + + *filterfunc*, if given, must be a function taking a single string + argument. It will be passed each path (including each individual full + file path) before it is added to the archive. If *filterfunc* returns a + false value, the path will not be added, and if it is a directory its + contents will be ignored. For example, if our test files are all either + in ``test`` directories or start with the string ``test_``, we can use a + *filterfunc* to exclude them:: + + >>> zf = PyZipFile('myprog.zip') + >>> def notests(s): + ... fn = os.path.basename(s) + ... return (not (fn == 'test' or fn.startswith('test_'))) + >>> zf.writepy('myprog', filterfunc=notests) + The :meth:`writepy` method makes archives with file names like this:: diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1226,6 +1226,10 @@ The return value must be a string object. + .. versionchanged:: 3.4 + The __format__ method of ``object`` itself raises a :exc:`TypeError` + if passed any non-empty string. + .. _richcmpfuncs: .. method:: object.__lt__(self, other) @@ -1643,6 +1647,8 @@ that criterion, then the class definition will fail with ``TypeError``. +.. _prepare: + Preparing the class namespace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -2044,11 +2050,13 @@ ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation in-place (modifying *self*) and return the result (which could be, but does not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, to execute the - statement ``x += y``, where *x* is an instance of a class that has an - :meth:`__iadd__` method, ``x.__iadd__(y)`` is called. If *x* is an instance - of a class that does not define a :meth:`__iadd__` method, ``x.__add__(y)`` - and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. + assignment falls back to the normal methods. For instance, if *x* is an + instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent + to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` + are considered, as with the evaluation of ``x + y``. In certain situations, + augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in + fact part of the data model. .. method:: object.__neg__(self) diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -16,14 +16,14 @@

Python HOWTOs
in-depth documents on specific topics

+

Installing Python Modules
+ installing from the Python Package Index & other sources

+

Distributing Python Modules
+ publishing modules for installation by others

Extending and Embedding
tutorial for C/C++ programmers

Python/C API
reference for C/C++ programmers

-

Installing Python Modules
- information for installers & sys-admins

-

Distributing Python Modules
- sharing modules with others

FAQs
frequently asked questions (with answers!)

diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -282,3 +282,4 @@ whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" whatsnew/changelog,,:password,user:password +whatsnew/changelog,,:gz,w:gz diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -371,9 +371,9 @@ comma-separated values (items) between square brackets. Lists might contain items of different types, but usually the items all have the same type. :: - >>> squares = [1, 2, 4, 9, 16, 25] + >>> squares = [1, 4, 9, 16, 25] >>> squares - [1, 2, 4, 9, 16, 25] + [1, 4, 9, 16, 25] Like strings (and all other built-in :term:`sequence` type), lists can be indexed and sliced:: @@ -389,12 +389,12 @@ means that the following slice returns a new (shallow) copy of the list:: >>> squares[:] - [1, 2, 4, 9, 16, 25] + [1, 4, 9, 16, 25] Lists also supports operations like concatenation:: >>> squares + [36, 49, 64, 81, 100] - [1, 2, 4, 9, 16, 25, 36, 49, 64, 81, 100] + [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] Unlike strings, which are :term:`immutable`, lists are a :term:`mutable` type, i.e. it is possible to change their content:: diff --git a/Doc/tutorial/whatnow.rst b/Doc/tutorial/whatnow.rst --- a/Doc/tutorial/whatnow.rst +++ b/Doc/tutorial/whatnow.rst @@ -21,8 +21,8 @@ and many other tasks. Skimming through the Library Reference will give you an idea of what's available. -* :ref:`install-index` explains how to install external modules written by other - Python users. +* :ref:`installing-index` explains how to install additional modules written + by other Python users. * :ref:`reference-index`: A detailed explanation of Python's syntax and semantics. It's heavy reading, but is useful as a complete guide to the diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -60,6 +60,8 @@ prompt. +.. _building-python-on-unix: + Building Python =============== diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -11,6 +11,11 @@ Windows). It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory (on Windows, this is ``Lib\site-packages``). +.. seealso:: + + `Python Packaging User Guide: Creating and using virtual environments + `__ + .. highlight:: none On Windows, you may have to invoke the ``pyvenv`` script as follows, if you @@ -38,6 +43,8 @@ virtual environment. --symlinks Try to use symlinks rather than copies, when symlinks are not the default for the platform. + --copies Try to use copies rather than symlinks, even when + symlinks are the default for the platform. --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. @@ -47,7 +54,8 @@ environment (pip is bootstrapped by default) .. versionchanged:: 3.4 - Installs pip by default, added the ``--without-pip`` option + Installs pip by default, added the ``--without-pip`` and ``--copies`` + options If the target directory already exists an error will be raised, unless the ``--clear`` or ``--upgrade`` option was provided. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -11,6 +11,10 @@ This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. +.. XXX (ncoghlan) + + This looks rather stale to me... + Installing Python ================= diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2,8 +2,7 @@ What's New In Python 3.4 **************************** -.. :Author: Someone - (uncomment if there is a principal author) +:Author: R. David Murray (Editor) .. Rules for maintenance: @@ -68,11 +67,6 @@ For full details, see the `changelog `_. -.. note:: Prerelease users should be aware that this document is currently in - draft form. While it should be close to complete for the Python 3.4 - release candidates, adjustments and additions to the document may be made - up until the final release. - .. seealso:: @@ -90,17 +84,19 @@ * No new syntax features were added in Python 3.4. -New expected features for Python implementations: - -* :ref:`pip should always be "available" ` (:pep:`453`). -* :ref:`Make newly created file descriptors non-inheritable ` +Other new features: + +* :ref:`pip should always be available ` (:pep:`453`). +* :ref:`Newly created file descriptors are non-inheritable ` (:pep:`446`). -* command line option for :ref:`isolated mode `, +* command line option for :ref:`isolated mode ` (:issue:`16499`). * :ref:`improvements in the handling of codecs ` that are not text encodings (multiple issues). * :ref:`A ModuleSpec Type ` for the Import System (:pep:`451`). (Affects importer authors.) +* The :mod:`marshal` format has been made :ref:`more compact and efficient + ` (:issue:`16475`). New library modules: @@ -120,33 +116,57 @@ * :mod:`tracemalloc`: :ref:`Trace Python memory allocations ` (:pep:`454`). -Significantly Improved Library Modules: +Significantly improved library modules: * :ref:`Single-dispatch generic functions ` in :mod:`functools` (:pep:`443`). * New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`). -* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl` - (:issue:`16692`). * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME handling ` (:issue:`18891`). +* The :mod:`inspect` and :mod:`pydoc` modules are now capable of + correct introspection of a much wider variety of callable objects, + which improves the output of the Python :func:`help` system. * The :mod:`ipaddress` module API has been declared stable - -CPython implementation improvements: - -* :ref:`Safe object finalization ` (:pep:`442`). -* Leveraging :pep:`442`, :ref:`module globals are no longer set to None - during finalization `, in most cases (:issue:`18214`). -* :ref:`Configurable memory allocators ` (:pep:`445`). +Security improvements: + * :ref:`Secure and interchangeable hash algorithm ` (:pep:`456`). +* :ref:`Make newly created file descriptors non-inheritable ` + (:pep:`446`) to avoid leaking file descriptors to child processes. +* New command line option for :ref:`isolated mode `, + (:issue:`16499`). +* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork + on Unix `. *spawn* and *forkserver* are + more secure because they avoid sharing data with child processes. +* :mod:`multiprocessing` child processes on Windows no longer inherit + all of the parent's inheritable handles, only the necessary ones. +* A new :func:`hashlib.pbkdf2_hmac` function provides + the `PKCS#5 password-based key derivation function 2 + `_. +* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. +* :ref:`Retrieving certificates from the Windows system cert store support + ` for :mod:`ssl`. +* :ref:`Server-side SNI (Server Name Indication) support + ` for :mod:`ssl`. +* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements + `. +* All modules in the standard library that support SSL now support server + certificate verification, including hostname matching + (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see + :func:`ssl.SSLContext.load_verify_locations`). + +CPython implementation improvements: + +* :ref:`Safe object finalization ` (:pep:`442`). +* Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set + to None during finalization ` (:issue:`18214`). +* :ref:`Configurable memory allocators ` (:pep:`445`). * :ref:`Argument Clinic ` (:pep:`436`). -* The :mod:`marshal` format has been made :ref:`more compact and efficient - ` (:issue:`16475`). Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential @@ -154,38 +174,54 @@ -New Expected Features for Python Implementations -================================================ +New Features +============ .. _whatsnew-pep-453: PEP 453: Explicit Bootstrapping of PIP in Python Installations -------------------------------------------------------------- +Bootstrapping pip By Default +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard cross-platform mechanism to bootstrap the pip installer into Python -installations and virtual environments. - -By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where -X.Y stands for the version of the Python installation), along with the -``pip`` Python package and its dependencies. - -The :mod:`venv` module and the :command:`pyvenv` utility make use of this -module to make ``pip`` readily available in virtual environments. When -using the command line interface, ``pip`` is installed by default, while -for the module API installation of ``pip`` must be requested explicitly. - -For CPython source builds on POSIX systems, the ``make install`` and -``make altinstall`` commands bootstrap ``pip`` by default. This behaviour -can be controlled through configure options, and overridden through -Makefile options. - -On Windows and Mac OS X, the CPython installers now offer the option to -install ``pip`` along with CPython itself. +installations and virtual environments. The version of ``pip`` included +with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases +will update the bundled version to the latest version of ``pip`` that is +available at the time of creating the release candidate. + +By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all +platforms (where X.Y stands for the version of the Python installation), +along with the ``pip`` Python package and its dependencies. On Windows and +in virtual environments on all platforms, the unversioned ``pip`` command +will also be installed. On other platforms, the system wide unversioned +``pip`` command typically refers to the separately installed Python 2 +version. + +The :ref:`pyvenv ` command line utility and the :mod:`venv` +module make use of the :mod:`ensurepip` module to make ``pip`` readily +available in virtual environments. When using the command line utility, +``pip`` is installed by default, while when using the :mod:`venv` module +:ref:`venv-api` installation of ``pip`` must be requested explicitly. + +For CPython :ref:`source builds on POSIX systems `, +the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by +default. This behaviour can be controlled through configure options, and +overridden through Makefile options. + +On Windows and Mac OS X, the CPython installers now default to installing +``pip`` along with CPython itself (users may opt out of installing it +during the installation process). Window users will need to opt in to the +automatic ``PATH`` modifications to have ``pip`` available from the command +line by default, otherwise it can still be accessed through the Python +launcher for Windows as ``py -m pip``. As `discussed in the PEP`__, platform packagers may choose not to install -``pip`` by default, as long as the command ``pip``, when invoked, provides -clear and simple directions on how to install ``pip`` on the platform. +these commands by default, as long as, when invoked, they provide clear and +simple directions on how to install them on that platform (usually using +the system package manager). __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors @@ -193,12 +229,29 @@ To avoid conflicts between parallel Python 2 and Python 3 installations, only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by - default when ``ensurepip`` is invoked directly (including by the CPython - installers). ``pyvenv`` ensures that the unqualified ``pip`` command is - made available in virtual environments, and ``pip`` can always be + default when ``ensurepip`` is invoked directly - the ``--default-pip`` + option is needed to also request the unversioned ``pip`` command. + ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` + command is made available in those environments, and ``pip`` can always be invoked via the ``-m`` switch rather than directly to avoid ambiguity on systems with multiple Python installations. + +Documentation Changes +~~~~~~~~~~~~~~~~~~~~~ + +As part of this change, the :ref:`installing-index` and +:ref:`distributing-index` sections of the documentation have been +completely redesigned as short getting started and FAQ documents. Most +packaging documentation has now been moved out to the Python Packaging +Authority maintained `Python Packaging User Guide +`__ and the documentation of the individual +projects. + +However, as this migration is currently still incomplete, the legacy +versions of those guides remaining available as :ref:`install-index` +and :ref:`distutils-index`. + .. seealso:: :pep:`453` -- Explicit bootstrapping of pip in Python installations @@ -208,11 +261,17 @@ .. _whatsnew-pep-446: -PEP 446: Make Newly Created File Descriptors Non-Inheritable ------------------------------------------------------------- +PEP 446: Newly Created File Descriptors Are Non-Inheritable +----------------------------------------------------------- :pep:`446` makes newly created file descriptors :ref:`non-inheritable -`. New functions and methods: +`. In general, this is the behavior an application will +want: when launching a new process, having currently open files also +open in the new process can lead to all sorts of hard to find bugs, +and potentially to security issues. + +However, there are occasions when inheritance is desired. To support +these cases, the following new functions and methods are available: * :func:`os.get_inheritable`, :func:`os.set_inheritable` * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` @@ -243,10 +302,10 @@ but were previously only discoverable through runtime introspection. Unlike the convenience methods on :class:`str`, :class:`bytes` and -:class:`bytearray`, these convenience functions support arbitrary codecs -in both Python 2 and Python 3, rather than being limited to Unicode text -encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions -(in Python 2). +:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary +codecs in both Python 2 and Python 3, rather than being limited to Unicode text +encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in +Python 2). In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these @@ -269,7 +328,7 @@ In a related change, whenever it is feasible without breaking backwards compatibility, exceptions raised during encoding and decoding operations -will be wrapped in a chained exception of the same type that mentions the +are wrapped in a chained exception of the same type that mentions the name of the codec responsible for producing the error:: >>> import codecs @@ -315,7 +374,7 @@ The binary and text transforms provided in the standard library are detailed in :ref:`binary-transforms` and :ref:`text-transforms`. -(Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, +(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, :issue:`17828` and :issue:`19619`) @@ -336,7 +395,9 @@ Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing -importers should be updated to implement the new methods. +importers should be updated to implement the new methods. See the +:ref:`deprecated-3.4` section for a list of methods that should be replaced and +their replacements. Other Language Changes @@ -346,9 +407,10 @@ * Unicode database updated to UCD version 6.3. -* :func:`min` and :func:`max` now accept a *default* argument that can be used - to specify the value they return if the iterable they are evaluating has no - elements. Contributed by Julian Berman in :issue:`18111`. +* :func:`min` and :func:`max` now accept a *default* keyword-only argument that + can be used to specify the value they return if the iterable they are + evaluating has no elements. (Contributed by Julian Berman in + :issue:`18111`.) * Module objects are now :mod:`weakref`'able. @@ -357,16 +419,42 @@ ``__main__.__file__`` when a script has been executed directly using a relative path (Contributed by Brett Cannon in :issue:`18416`). -* Now all the UTF-\* codecs (except UTF-7) reject surrogates during both +* All the UTF-\* codecs (except UTF-7) now reject surrogates during both encoding and decoding unless the ``surrogatepass`` error handler is used, - with the exception of the UTF-16 decoder that accepts valid surrogate pairs, - and the UTF-16 encoder that produces them while encoding non-BMP characters. + with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) + and the UTF-16 encoder (which produces them while encoding non-BMP characters). Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. -* New EBCDIC :ref:`codec ` ``cp273``. (Contributed by - Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) - +* New German EBCDIC :ref:`codec ` ``cp273``. (Contributed + by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) + +* New Ukrainian :ref:`codec ` ``cp1125``. (Contributed by + Serhiy Storchaka in :issue:`19668`.) + +* :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary + buffer objects as arguments. (Contributed by Antoine Pitrou in + :issue:`15958`.) + +* The :class:`int` constructor now accepts any object that has an ``__index__`` + method for its *base* argument. (Contributed by Mark Dickinson in + :issue:`16772`.) + +* Frame objects now have a :func:`~frame.clear` method that clears all + references to local variables from the frame. (Contributed by Antoine Pitrou + in :issue:`17934`.) + +* :class:`memoryview` is now registered as a :class:`Sequence `, + and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan + and Claudiu Popa in :issue:`18690` and :issue:`19078`.) + +* Signatures reported by :func:`help` have been modified and improved in + several cases as a result of the introduction of Argument Clinic and other + changes to the :mod:`inspect` and :mod:`pydoc` modules. + +* :meth:`~object.__length_hint__` is now part of the formal language + specification (see :pep:`424`). (Contributed by Armin Ronacher in + :issue:`16148`.) New Modules @@ -405,9 +493,11 @@ :mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ``ensurepip`` does not -access the internet. (If the installation has Internet access, it is of course -possible to upgrade ``pip`` to a release more recent than the bundled ``pip`` -by using the bundled ``pip`` command itself once it is installed.) +access the internet. If the installation has Internet access, after +``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a +more recent release than the bundled one. (Note that such an upgraded version +of ``pip`` is considered to be a separately installed package and will not be +removed if Python is uninstalled.) The module is named *ensure*\ pip because if called when ``pip`` is already installed, it does nothing. It also has an ``--upgrade`` option that will @@ -486,7 +576,7 @@ The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to trace memory blocks allocated by Python. It provides the following information: -* Traceback where an object was allocated +* Trace where an object was allocated * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Compute the differences between two snapshots to detect memory leaks @@ -518,7 +608,7 @@ aifc ---- -The :meth:`~aifc.getparams` method now returns a namedtuple rather than a +The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) :func:`aifc.open` now supports the context manager protocol: when used in a @@ -526,6 +616,10 @@ object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in :issue:`16486`.) +The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` +methods now accept any :term:`bytes-like object`. (Contributed by Serhiy +Storchaka in :issue:`8311`.) + argparse -------- @@ -538,10 +632,16 @@ audioop ------- -Added support for 24-bit samples (:issue:`12866`). - -Added the :func:`~audioop.byteswap` function to convert big-endian samples -to little-endian and vice versa (:issue:`19641`). +:mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka +in :issue:`12866`.) + +New :func:`~audioop.byteswap` function converts big-endian samples to +little-endian and vice versa (Contributed by Serhiy Storchaka in +:issue:`19641`). + +All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings +are not accepted: they didn't work before, now they raise an error right away. +(Contributed by Serhiy Storchaka in :issue:`16685`.) base64 @@ -549,7 +649,25 @@ The encoding and decoding functions in :mod:`base64` now accept any :term:`bytes-like object` in cases where it previously required a -:class:`bytes` or :class:`bytearray` instance (:issue:`17839`). +:class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in +:issue:`17839`.) + +New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, +:func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to +encode and decode binary data from and to ``Ascii85`` and the git/mercurial +``Base85`` formats, respectively. The ``a85`` functions have options that can +be used to make them compatible with the variants of the ``Ascii85`` encoding, +including the Adobe variant. (Contributed by Martin Morrison, the Mercurial +project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) + + +collections +----------- + +The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying +the child map to add to the chain. This allows an existing mapping and/or a +custom mapping type to be used for the child. (Contributed by Vinay Sajip in +:issue:`16613`.) colorsys @@ -570,17 +688,30 @@ Zero Piraeus in :issue:`19266`) The new :func:`contextlib.redirect_stdout` context manager makes it easier -for utility scripts to handle inflexible APIs that don't provide any -options to retrieve their output as a string or direct it to somewhere -other than :data:`sys.stdout`. In conjunction with :class:`io.StringIO`, -this context manager is also useful for checking expected output from -command line utilities. (Contribute by Raymond Hettinger in :issue:`15805`) +for utility scripts to handle inflexible APIs that write their output to +:data:`sys.stdout` and don't provide any options to redirect it. Using the +context manager, the :data:`sys.stdout` output can be redirected to any +other stream or, in conjunction with :class:`io.StringIO`, to a string. +The latter can be especially useful, for example, to capture output +from a function that was written to implement a command line interface. +It is recommended only for utility scripts because it affects the +global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger +in :issue:`15805`) The :mod:`contextlib` documentation has also been updated to include a :ref:`discussion ` of the differences between single use, reusable and reentrant context managers. +dbm +--- + +:func:`dbm.open` objects now support the context management protocol. When +used in a :keyword:`with` statement, the ``close`` method of the database +object will be called automatically at the end of the block. (Contributed by +Claudiu Popa and Nick Coghlan in :issue:`19282`.) + + dis --- @@ -639,6 +770,10 @@ (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` and Claudiu Popa in :issue:`17916`) +New function :func:`~dis.stack_effect` computes the effect on the Python stack +of a given opcode and argument, information that is not otherwise available. +(Contributed by Larry Hastings in :issue:`19722`.) + doctest ------- @@ -653,6 +788,9 @@ shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the :mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) +:mod:`doctest` will now find doctests in extension module ``__doc__`` strings. +(Contributed by Zachary Ware in :issue:`3158`.) + email ----- @@ -661,7 +799,8 @@ override the default policy of the message when generating a string representation of it. This means that ``as_string`` can now be used in more circumstances, instead of having to create and use a :mod:`~email.generator` in -order to pass formatting parameters to its ``flatten`` method. +order to pass formatting parameters to its ``flatten`` method. (Contributed by +R. David Murray in :issue:`18600`.) New method :meth:`~email.message.Message.as_bytes` added to produce a bytes representation of the message in a fashion similar to how ``as_string`` @@ -669,26 +808,30 @@ argument, but does accept the *unixfrom* and *policy* arguments. The :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive -result: a bytes object containing the fully formatted message. - -(Contributed by R. David Murray in :issue:`18600`.) +result: a bytes object containing the fully formatted message. (Contributed +by R. David Murray in :issue:`18600`.) + +The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. +When specified, the associated header will be updated without changing +its location in the list of headers. For backward compatibility, the default +is ``False``. (Contributed by R. David Murray in :issue:`18891`.) + .. _whatsnew_email_contentmanager: -A pair of new subclasses of :class:`~email.message.Message` have been added, -along with a new sub-module, :mod:`~email.contentmanager`. All documentation -is currently in the new module, which is being added as part of the new -:term:`provisional ` email API. These classes provide a -number of new methods that make extracting content from and inserting content -into email messages much easier. See the :mod:`~email.contentmanager` -documentation for details. - -These API additions complete the bulk of the work that was planned as part of -the email6 project. The currently provisional API is scheduled to become final -in Python 3.5 (possibly with a few minor additions in the area of error -handling). - -(Contributed by R. David Murray in :issue:`18891`.) +A pair of new subclasses of :class:`~email.message.Message` have been added +(:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, +:mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute +:attr:`~email.policy.EmailPolicy.content_manager`. All documentation is +currently in the new module, which is being added as part of email's new +:term:`provisional API`. These classes provide a number of new methods that +make extracting content from and inserting content into email messages much +easier. For details, see the :mod:`~email.contentmanager` documentation and +the :ref:`email-contentmanager-api-examples`. These API additions complete the +bulk of the work that was planned as part of the email6 project. The currently +provisional API is scheduled to become final in Python 3.5 (possibly with a few +minor additions in the area of error handling). (Contributed by R. David +Murray in :issue:`18891`.) filecmp @@ -701,6 +844,11 @@ than the resolution of a particular filesystem's file modification time field. (Contributed by Mark Levitt in :issue:`18149`.) +New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of +directories that are used as the default value for the *ignore* parameter of +the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in +:issue:`15442`.) + functools --------- @@ -726,6 +874,10 @@ :pep:`443` -- Single-dispatch generic functions PEP written and implemented by ?ukasz Langa. +:func:`~functools.total_ordering` now supports a return value of +:const:`NotImplemented` from the underlying comparison function. (Contributed +by Katie Miller in :issue:`10042`.) + A pure-python version of the :func:`~functools.partial` function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in @@ -740,11 +892,28 @@ (Contributed by Antoine Pitrou in :issue:`16351`.) +glob +---- + +A new function :func:`~glob.escape` provides a way to escape special characters +in a filename so that they do not become part of the globbing expansion but are +instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) + + hashlib ------- -New :func:`hashlib.pbkdf2_hmac` function. -(Contributed by Christian Heimes in :issue:`18582`) +A new :func:`hashlib.pbkdf2_hmac` function provides +the `PKCS#5 password-based key derivation function 2 +`_. (Contributed by Christian +Heimes in :issue:`18582`) + +The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now +a formally supported interface. It has always existed in CPython's +:mod:`hashlib` (although it did not return lower case names for all supported +hashes), but it was not a public interface and so some other Python +implementations have not previously supported it. (Contributed by Jason R. +Coombs in :issue:`18532`.) hmac @@ -756,20 +925,31 @@ accepts any type supported by the :mod:`hashlib` module. (Contributed by Jonas Borgstr?m in :issue:`18240`.) +The *digestmod* argument to the :func:`hmac.new` function may now be any hash +digest name recognized by :mod:`hashlib`. In addition, the current behavior in +which the value of *digestmod* defaults to ``MD5`` is deprecated: in a +future version of Python there will be no default value. (Contributed by +Christian Heimes in :issue:`17276`.) + +With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` +attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` +attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. +(Contributed by Christian Heimes in :issue:`18775`.) + html ---- -Added a new :func:`html.unescape` function that converts HTML5 character -references to the corresponding Unicode characters. -(Contributed by Ezio Melotti in :issue:`2927`) - -Added a new *convert_charrefs* keyword argument to -:class:`~html.parser.HTMLParser` that, when ``True``, automatically converts -all character references. For backward-compatibility, its value defaults -to ``False``, but it will change to ``True`` in future versions, so you -are invited to set it explicitly and update your code to use this new feature. -(Contributed by Ezio Melotti in :issue:`13633`) +New function :func:`~html.unescape` function converts HTML5 character references to +the corresponding Unicode characters. (Contributed by Ezio Melotti in +:issue:`2927`) + +:class:`~html.parser.HTMLParser` accepts a new keyword argument +*convert_charrefs* that, when ``True``, automatically converts all character +references. For backward-compatibility, its value defaults to ``False``, but +it will change to ``True`` in a future version of Python, so you are invited to +set it explicitly and update your code to use this new feature. (Contributed +by Ezio Melotti in :issue:`13633`) The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. (Contributed by Ezio Melotti in :issue:`15114`) @@ -785,6 +965,10 @@ :attr:`~http.server.HTTP.error_message_format` attribute and sent as the body of the error response. (Contributed by Karl Cow in :issue:`12921`.) +The :mod:`http.server` :ref:`command line interface ` now has +a ``-b/--bind`` option that causes the server to listen on a specific address. +(Contributed by Malte Swart in :issue:`17764`.) + importlib --------- @@ -823,11 +1007,16 @@ that decodes source from bytes using universal newline processing. This is useful for implementing :meth:`.InspectLoader.get_source` methods. +:class:`importlib.machinery.ExtensionFileLoader` now has a +:meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was +inadvertently omitted in the original implementation. (Contributed by Eric +Snow in :issue:`19152`.) + inspect ------- -The inspect module now offers a basic :ref:`command line interface +The :mod:`inspect` module now offers a basic :ref:`command line interface ` to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in :issue:`18626`) @@ -845,10 +1034,14 @@ :func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` now use the :func:`~inspect.signature` API. This allows them to -support much broader range of functions, including some builtins and -callables that follow ``__signature__`` protocol. It is still -recommended to update your code to use :func:`~inspect.signature` -directly. (Contributed by Yury Selivanov in :issue:`17481`) +support a much broader range of callables, including those with +``__signature__`` attributes, those with metadata provided by argument +clinic, :func:`functools.partial` objects and more. Note that, unlike +:func:`~inspect.signature`, these functions still ignore ``__wrapped__`` +attributes, and report the already bound first argument for bound methods, +so it is still necessary to update your code to use +:func:`~inspect.signature` directly if those features are desired. +(Contributed by Yury Selivanov in :issue:`17481`) :func:`~inspect.signature` now supports duck types of CPython functions, which adds support for functions compiled with Cython. (Contributed @@ -864,6 +1057,10 @@ by the normal standard library requirements to maintain backwards compatibility. +A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if +an address is globally routeable. (Contributed by Peter Moody in +:issue:`17400`.) + logging ------- @@ -872,6 +1069,24 @@ parameter that can be used to specify the time of day when rollover should happen. (Contributed by Ronald Oussoren in :issue:`9556`.) +:class:`~logging.handlers.SocketHandler` and +:class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by +setting *port* to ``None``). (Contributed by Vinay Sajip in commit +ce46195b56a9.) + +:func:`~logging.config.fileConfig` now accepts a +:class:`configparser.RawConfigParser` subclass instance for the *fname* +parameter. This facilitates using a configuration file when logging +configuration is just a part of the overall application configuration, or where +the application modifies the configuration before passing it to +:func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in +:issue:`16110`.) + +Logging configuration data received from a socket via the +:func:`logging.config.listen` function can now be validated before being +processed by supplying a verification function as the argument to the new +*verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) + .. _whatsnew-marshal-3: @@ -884,26 +1099,15 @@ this "one copy" ability to any object type (including handling recursive references). This reduces both the size of ``.pyc`` files and the amount of memory a module occupies in memory when it is loaded from a ``.pyc`` (or -``.pyo``) file. (Contributed by Kristj?n Valur J?nsson in :issue:`16475`.) +``.pyo``) file. (Contributed by Kristj?n Valur J?nsson in :issue:`16475`, +with additional speedups by Antoine Pitrou in :issue:`19219`.) mmap ---- -mmap objects can now be weakref'ed. -(Contributed by Valerie Lambert in :issue:`4885`.) - - -mock ----- - -:mod:`~unittest.mock` objects now inspect their specification signatures when -matching calls, which means an argument can now be matched by either position -or name, instead of only by position. (Contributed by Antoine Pitrou in -:issue:`17015`.) - -:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` -methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) +mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in +:issue:`4885`.) multiprocessing @@ -911,20 +1115,34 @@ .. _whatsnew-multiprocessing-no-fork: -On Unix, two new :ref:`start methods ` -(``spawn`` and ``forkserver``) have been added for starting processes using +On Unix two new :ref:`start methods `, +(``spawn`` and ``forkserver``, have been added for starting processes using :mod:`multiprocessing`. These make the mixing of processes with threads more robust, and the ``spawn`` method matches the semantics that multiprocessing has -always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). - -Also, except when using the old *fork* start method, child processes -will no longer inherit unneeded handles/file descriptors from their parents -(part of :issue:`8713`). +always used on Windows. New function +:func:`~multiprocessing.get_all_start_methods` reports all start methods +available on the platform, :func:`~multiprocessing.get_start_method` reports +the current start method, and :func:`~multiprocessing.set_start_method` sets +the start method. (Contributed by Richard Oudkerk in :issue:`8713`). + +:mod:`multiprocessing` also now has the concept of a ``context``, which +determines how child processes are created. New function +:func:`~multiprocessing.get_context` returns a context that uses a specified +start method. It has the same API as the :mod:`multiprocessing` module itself, +so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other +objects that will operate within that context. This allows a framework and an +application or different parts of the same application to use multiprocessing +without interfering with each other. (Contributed by Richard Oudkerk in +:issue:`18999`.) + +Except when using the old *fork* start method, child processes no longer +inherit unneeded handles/file descriptors from their parents (part of +:issue:`8713`). :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes when using the ``spawn`` or ``forkserver`` start methods. This resolves some -edge cases where combining multiprocessing, the ``-m`` command line switch +edge cases where combining multiprocessing, the ``-m`` command line switch, and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in :issue:`19946`) @@ -932,6 +1150,11 @@ operator -------- +New function :func:`~operator.length_hint` provides an implementation of the +specification for how the :meth:`~object.__length_hint__` special method should +be used, as part of the :pep:`424` formal specification of this language +feature. (Contributed by Armin Ronacher in :issue:`16148`.) + There is now a pure-python version of the :mod:`operator` module available for reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in :issue:`16694`.) @@ -940,32 +1163,48 @@ os -- -New functions to get and set the :ref:`inheritable flag ` of a file -descriptors or a Windows handle: - -* :func:`os.get_inheritable`, :func:`os.set_inheritable` -* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` - -The :mod:`os` module now provides a :func:`~os.cpu_count` function, analogous to -the :func:`multiprocessing.cpu_count` function (which is now implemented in -terms of the new :mod:`os` function). (Contributed by Trent Nelson, Yogesh -Chaudhari, Victor Stinner, and Charles-Fran?ois Natali in :issue:`17914`.) +There are new functions to get and set the :ref:`inheritable flag +` of a file descriptor (:func:`os.get_inheritable`, +:func:`os.set_inheritable`) or a Windows handle +(:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). + +New function :func:`~os.cpu_count` reports the number of CPUs available on the +platform on which Python is running (or ``None`` if the count can't be +determined). The :func:`multiprocessing.cpu_count` function is now implemented +in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, +Victor Stinner, and Charles-Fran?ois Natali in :issue:`17914`.) + +:func:`os.path.samestat` is now available on the Windows platform (and the +:func:`os.path.samefile` implementation is now shared between Unix and +Windows). (Contributed by Brian Curtin in :issue:`11939`.) + +:func:`os.path.ismount` now recognizes volumes mounted below a drive +root on Windows. (Contributed by Tim Golden in :issue:`9035`.) + +:func:`os.open` supports two new flags on platforms that provide them, +:data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` +(unnamed temporary file; as of 3.4.0 release available only on Linux systems +with a kernel version of 3.11 or newer that have uapi headers). (Contributed +by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) pdb --- +:mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and +``yield from`` in a more useful fashion. This is especially helpful when +debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and +Xavier de Gaye in :issue:`16596`.) + The ``print`` command has been removed from :mod:`pdb`, restoring access to the -``print`` function. - -Rationale: Python2's ``pdb`` did not have a ``print`` command; instead, -entering ``print`` executed the ``print`` statement. In Python3 ``print`` was -mistakenly made an alias for the pdb :pdbcmd:`p` command. ``p``, however, -prints the ``repr`` of its argument, not the ``str`` like the Python2 ``print`` -command did. Worse, the Python3 ``pdb print`` command shadowed the Python3 -``print`` function, making it inaccessible at the ``pdb`` prompt. - -(Contributed by Connor Osborn in :issue:`18764`.) +Python :func:`print` function from the pdb command line. Python2's ``pdb`` did +not have a ``print`` command; instead, entering ``print`` executed the +``print`` statement. In Python3 ``print`` was mistakenly made an alias for the +pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument, +not the ``str`` like the Python2 ``print`` command did. Worse, the Python3 +``pdb print`` command shadowed the Python3 ``print`` function, making it +inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in +:issue:`18764`.) .. _whatsnew-protocol-4: @@ -976,7 +1215,7 @@ :mod:`pickle` now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large -strings and containers, or classes whose :meth:`__new__` method takes +strings and containers, and classes whose :meth:`__new__` method takes keyword-only arguments. It also provides some efficiency improvements. .. seealso:: @@ -988,9 +1227,13 @@ plistlib -------- -:mod:`plistlib` now supports binary plist files, and offers the common -``load``/``loads``/``dump``/``dumps`` API pattern for serialization formats -(Contributed by Ronald Oussoren and others in :issue:`14455`). +:mod:`plistlib` now has an API that is similar to the standard pattern for +stdlib serialization protocols, with new :func:`~plistlib.load`, +:func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` +functions. (The older API is now deprecated.) In addition to the already +supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports +the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald +Oussoren and others in :issue:`14455`). poplib @@ -1006,11 +1249,15 @@ pprint ------ -The :mod:`pprint` module now supports *compact* mode for formatting long -sequences (:issue:`19132`). +The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its +:func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new +option, *compact*, that controls how the output is formatted. Currently +setting *compact* to ``True`` means that sequences will be printed with as many +sequence elements as will fit within *width* on each (indented) line. +(Contributed by Serhiy Storchaka in :issue:`19132`.) Long strings are now wrapped using Python's normal line continuation -syntax (Contributed by Antoine Pitrou in :issue:`17150`.) +syntax. (Contributed by Antoine Pitrou in :issue:`17150`). pty @@ -1023,30 +1270,61 @@ pydoc ----- -While significant changes have not been made to :mod:`pydoc` directly, +The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` +introspection API, allowing it to provide signature information for a wider +variety of callable objects. This change also means that ``__wrapped__`` +attributes are now taken into account when displaying help information +(Contributed by Larry Hastings in :issue:`19674`) + +The :mod:`pydoc` module no longer displays the ``self`` parameter for +already bound methods. Instead, it aims to always display the exact current +signature of the supplied callable (Contributed by Larry Hastings in +:issue:`20710`) + +In addition to the changes that have been made to :mod:`pydoc` directly, its handling of custom ``__dir__`` methods and various descriptor -behaviours has been improved substantially by the underlying changes in +behaviours has also been improved substantially by the underlying changes in the :mod:`inspect` module. +As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also +affect the behaviour of :func:`help`. + re -- -Added :func:`re.fullmatch` function and :meth:`regex.fullmatch` method, -which anchor the pattern at both ends of the string to match. -(Contributed by Matthew Barnett in :issue:`16203`.) +New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor +the pattern at both ends of the string to match. This provides a way to be +explicit about the goal of the match, which avoids a class of subtle bugs where +``$`` characters get lost during code changes or the addition of alternatives +to an existing regular expression. (Contributed by Matthew Barnett in +:issue:`16203`.) The repr of :ref:`regex objects ` now includes the pattern and the flags; the repr of :ref:`match objects ` now -includes the start, end, and the part of the string that matched. -(Contributed by Serhiy Storchaka in :issue:`13592` and :issue:`17087`.) +includes the start, end, and the part of the string that matched. (Contributed +by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and +:issue:`17087`.) resource -------- -New :func:`resource.prlimit` function and Linux specific constants. -(Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) +New :func:`~resource.prlimit` function, available on Linux platforms with a +kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the +ability to query or set the resource limits for processes other than the one +making the call. (Contributed by Christian Heimes in :issue:`16595`.) + +On Linux kernel version 2.6.36 or later, there are there are also some new +Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, +:attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, +:attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. +(Contributed by Christian Heimes in :issue:`19324`.) + +On FreeBSD version 9 and later, there some new FreeBSD specific constants: +:attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and +:attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in +:issue:`19343`.) select @@ -1057,6 +1335,11 @@ method will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in :issue:`16488`.) +:class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and +:meth:`~select.devpoll.close` methods, as well as a new attribute +:attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in +:issue:`18794`.) + shelve ------ @@ -1066,11 +1349,21 @@ (Contributed by Filip Gruszczy?ski in :issue:`13896`.) +shutil +------ + +:func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, +:exc:`~shutil.SameFileError`, when the source and destination are the same +file, which allows an application to take appropriate action on this specific +error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in +:issue:`1492704`.) + + smtpd ----- The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now -accept a *map* keyword argument, which if specified is passed in to +accept a *map* keyword argument which, if specified, is passed in to :class:`asynchat.async_chat` as its *map* argument. This allows an application to avoid affecting the global socket map. (Contributed by Vinay Sajip in :issue:`11959`.) @@ -1082,7 +1375,7 @@ :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. -(:issue:`2118`). +(Contributed by Ned Jackson Lovely in :issue:`2118`). socket @@ -1101,14 +1394,17 @@ The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. +:func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported +on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) + sqlite3 ------- -A new boolean parameter, *uri*, to the :func:`~sqlite3.connect` function can -be used to indicate that the *database* parameter is a ``uri`` (see -the `SQLite URI documentation `_). -(Contributed by poq in :issue:`13773`.) +A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be +used to indicate that the *database* parameter is a ``uri`` (see the `SQLite +URI documentation `_). (Contributed by poq in +:issue:`13773`.) ssl @@ -1121,6 +1417,25 @@ Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orr? and Antoine Pitrou in :issue:`16692`) +.. _whatsnew34-sslcontext: + +New function :func:`~ssl.create_default_context` provides a standard way to +obtain an :class:`~ssl.SSLContext` whose settings are intended to be a +reasonable balance between compatibility and security. These settings are +more stringent than the defaults provided by the :class:`~ssl.SSLContext` +constructor, and may be adjusted in the future, without prior deprecation, if +best-practice security requirements change. The new recommended best +practice for using stdlib libraries that support SSL is to use +:func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` +object, modify it if needed, and then pass it as the *context* argument +of the appropriate stdlib API. (Contributed by Christian Heimes +in :issue:`19689`.) + +:class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` +accepts a new optional argument *cadata*, which can be used to provide PEM or +DER encoded certificates directly via strings or bytes, respectively. +(Contributed by Christian Heimes in :issue:`18138`.) + New function :func:`~ssl.get_default_verify_paths` returns a named tuple of the paths and environment variables that the :meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set @@ -1133,36 +1448,62 @@ ``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists (``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that returns a list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in -and :issue:`18147`.) +:issue:`18147`.) + +If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new +attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the +certificate verification process by setting it to some combination of the new +constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, +:data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. +OpenSSL does not do any CRL verification by default. (Contributed by +Christien Heimes in :issue:`8813`.) + +New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` +loads a set of dfault "certificate authority" (CA) certificates from default +locations, which vary according to the platform. It can be used to load both +TLS web server authentication certificates +(``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a +server, and certificates for a server to use in verifying client certificates +(``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian +Heimes in :issue:`19292`.) + +.. _whatsnew34-win-cert-store: Two new windows-only functions, :func:`~ssl.enum_certificates` and :func:`~ssl.enum_crls` provide the ability to retrieve certificates, certificate information, and CRLs from the Windows cert store. (Contributed by Christian Heimes in :issue:`17134`.) -Support for server-side SNI using the new +.. _whatsnew34-sni: + +Support for server-side SNI (Server Name Indication) using the new :meth:`ssl.SSLContext.set_servername_callback` method. (Contributed by Daniel Black in :issue:`8109`.) +The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional +``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and +``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) + stat ---- The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C implementation is required as most of the values aren't standardized and -platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) - -The module supports new file types: door, event port and whiteout. +are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) + +The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, +:attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by +Christian Hiemes in :issue:`11016`.) struct ------ -:mod:`struct` now supports the streamed unpacking of a buffer containing -repeated instances of a given format of data. Both a module level -:mod:`~struct.iter_unpack` function and a :meth:`struct.Struct.iter_unpack` -method on compiled formats have been added. (Contributed by Antoine Pitrou in -:issue:`17804`.) +New function :mod:`~struct.iter_unpack` and a new +:meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed +unpacking of a buffer containing repeated instances of a given format of data. +(Contributed by Antoine Pitrou in :issue:`17804`.) subprocess @@ -1172,6 +1513,10 @@ be used to provide the contents of ``stdin`` for the command that is run. (Contributed by Zack Weinberg in :issue:`16624`.) +:func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now +work on Windows. This change was actually inadvertently made in 3.3.4. +(Contributed by Tim Golden in :issue:`10197`.) + sunau ----- @@ -1179,26 +1524,93 @@ The :meth:`~sunau.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) -:meth:`sunau.open` now supports the context manager protocol (:issue:`18878`). +:meth:`sunau.open` now supports the context manager protocol: when used in a +:keyword:`with` block, the ``close`` method of the returned object will be +called automatically at the end of the block. (Contributed by Serhiy Storchaka +in :issue:`18878`.) + +:meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding +support for writing 24 sample using the module. (Contributed by +Serhiy Storchaka in :issue:`19261`.) + +The :meth:`~sunau.AU_write.writeframesraw` and +:meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) sys --- New function :func:`sys.getallocatedblocks` returns the current number of -blocks allocated by the interpreter (in CPython with the default +blocks allocated by the interpreter. (In CPython with the default ``--with-pymalloc`` setting, this is allocations made through the -:c:func:`PyObject_Malloc` API). This can be useful for tracking memory leaks, +:c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in :issue:`13390`.) +When the Python interpreter starts in :ref:`interactive mode +`, it checks for an :data:`~sys.__interactivehook__` attribute +on the :mod:`sys` module. If the attribute exists, its value is called with no +arguments just before interactive mode is started. The check is made after the +:envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` +module :ref:`sets it ` to a function that enables tab +completion and history saving (in :file:`~/.python-history`) if the platform +supports :mod:`readline`. If you do not want this (new) behavior, you can +override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or +:mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it +to some other callable). (Contributed by ?ric Araujo and Antoine Pitrou in +:issue:`5845`.) + + +tarfile +------- + +The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when +called as a script directly or via ``-m``. This can be used to create and +extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) + + +textwrap +-------- + +The :class:`~textwrap.TextWrapper` class has two new attributes/constructor +arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of +lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a +string that will appear at the end of the output if it has been truncated +because of *max_lines*. Building on these capabilities, a new convenience +function :func:`~textwrap.shorten` collapses all of the whitespace in the input +to single spaces and produces a single line of a given *width* that ends with +the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and +Serhiy Storchaka in :issue:`18585` and :issue:`18725`.) + + +threading +--------- + +The :class:`~threading.Thread` object representing the main thread can be +obtained from the new :func:`~threading.main_thread` function. In normal +conditions this will be the thread from which the Python interpreter was +started. (Contributed by Andrew Svetlov in :issue:`18882`.) + traceback --------- A new :func:`traceback.clear_frames` function takes a traceback object and clears the local variables in all of the frames it references, -reducing the amount of memory consumed (:issue:`1565525`). +reducing the amount of memory consumed. (Contributed by Andrew Kuchling in +:issue:`1565525`). + + +types +----- + +A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define +an attribute that acts normally when looked up through an instance object, but +which is routed to the *class* ``__getattr__`` when looked up through the +class. This allows one to have properties active on a class, and have virtual +attributes on the class with the same name (see :mod:`Enum` for an example). +(Contributed by Ethan Furman in :issue:`19030`.) urllib @@ -1208,6 +1620,29 @@ :class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenb?ck in :issue:`16423`.) +The http method that will be used by a :class:`~urllib.request.Request` class +can now be specified by setting a :class:`~urllib.request.Request.method` +class attribute on the subclass. (Contributed by Jason R Coombs in +:issue:`18978`.) + +:class:`~urllib.request.Request` objects are now reusable: if the +:attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` +attributes are modified, all relevant internal properties are updated. This +means, for example, that it is now possible to use the same +:class:`~urllib.request.Request` object in more than one +:meth:`.OpenerDirector.open` call with different *data* arguments, or to +modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it +from scratch. There is also a new +:meth:`~urllib.request.Request.remove_header` method that can be used to remove +headers from a :class:`~urllib.request.Request`. (Contributed by Alexey +Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht +and Senthil Kumaran in :issue:`17272`.) + +:class:`~urllib.error.HTTPError` objects now have a +:attr:`~urllib.error.HTTPError.headers` attribute that provides access to the +HTTP response headers associated with the error. (Contributed by +Berker Peksag in :issue:`15701`.) + unittest -------- @@ -1223,7 +1658,7 @@ class NumbersTest(unittest.TestCase): def test_even(self): for i in range(6): - with self.subTest(i=1): + with self.subTest(i=i): self.assertEqual(i % 2, 0) will result in six subtests, each identified in the unittest verbose output @@ -1243,6 +1678,34 @@ consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in :issue:`16709`.) +:class:`~unittest.TestSuite` now drops references to tests as soon as the test +has been run, if the test is successful. On Python interpreters that do +garbage collection, this allows the tests to be garbage collected if nothing +else is holding a reference to the test. It is possible to override this +behavior by creating a :class:`~unittest.TestSuite` subclass that defines a +custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt +McClure, and Andrew Svetlov in :issue:`11798`.) + +A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`, +will ensure that a given block of code emits a log message using the +:mod:`logging` module. By default the message can come from any logger and +have a priority of ``INFO`` or higher, but both the logger name and an +alternative minimum logging level may be specified. The object returned by the +context manager can be queried for the :class:`~logging.LogRecord`\ s and/or +formatted messages that were logged. (Contributed by Antoine Pitrou in +:issue:`18937`.) + +Test discovery now works with namespace packages (Contributed by Claudiu Popa +in :issue:`17457`.) + +:mod:`unittest.mock` objects now inspect their specification signatures when +matching calls, which means an argument can now be matched by either position +or name, instead of only by position. (Contributed by Antoine Pitrou in +:issue:`17015`.) + +:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` +methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) + venv ---- @@ -1250,6 +1713,12 @@ :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` shells (Contributed by Andrew Svetlov in :issue:`15417`.) +:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function +take a new keyword argument *with_pip*, which defaults to ``False``, that +controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is +installed in the virtual environment. (Contributed by Nick Coghlan in +:issue:`19552` as part of the :pep:`453` implementation.) + wave ---- @@ -1260,6 +1729,14 @@ :meth:`wave.open` now supports the context manager protocol. (Contributed by Claudiu Popa in :issue:`17616`.) +:mod:`wave` can now :ref:`write output to unseekable files +`. (Contributed by David Jones, Guilherme Polo, and Serhiy +Storchaka in :issue:`5202`.) + +The :meth:`~wave.Wave_write.writeframesraw` and +:meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) + weakref ------- @@ -1295,13 +1772,19 @@ Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) -zipfile.PyZipfile ------------------ - -Add a filter function to ignore some packages (tests for instance), -:meth:`~zipfile.PyZipFile.writepy`. +zipfile +------- + +The :meth:`~zipfile.PyZipFile.writepy` method of the +:class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be +used to control which directories and files are added to the archive. For +example, this could be used to exclude test files from the archive. (Contributed by Christian Tismer in :issue:`19274`.) +The *allowZip64* parameter to :class:`~zipfile.ZipFile` and +:class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by +William Mallard in :issue:`17201`.) + CPython Implementation Changes @@ -1375,7 +1858,12 @@ implemented in C. Some standard library extension modules have been converted to use Argument -Clinic in Python 3.4, and :mod:`inspect` has been updated accordingly. +Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated +accordingly. + +It is expected that signature metadata for programmatic introspection will +be added to additional callables implemented in C as part of Python 3.4 +maintenance releases. .. note:: The Argument Clinic PEP is not fully up to date with the state of the @@ -1405,15 +1893,53 @@ marked as accepting ``const char *`` rather than ``char *`` (Contributed by Serhiy Storchaka in :issue:`1772673`). -* New shell version of ``python-config``; can be used even when a python +* A new shell version of ``python-config`` can be used even when a python interpreter is not available (for example, in cross compilation scenarios). - +* :c:func:`PyUnicode_FromFormat` now supports width and precision + specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``. + (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.) + +* New function :c:func:`PyStructSequence_InitType2` supplements the + existing :c:func:`PyStructSequence_InitType` function. The difference + is that it returns ``0`` on success and ``-1`` on failure. + +* The CPython source can now be compiled using the address sanity checking + features of recent versions of GCC and clang: the false alarms in the small + object allocator have been silenced. (Contributed by Dhiru Kholia in + :issue:`18596`.) + +* The Windows build now uses `Address Space Layout Randomization + `_ and `Data Execution Prevention + `_. (Contributed by + Christian Heimes in :issue:`16632`.) + +* New function :c:func:`PyObject_LengthHint` is the C API equivalent + of :func:`operator.length_hint`. (Contributed by Armin Ronacher in + :issue:`16148`.) + + +.. _other-improvements-3.4: Other Improvements -================== - -* Tab-completion is now enabled by default in the interactive interpreter. +------------------ + +.. _whatsnew-isolated-mode: + +* The :ref:`python ` command has a new :ref:`option + `, ``-I``, which causes it to run in "isolated mode", + which means that :data:`sys.path` contains neither the script's directory nor + the user's ``site-packages`` directory, and all :envvar:`PYTHON*` environment + variables are ignored (it implies both ``-s`` and ``-E``). Other + restrictions may also be applied in the future, with the goal being to + isolate the execution of a script from the user's environment. This is + appropriate, for example, when Python is used to run a system script. On + most POSIX systems it can and should be used in the ``#!`` line of system + scripts. (Contributed by Christian Heimes in :issue:`16499`.) + +* Tab-completion is now enabled by default in the interactive interpreter + on systems that support :mod:`readline`. History is also enabled by default, + and is written to (and read from) the file :file:`~/.python-history`. (Contributed by Antoine Pitrou and ?ric Araujo in :issue:`5845`.) * Invoking the Python interpreter with ``--version`` now outputs the version to @@ -1443,10 +1969,40 @@ values for its constants from the C header files, instead of having the values hard-coded in the python module as was previously the case. +* Loading multiple python modules from a single OS module (``.so``, ``.dll``) + now works correctly (previously it silently returned the first python + module in the file). (Contributed by V?clav ?milauer in :issue:`16421`.) + +* A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the + loading of free variables in class bodies that could be triggered by certain + uses of :ref:`__prepare__ `. (Contributed by Benjamin Peterson in + :issue:`17853`.) + +* A number of MemoryError-related crashes were identified and fixed by Victor + Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, + :issue:`18520`). + +* The :ref:`pyvenv ` command now accepts a ``--copies`` option + to use copies rather than symlinks even on systems where symlinks are the + default. (Contributed by Vinay Sajip in :issue:`18807`.) + +* The :ref:`pyvenv ` command also accepts a ``--without-pip`` + option to suppress the otherwise-automatic bootstrapping of pip into + the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` + as part of the :pep:`453` implementation.) + +* The encoding name is now optional in the value set for the + :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to + set just the error handler, without changing the default encoding. + (Contributed by Serhiy Storchaka in :issue:`18818`.) + +* The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now + support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and + Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) Significant Optimizations -========================= +------------------------- * The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka in :issue:`14625`.) @@ -1489,6 +2045,16 @@ * :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in :issue:`18020`.) +* On Windows, the native ``VirtualAlloc`` is now used instead of the CRT + ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory + savings. + +* :func:`os.urandom` now uses a lazily-opened persistent file descriptor + so as to avoid using many file descriptors when run in parallel from + multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) + + +.. _deprecated-3.4: Deprecated ========== @@ -1500,8 +2066,8 @@ using ``-Wd``). -Deprecated Python Modules, Functions and Methods ------------------------------------------------- +Deprecations in the Python API +------------------------------ * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` methods and functions are deprecated: :meth:`importlib.find_loader` is @@ -1534,18 +2100,39 @@ * The :mod:`formatter` module is pending deprecation and is slated for removal in Python 3.6. -* MD5 as default digestmod for :mod:`hmac` is deprecated. Python 3.6 will - require an explicit digest name or constructor as *digestmod* argument. +* ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is + deprecated. Python 3.6 will require an explicit digest name or constructor as + *digestmod* argument. * The internal ``Netrc`` class in the :mod:`ftplib` module has been documented as deprecated in its docstring for quite some time. It now emits a :exc:`DeprecationWarning` and will be removed completely in Python 3.5. - -Deprecated Functions and Types in the C API -------------------------------------------- - -XXX: None so far +* The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should + not have been exposed and is hopefully not in use; it is deprecated and + will mostly likely be removed in Python 3.5. + +* The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. + +* The :mod:`plistlib` :func:`~plistlib.readPlist`, + :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and + :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the + corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, + :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` + is deprecated in favor of just using the :class:`bytes` constructor. + +* The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by + ``EXT_SUFFIX``. + +* The ``U`` mode accepted by various ``open`` functions is deprecated. + In Python3 it does not do anything useful, and should be replaced by + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* + argument. + +* The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has + been deprecated, as has the *html* argument of + :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the + latter, all arguments to ``XMLParser`` should be passed by keyword. Deprecated Features @@ -1571,6 +2158,7 @@ * OS/2 (:issue:`16135`). * Windows 2000 (changeset e52df05b496a). +* Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`). * VMS (:issue:`16136`). @@ -1588,7 +2176,8 @@ ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). * The ``PyThreadState.tick_counter`` field has been removed; its value has - been meaningless since Python 3.2, when the "new GIL" was introduced. + been meaningless since Python 3.2, when the "new GIL" was introduced + (:issue:`19199`). * ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. (Contributed by Taras Lyapun in :issue:`15641`.) @@ -1608,6 +2197,15 @@ * :class:`inspect.Signature`: positional-only parameters are now required to have a valid name. +* :meth:`object.__format__` no longer accepts non-empty format strings, it now + raises a :exc:`TypeError` instead. Using a non-empty string has been + deprecated since Python 3.2. This change has been made to prevent a + situation where previously working (but incorrect) code would start failing + if an object gained a __format__ method, which means that your code may now + raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects + that do not have a __format__ method that handles it. See :issue:`7994` for + background. + * :meth:`difflib.SequenceMatcher.isbjunk` and :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have now been removed: use ``x in sm.bjunk`` and @@ -1626,6 +2224,9 @@ ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called on badly broken OSX systems (see :issue:`18393`). +* The hardcoded copies of certain :mod:`stat` constants that were included in + the :mod:`tarfile` module namespace have been removed. + Porting to Python 3.4 @@ -1634,6 +2235,27 @@ This section lists previously described changes and other bugfixes that may require changes to your code. + +Changes in 'python' Command Behavior +------------------------------------ + +* In a posix shell, setting the :envvar:`PATH` environment variable to + an empty value is equivalent to not setting it at all. However, setting + :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it + at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to + setting it to ``.``, which leads to confusion when reasoning by analogy to + how :envvar:`PATH` works. The behavior now conforms to the posix convention + for :envvar:`PATH`. + +* The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the + CPython interpreter is now off by default. It can be re-enabled using the + ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.) + +* The python command and most stdlib scripts (as well as :mod:`argparse`) now + output ``--version`` information to ``stdout`` instead of ``stderr`` (for + issue list see :ref:`other-improvements-3.4` above). + + Changes in the Python API ------------------------- @@ -1646,7 +2268,7 @@ * The module type now initializes the :attr:`__package__` and :attr:`__loader__` attributes to ``None`` by default. To determine if these attributes were set in a backwards-compatible fashion, use e.g. - ``getattr(module, '__loader__', None) is not None``. + ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.) * :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and ``__package__`` unconditionally to properly support reloading. If this is not @@ -1655,10 +2277,23 @@ * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. + Note that this restores a pre-3.3 behavior in that it means a module is + re-found when re-loaded (:issue:`19413`). * Frozen packages no longer set ``__path__`` to a list containing the package - name but an empty list instead. Determing if a module is a package should be - done using ``hasattr(module, '__path__')``. + name, they now set it to an empty list. The previous behavior could cause + the import system to do the wrong thing on submodule imports if there was + also a directory with the same name as the frozen package. The correct way + to determine if a module is a package or not is to use``hasattr(module, + '__path__')`` (:issue:`18065`). + +* Frozen modules no longer define a ``__file__`` attribute. It's semantically + incorrect for frozen modules to set the attribute as they are not loaded from + any explicit location. If you must know that a module comes from frozen code + then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, + check if the loader is a subclass of + :class:`importlib.machinery.FrozenImporter`, + or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path it would write to is a symlink or a non-regular file. This is to act as a @@ -1684,14 +2319,24 @@ :func:`inspect.unwrap` to access the first function in the chain that has no ``__wrapped__`` attribute. +* :func:`inspect.getfullargspec` has been reimplemented on top of + :func:`inspect.signature` and hence handles a much wider variety of callable + objects than it did in the past. It is expected that additional builtin and + extension module callables will gain signature metadata over the course of + the Python 3.4 series. Code that assumes that + :func:`inspect.getfullargspec` will fail on non-Python callables may need + to be adjusted accordingly. + * :class:`importlib.machinery.PathFinder` now passes on the current working directory to objects in :data:`sys.path_hooks` for the empty string. This results in :data:`sys.path_importer_cache` never containing ``''``, thus iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` will not find all keys. A module's ``__file__`` when imported in the current working directory will also now have an absolute path, including when using - ``-m`` with the interpreter (this does not influence when the path to a file - is specified on the command-line). + ``-m`` with the interpreter (except for ``__main__.__file__`` when a script + has been executed directly using a relative path) (Contributed by Brett + Cannon in :issue:`18416`). is specified on the command-line) + (:issue:`18416`). * The removal of the *strict* argument to :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` changes the meaning of the @@ -1729,10 +2374,106 @@ informative :exc:`ValueError` rather than the previous more mysterious :exc:`AttributeError` (:issue:`9177`). -* Parameter names in ``__annotations__`` dict are now mangled properly, +* :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge + values. As a consequence of this fix, :meth:`slice.indices` now raises a + :exc:`ValueError` if given a negative length; previously it returned nonsense + values (:issue:`14794`). + +* The :class:`complex` constructor, unlike the :mod:`cmath` functions, was + incorrectly accepting :class:`float` values if an object's ``__complex__`` + special method returned one. This now raises a :exc:`TypeError`. + (:issue:`16290`.) + +* The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float` + values for the *base* parameter. It is unlikely anyone was doing this, but + if so, it will now raise a :exc:`TypeError` (:issue:`16772`). + +* Defaults for keyword-only arguments are now evaluated *after* defaults for + regular keyword arguments, instead of before. Hopefully no one wrote any + code that depends on the previous buggy behavior (:issue:`16967`). + +* Stale thread states are now cleared after :func:`~os.fork`. This may cause + some system resources to be released that previously were incorrectly kept + perpetually alive (for example, database connections kept in thread-local + storage). (:issue:`17094`.) + +* Parameter names in ``__annotations__`` dicts are now mangled properly, similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in :issue:`20625`). +* :attr:`hashlib.hash.name` now always returns the identifier in lower case. + Previously some builtin hashes had uppercase names, but now that it is a + formal public interface the naming has been made consistent (:issue:`18532`). + +* Because :mod:`unittest.TestSuite` now drops references to tests after they + are run, test harnesses that re-use a :class:`~unittest.TestSuite` to re-run + a set of tests may fail. Test suites should not be re-used in this fashion + since it means state is retained between test runs, breaking the test + isolation that :mod:`unittest` is designed to provide. However, if the lack + of isolation is considered acceptable, the old behavior can be restored by + creating a :mod:`~unittest.TestSuite` subclass that defines a + ``_removeTestAtIndex`` method that does nothing (see + :meth:`.TestSuite.__iter__`) (:issue:`11798`). + +* :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are + certain invalid command forms that used to work that are no longer allowed; + in theory this should not cause backward compatibility issues since the + disallowed command forms didn't make any sense and are unlikely to be in use. + +* The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and + the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of + ``match`` objects now always return a *bytes* object when the string + to be matched is a :term:`bytes-like object`. Previously the return type + matched the input type, so if your code was depending on the return value + being, say, a ``bytearray``, you will need to change your code. + +* :mod:`audioop` functions now raise an error immediately if passed string + input, instead of failing randomly later on (:issue:`16685`). + +* The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` + currently defaults to ``False`` for backward compatibility, but will + eventually be changed to default to ``True``. It is recommended that you add + this keyword, with the appropriate value, to any + :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). + +* Since the *digestmod* argument to the :func:`hmac.new` function will in the + future have no default, all calls to :func:`hmac.new` should be changed to + explicitly specify a *digestmod* (:issue:`17276`). + +* Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking + ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` + is deprecated. This key should be replaced by ``EXT_SUFFIX`` or + ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). + +* Any calls to ``open`` functions that specify ``U`` should be modified. + ``U`` is ineffective in Python3 and will eventually raise an error if used. + Depending on the function, the equivalent of its old Python2 behavior can be + achieved using either a *newline* argument, or if necessary by wrapping the + stream in :mod:`~io.TextIOWrapper` to use its *newline* argument + (:issue:`15204`). + +* If you use :ref:`pyvenv ` in a script and desire that pip + *not* be installed, you must add ``--without-pip`` to your command + invocation. + +* The default behavior of :func:`json.dump` and :func:`json.dumps` when + an indent is specified has changed: it no longer produces trailing + spaces after the item separating commas at the ends of lines. This + will matter only if you have tests that are doing white-space-sensitive + comparisons of such output (:issue:`16333`). + +* :mod:`doctest` now looks for doctests in extension module ``__doc__`` + strings, so if your doctest test discovery includes extension modules that + have things that look like doctests in them you may see test failures you've + never seen before when running your tests (:issue:`3158`). + +* The :mod:`collections.abc` module has been slightly refactored as + part of the Python startup improvements. As a consequence of this, it is no + longer the case that importing :mod:`collections` automatically imports + :mod:`collections.abc`. If your program depended on the (undocumented) + implicit import, you will need to add an explicit ``import collections.abc`` + (:issue:`20784`). + Changes in the C API -------------------- @@ -1756,7 +2497,8 @@ * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must now be a string allocated by :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a - string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. + string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` + (:issue:`16742`) * :c:func:`PyThread_set_key_value` now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -7,15 +7,17 @@ Release date: 2014-03-16 -Core and Builtins ------------------ - Library ------- - Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. +Documentation +------------- + +- Merge in all documentation changes since branching 3.4.0rc1. + What's New in Python 3.4.0 release candidate 3? =============================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:38 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Regenerate_pyd?= =?utf-8?q?oc=5Ftopics=2C_fix_markup_errors=2C_in_preparation_for_3=2E4=2E?= =?utf-8?q?0_final=2E?= Message-ID: <3fnSQB5qWLz7LkL@mail.python.org> http://hg.python.org/cpython/rev/28a92fd64ba9 changeset: 89798:28a92fd64ba9 branch: 3.4 user: Larry Hastings date: Sat Mar 15 22:29:19 2014 -0700 summary: Regenerate pydoc_topics, fix markup errors, in preparation for 3.4.0 final. files: Doc/tools/sphinxext/susp-ignored.csv | 1 - Doc/whatsnew/3.4.rst | 2 +- Lib/pydoc_data/topics.py | 8 ++++---- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -282,4 +282,3 @@ whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" whatsnew/changelog,,:password,user:password -whatsnew/changelog,,:gz,w:gz diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2284,7 +2284,7 @@ name, they now set it to an empty list. The previous behavior could cause the import system to do the wrong thing on submodule imports if there was also a directory with the same name as the frozen package. The correct way - to determine if a module is a package or not is to use``hasattr(module, + to determine if a module is a package or not is to use ``hasattr(module, '__path__')`` (:issue:`18065`). * Frozen modules no longer define a ``__file__`` attribute. It's semantically diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Sun Mar 9 03:59:56 2014 +# Autogenerated by Sphinx on Sat Mar 15 22:21:45 2014 topics = {'assert': '\nThe "assert" statement\n**********************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, "assert expression", is equivalent to\n\n if __debug__:\n if not expression: raise AssertionError\n\nThe extended form, "assert expression1, expression2", is equivalent to\n\n if __debug__:\n if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that "__debug__" and "AssertionError" refer\nto the built-in variables with those names. In the current\nimplementation, the built-in variable "__debug__" is "True" under\nnormal circumstances, "False" when optimization is requested (command\nline option -O). The current code generator emits no code for an\nassert statement when optimization is requested at compile time. Note\nthat it is unnecessary to include the source code for the expression\nthat failed in the error message; it will be displayed as part of the\nstack trace.\n\nAssignments to "__debug__" are illegal. The value for the built-in\nvariable is determined when the interpreter starts.\n', 'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n target_list ::= target ("," target)* [","]\n target ::= identifier\n | "(" target_list ")"\n | "[" target_list "]"\n | attributeref\n | subscription\n | slicing\n | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable. The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n that target.\n\n* If the target list is a comma-separated list of targets: The\n object must be an iterable with the same number of items as there\n are targets in the target list, and the items are assigned, from\n left to right, to the corresponding targets.\n\n * If the target list contains one target prefixed with an\n asterisk, called a "starred" target: The object must be a sequence\n with at least as many items as there are targets in the target\n list, minus one. The first items of the sequence are assigned,\n from left to right, to the targets before the starred target. The\n final items of the sequence are assigned to the targets after the\n starred target. A list of the remaining items in the sequence is\n then assigned to the starred target (the list can be empty).\n\n * Else: The object must be a sequence with the same number of\n items as there are targets in the target list, and the items are\n assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n * If the name does not occur in a "global" or "nonlocal" statement\n in the current code block: the name is bound to the object in the\n current local namespace.\n\n * Otherwise: the name is bound to the object in the global\n namespace or the outer namespace determined by "nonlocal",\n respectively.\n\n The name is rebound if it was already bound. This may cause the\n reference count for the object previously bound to the name to reach\n zero, causing the object to be deallocated and its destructor (if it\n has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in\n square brackets: The object must be an iterable with the same number\n of items as there are targets in the target list, and its items are\n assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n the reference is evaluated. It should yield an object with\n assignable attributes; if this is not the case, "TypeError" is\n raised. That object is then asked to assign the assigned object to\n the given attribute; if it cannot perform the assignment, it raises\n an exception (usually but not necessarily "AttributeError").\n\n Note: If the object is a class instance and the attribute reference\n occurs on both sides of the assignment operator, the RHS expression,\n "a.x" can access either an instance attribute or (if no instance\n attribute exists) a class attribute. The LHS target "a.x" is always\n set as an instance attribute, creating it if necessary. Thus, the\n two occurrences of "a.x" do not necessarily refer to the same\n attribute: if the RHS expression refers to a class attribute, the\n LHS creates a new instance attribute as the target of the\n assignment:\n\n class Cls:\n x = 3 # class variable\n inst = Cls()\n inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3\n\n This description does not necessarily apply to descriptor\n attributes, such as properties created with "property()".\n\n* If the target is a subscription: The primary expression in the\n reference is evaluated. It should yield either a mutable sequence\n object (such as a list) or a mapping object (such as a dictionary).\n Next, the subscript expression is evaluated.\n\n If the primary is a mutable sequence object (such as a list), the\n subscript must yield an integer. If it is negative, the sequence\'s\n length is added to it. The resulting value must be a nonnegative\n integer less than the sequence\'s length, and the sequence is asked\n to assign the assigned object to its item with that index. If the\n index is out of range, "IndexError" is raised (assignment to a\n subscripted sequence cannot add new items to a list).\n\n If the primary is a mapping object (such as a dictionary), the\n subscript must have a type compatible with the mapping\'s key type,\n and the mapping is then asked to create a key/datum pair which maps\n the subscript to the assigned object. This can either replace an\n existing key/value pair with the same key value, or insert a new\n key/value pair (if no key with the same value existed).\n\n For user-defined objects, the "__setitem__()" method is called with\n appropriate arguments.\n\n* If the target is a slicing: The primary expression in the\n reference is evaluated. It should yield a mutable sequence object\n (such as a list). The assigned object should be a sequence object\n of the same type. Next, the lower and upper bound expressions are\n evaluated, insofar they are present; defaults are zero and the\n sequence\'s length. The bounds should evaluate to integers. If\n either bound is negative, the sequence\'s length is added to it. The\n resulting bounds are clipped to lie between zero and the sequence\'s\n length, inclusive. Finally, the sequence object is asked to replace\n the slice with the items of the assigned sequence. The length of\n the slice may be different from the length of the assigned sequence,\n thus changing the length of the target sequence, if the object\n allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample "a, b = b, a" swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe! For instance, the\nfollowing program prints "[0, 2]":\n\n x = [0, 1]\n i = 0\n i, x[i] = 1, 2\n print(x)\n\nSee also: **PEP 3132** - Extended Iterable Unpacking\n\n The specification for the "*target" feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n augtarget ::= identifier | attributeref | subscription | slicing\n augop ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like "x += 1" can be rewritten as\n"x = x + 1" to achieve a similar, but not exactly equal effect. In the\naugmented version, "x" is only evaluated once. Also, when possible,\nthe actual operation is performed *in-place*, meaning that rather than\ncreating a new object and assigning that to the target, the old object\nis modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n', 'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name. See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a "NameError" exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them. The transformation inserts the\nclass name, with leading underscores removed and a single underscore\ninserted, in front of the name. For example, the identifier "__spam"\noccurring in a class named "Ham" will be transformed to "_Ham__spam".\nThis transformation is independent of the syntactical context in which\nthe identifier is used. If the transformed name is extremely long\n(longer than 255 characters), implementation defined truncation may\nhappen. If the class name consists only of underscores, no\ntransformation is done.\n', @@ -23,7 +23,7 @@ 'context-managers': '\nWith Statement Context Managers\n*******************************\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a "with" statement. The context manager\nhandles the entry into, and the exit from, the desired runtime context\nfor the execution of the block of code. Context managers are normally\ninvoked using the "with" statement (described in section *The with\nstatement*), but can also be used by directly invoking their methods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The "with"\n statement will bind this method\'s return value to the target(s)\n specified in the "as" clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be "None".\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that "__exit__()" methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also: **PEP 0343** - The "with" statement\n\n The specification, background, and examples for the Python "with"\n statement.\n', 'continue': '\nThe "continue" statement\n************************\n\n continue_stmt ::= "continue"\n\n"continue" may only occur syntactically nested in a "for" or "while"\nloop, but not nested in a function or class definition or "finally"\nclause within that loop. It continues with the next cycle of the\nnearest enclosing loop.\n\nWhen "continue" passes control out of a "try" statement with a\n"finally" clause, that "finally" clause is executed before really\nstarting the next loop cycle.\n', 'conversions': '\nArithmetic conversions\n**********************\n\nWhen a description of an arithmetic operator below uses the phrase\n"the numeric arguments are converted to a common type," this means\nthat the operator implementation for built-in types works that way:\n\n* If either argument is a complex number, the other is converted to\n complex;\n\n* otherwise, if either argument is a floating point number, the\n other is converted to floating point;\n\n* otherwise, both must be integers and no conversion is necessary.\n\nSome additional rules apply for certain operators (e.g., a string left\nargument to the \'%\' operator). Extensions must define their own\nconversion behavior.\n', - 'customization': '\nBasic customization\n*******************\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. "__new__()" is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of "__new__()" should be the new object instance (usually an\n instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s "__new__()" method using\n "super(currentclass, cls).__new__(cls[, ...])" with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If "__new__()" returns an instance of *cls*, then the new\n instance\'s "__init__()" method will be invoked like\n "__init__(self[, ...])", where *self* is the new instance and the\n remaining arguments are the same as were passed to "__new__()".\n\n If "__new__()" does not return an instance of *cls*, then the new\n instance\'s "__init__()" method will not be invoked.\n\n "__new__()" is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n "__init__()" method, the derived class\'s "__init__()" method, if\n any, must explicitly call it to ensure proper initialization of the\n base class part of the instance; for example:\n "BaseClass.__init__(self, [args...])". As a special constraint on\n constructors, no value may be returned; doing so will cause a\n "TypeError" to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a "__del__()" method, the\n derived class\'s "__del__()" method, if any, must explicitly call it\n to ensure proper deletion of the base class part of the instance.\n Note that it is possible (though not recommended!) for the\n "__del__()" method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n "__del__()" methods are called for objects that still exist when\n the interpreter exits.\n\n Note: "del x" doesn\'t directly call "x.__del__()" --- the former\n decrements the reference count for "x" by one, and the latter is\n only called when "x"\'s reference count reaches zero. Some common\n situations that may prevent the reference count of an object from\n going to zero include: circular references between objects (e.g.,\n a doubly-linked list or a tree data structure with parent and\n child pointers); a reference to the object on the stack frame of\n a function that caught an exception (the traceback stored in\n "sys.exc_info()[2]" keeps the stack frame alive); or a reference\n to the object on the stack frame that raised an unhandled\n exception in interactive mode (the traceback stored in\n "sys.last_traceback" keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing "None" in\n "sys.last_traceback". Circular references which are garbage are\n detected and cleaned up when the cyclic garbage collector is\n enabled (it\'s on by default). Refer to the documentation for the\n "gc" module for more information about this topic.\n\n Warning: Due to the precarious circumstances under which\n "__del__()" methods are invoked, exceptions that occur during\n their execution are ignored, and a warning is printed to\n "sys.stderr" instead. Also, when "__del__()" is invoked in\n response to a module being deleted (e.g., when execution of the\n program is done), other globals referenced by the "__del__()"\n method may already have been deleted or in the process of being\n torn down (e.g. the import machinery shutting down). For this\n reason, "__del__()" methods should do the absolute minimum needed\n to maintain external invariants. Starting with version 1.5,\n Python guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the "__del__()" method is called.\n\nobject.__repr__(self)\n\n Called by the "repr()" built-in function to compute the "official"\n string representation of an object. If at all possible, this\n should look like a valid Python expression that could be used to\n recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n "<...some useful description...>" should be returned. The return\n value must be a string object. If a class defines "__repr__()" but\n not "__str__()", then "__repr__()" is also used when an "informal"\n string representation of instances of that class is required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by "str(object)" and the built-in functions "format()" and\n "print()" to compute the "informal" or nicely printable string\n representation of an object. The return value must be a *string*\n object.\n\n This method differs from "object.__repr__()" in that there is no\n expectation that "__str__()" return a valid Python expression: a\n more convenient or concise representation can be used.\n\n The default implementation defined by the built-in type "object"\n calls "object.__repr__()".\n\nobject.__bytes__(self)\n\n Called by "bytes()" to compute a byte-string representation of an\n object. This should return a "bytes" object.\n\nobject.__format__(self, format_spec)\n\n Called by the "format()" built-in function (and by extension, the\n "str.format()" method of class "str") to produce a "formatted"\n string representation of an object. The "format_spec" argument is a\n string that contains a description of the formatting options\n desired. The interpretation of the "format_spec" argument is up to\n the type implementing "__format__()", however most classes will\n either delegate formatting to one of the built-in types, or use a\n similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: "xy" calls\n "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n\n A rich comparison method may return the singleton "NotImplemented"\n if it does not implement the operation for a given pair of\n arguments. By convention, "False" and "True" are returned for a\n successful comparison. However, these methods can return any value,\n so if the comparison operator is used in a Boolean context (e.g.,\n in the condition of an "if" statement), Python will call "bool()"\n on the value to determine if the result is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of "x==y" does not imply that "x!=y" is false.\n Accordingly, when defining "__eq__()", one should also define\n "__ne__()" so that the operators will behave as expected. See the\n paragraph on "__hash__()" for some important notes on creating\n *hashable* objects which support custom comparison operations and\n are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, "__lt__()" and "__gt__()" are each other\'s\n reflection, "__le__()" and "__ge__()" are each other\'s reflection,\n and "__eq__()" and "__ne__()" are their own reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see "functools.total_ordering()".\n\nobject.__hash__(self)\n\n Called by built-in function "hash()" and for operations on members\n of hashed collections including "set", "frozenset", and "dict".\n "__hash__()" should return an integer. The only required property\n is that objects which compare equal have the same hash value; it is\n advised to somehow mix together (e.g. using exclusive or) the hash\n values for the components of the object that also play a part in\n comparison of objects.\n\n Note: "hash()" truncates the value returned from an object\'s\n custom "__hash__()" method to the size of a "Py_ssize_t". This\n is typically 8 bytes on 64-bit builds and 4 bytes on 32-bit\n builds. If an object\'s "__hash__()" must interoperate on builds\n of different bit sizes, be sure to check the width on all\n supported builds. An easy way to do this is with "python -c\n "import sys; print(sys.hash_info.width)""\n\n If a class does not define an "__eq__()" method it should not\n define a "__hash__()" operation either; if it defines "__eq__()"\n but not "__hash__()", its instances will not be usable as items in\n hashable collections. If a class defines mutable objects and\n implements an "__eq__()" method, it should not implement\n "__hash__()", since the implementation of hashable collections\n requires that a key\'s hash value is immutable (if the object\'s hash\n value changes, it will be in the wrong hash bucket).\n\n User-defined classes have "__eq__()" and "__hash__()" methods by\n default; with them, all objects compare unequal (except with\n themselves) and "x.__hash__()" returns an appropriate value such\n that "x == y" implies both that "x is y" and "hash(x) == hash(y)".\n\n A class that overrides "__eq__()" and does not define "__hash__()"\n will have its "__hash__()" implicitly set to "None". When the\n "__hash__()" method of a class is "None", instances of the class\n will raise an appropriate "TypeError" when a program attempts to\n retrieve their hash value, and will also be correctly identified as\n unhashable when checking "isinstance(obj, collections.Hashable").\n\n If a class that overrides "__eq__()" needs to retain the\n implementation of "__hash__()" from a parent class, the interpreter\n must be told this explicitly by setting "__hash__ =\n .__hash__".\n\n If a class that does not override "__eq__()" wishes to suppress\n hash support, it should include "__hash__ = None" in the class\n definition. A class which defines its own "__hash__()" that\n explicitly raises a "TypeError" would be incorrectly identified as\n hashable by an "isinstance(obj, collections.Hashable)" call.\n\n Note: By default, the "__hash__()" values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the iteration order of\n dicts, sets and other mappings. Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also "PYTHONHASHSEED".\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n "bool()"; should return "False" or "True". When this method is not\n defined, "__len__()" is called, if it is defined, and the object is\n considered true if its result is nonzero. If a class defines\n neither "__len__()" nor "__bool__()", all its instances are\n considered true.\n', + 'customization': '\nBasic customization\n*******************\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. "__new__()" is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of "__new__()" should be the new object instance (usually an\n instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s "__new__()" method using\n "super(currentclass, cls).__new__(cls[, ...])" with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If "__new__()" returns an instance of *cls*, then the new\n instance\'s "__init__()" method will be invoked like\n "__init__(self[, ...])", where *self* is the new instance and the\n remaining arguments are the same as were passed to "__new__()".\n\n If "__new__()" does not return an instance of *cls*, then the new\n instance\'s "__init__()" method will not be invoked.\n\n "__new__()" is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n "__init__()" method, the derived class\'s "__init__()" method, if\n any, must explicitly call it to ensure proper initialization of the\n base class part of the instance; for example:\n "BaseClass.__init__(self, [args...])". As a special constraint on\n constructors, no value may be returned; doing so will cause a\n "TypeError" to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a "__del__()" method, the\n derived class\'s "__del__()" method, if any, must explicitly call it\n to ensure proper deletion of the base class part of the instance.\n Note that it is possible (though not recommended!) for the\n "__del__()" method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n "__del__()" methods are called for objects that still exist when\n the interpreter exits.\n\n Note: "del x" doesn\'t directly call "x.__del__()" --- the former\n decrements the reference count for "x" by one, and the latter is\n only called when "x"\'s reference count reaches zero. Some common\n situations that may prevent the reference count of an object from\n going to zero include: circular references between objects (e.g.,\n a doubly-linked list or a tree data structure with parent and\n child pointers); a reference to the object on the stack frame of\n a function that caught an exception (the traceback stored in\n "sys.exc_info()[2]" keeps the stack frame alive); or a reference\n to the object on the stack frame that raised an unhandled\n exception in interactive mode (the traceback stored in\n "sys.last_traceback" keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing "None" in\n "sys.last_traceback". Circular references which are garbage are\n detected and cleaned up when the cyclic garbage collector is\n enabled (it\'s on by default). Refer to the documentation for the\n "gc" module for more information about this topic.\n\n Warning: Due to the precarious circumstances under which\n "__del__()" methods are invoked, exceptions that occur during\n their execution are ignored, and a warning is printed to\n "sys.stderr" instead. Also, when "__del__()" is invoked in\n response to a module being deleted (e.g., when execution of the\n program is done), other globals referenced by the "__del__()"\n method may already have been deleted or in the process of being\n torn down (e.g. the import machinery shutting down). For this\n reason, "__del__()" methods should do the absolute minimum needed\n to maintain external invariants. Starting with version 1.5,\n Python guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the "__del__()" method is called.\n\nobject.__repr__(self)\n\n Called by the "repr()" built-in function to compute the "official"\n string representation of an object. If at all possible, this\n should look like a valid Python expression that could be used to\n recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n "<...some useful description...>" should be returned. The return\n value must be a string object. If a class defines "__repr__()" but\n not "__str__()", then "__repr__()" is also used when an "informal"\n string representation of instances of that class is required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by "str(object)" and the built-in functions "format()" and\n "print()" to compute the "informal" or nicely printable string\n representation of an object. The return value must be a *string*\n object.\n\n This method differs from "object.__repr__()" in that there is no\n expectation that "__str__()" return a valid Python expression: a\n more convenient or concise representation can be used.\n\n The default implementation defined by the built-in type "object"\n calls "object.__repr__()".\n\nobject.__bytes__(self)\n\n Called by "bytes()" to compute a byte-string representation of an\n object. This should return a "bytes" object.\n\nobject.__format__(self, format_spec)\n\n Called by the "format()" built-in function (and by extension, the\n "str.format()" method of class "str") to produce a "formatted"\n string representation of an object. The "format_spec" argument is a\n string that contains a description of the formatting options\n desired. The interpretation of the "format_spec" argument is up to\n the type implementing "__format__()", however most classes will\n either delegate formatting to one of the built-in types, or use a\n similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\n Changed in version 3.4: The __format__ method of "object" itself\n raises a "TypeError" if passed any non-empty string.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: "xy" calls\n "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n\n A rich comparison method may return the singleton "NotImplemented"\n if it does not implement the operation for a given pair of\n arguments. By convention, "False" and "True" are returned for a\n successful comparison. However, these methods can return any value,\n so if the comparison operator is used in a Boolean context (e.g.,\n in the condition of an "if" statement), Python will call "bool()"\n on the value to determine if the result is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of "x==y" does not imply that "x!=y" is false.\n Accordingly, when defining "__eq__()", one should also define\n "__ne__()" so that the operators will behave as expected. See the\n paragraph on "__hash__()" for some important notes on creating\n *hashable* objects which support custom comparison operations and\n are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, "__lt__()" and "__gt__()" are each other\'s\n reflection, "__le__()" and "__ge__()" are each other\'s reflection,\n and "__eq__()" and "__ne__()" are their own reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see "functools.total_ordering()".\n\nobject.__hash__(self)\n\n Called by built-in function "hash()" and for operations on members\n of hashed collections including "set", "frozenset", and "dict".\n "__hash__()" should return an integer. The only required property\n is that objects which compare equal have the same hash value; it is\n advised to somehow mix together (e.g. using exclusive or) the hash\n values for the components of the object that also play a part in\n comparison of objects.\n\n Note: "hash()" truncates the value returned from an object\'s\n custom "__hash__()" method to the size of a "Py_ssize_t". This\n is typically 8 bytes on 64-bit builds and 4 bytes on 32-bit\n builds. If an object\'s "__hash__()" must interoperate on builds\n of different bit sizes, be sure to check the width on all\n supported builds. An easy way to do this is with "python -c\n "import sys; print(sys.hash_info.width)""\n\n If a class does not define an "__eq__()" method it should not\n define a "__hash__()" operation either; if it defines "__eq__()"\n but not "__hash__()", its instances will not be usable as items in\n hashable collections. If a class defines mutable objects and\n implements an "__eq__()" method, it should not implement\n "__hash__()", since the implementation of hashable collections\n requires that a key\'s hash value is immutable (if the object\'s hash\n value changes, it will be in the wrong hash bucket).\n\n User-defined classes have "__eq__()" and "__hash__()" methods by\n default; with them, all objects compare unequal (except with\n themselves) and "x.__hash__()" returns an appropriate value such\n that "x == y" implies both that "x is y" and "hash(x) == hash(y)".\n\n A class that overrides "__eq__()" and does not define "__hash__()"\n will have its "__hash__()" implicitly set to "None". When the\n "__hash__()" method of a class is "None", instances of the class\n will raise an appropriate "TypeError" when a program attempts to\n retrieve their hash value, and will also be correctly identified as\n unhashable when checking "isinstance(obj, collections.Hashable").\n\n If a class that overrides "__eq__()" needs to retain the\n implementation of "__hash__()" from a parent class, the interpreter\n must be told this explicitly by setting "__hash__ =\n .__hash__".\n\n If a class that does not override "__eq__()" wishes to suppress\n hash support, it should include "__hash__ = None" in the class\n definition. A class which defines its own "__hash__()" that\n explicitly raises a "TypeError" would be incorrectly identified as\n hashable by an "isinstance(obj, collections.Hashable)" call.\n\n Note: By default, the "__hash__()" values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the iteration order of\n dicts, sets and other mappings. Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also "PYTHONHASHSEED".\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n "bool()"; should return "False" or "True". When this method is not\n defined, "__len__()" is called, if it is defined, and the object is\n considered true if its result is nonzero. If a class defines\n neither "__len__()" nor "__bool__()", all its instances are\n considered true.\n', 'debugger': '\n"pdb" --- The Python Debugger\n*****************************\n\nThe module "pdb" defines an interactive source code debugger for\nPython programs. It supports setting (conditional) breakpoints and\nsingle stepping at the source line level, inspection of stack frames,\nsource code listing, and evaluation of arbitrary Python code in the\ncontext of any stack frame. It also supports post-mortem debugging\nand can be called under program control.\n\nThe debugger is extensible -- it is actually defined as the class\n"Pdb". This is currently undocumented but easily understood by reading\nthe source. The extension interface uses the modules "bdb" and "cmd".\n\nThe debugger\'s prompt is "(Pdb)". Typical usage to run a program under\ncontrol of the debugger is:\n\n >>> import pdb\n >>> import mymodule\n >>> pdb.run(\'mymodule.test()\')\n > (0)?()\n (Pdb) continue\n > (1)?()\n (Pdb) continue\n NameError: \'spam\'\n > (1)?()\n (Pdb)\n\nChanged in version 3.3: Tab-completion via the "readline" module is\navailable for commands and command arguments, e.g. the current global\nand local names are offered as arguments of the "p" command.\n\n"pdb.py" can also be invoked as a script to debug other scripts. For\nexample:\n\n python3 -m pdb myscript.py\n\nWhen invoked as a script, pdb will automatically enter post-mortem\ndebugging if the program being debugged exits abnormally. After post-\nmortem debugging (or after normal exit of the program), pdb will\nrestart the program. Automatic restarting preserves pdb\'s state (such\nas breakpoints) and in most cases is more useful than quitting the\ndebugger upon program\'s exit.\n\nNew in version 3.2: "pdb.py" now accepts a "-c" option that executes\ncommands as if given in a ".pdbrc" file, see *Debugger Commands*.\n\nThe typical usage to break into the debugger from a running program is\nto insert\n\n import pdb; pdb.set_trace()\n\nat the location you want to break into the debugger. You can then\nstep through the code following this statement, and continue running\nwithout the debugger using the "continue" command.\n\nThe typical usage to inspect a crashed program is:\n\n >>> import pdb\n >>> import mymodule\n >>> mymodule.test()\n Traceback (most recent call last):\n File "", line 1, in ?\n File "./mymodule.py", line 4, in test\n test2()\n File "./mymodule.py", line 3, in test2\n print(spam)\n NameError: spam\n >>> pdb.pm()\n > ./mymodule.py(3)test2()\n -> print(spam)\n (Pdb)\n\nThe module defines the following functions; each enters the debugger\nin a slightly different way:\n\npdb.run(statement, globals=None, locals=None)\n\n Execute the *statement* (given as a string or a code object) under\n debugger control. The debugger prompt appears before any code is\n executed; you can set breakpoints and type "continue", or you can\n step through the statement using "step" or "next" (all these\n commands are explained below). The optional *globals* and *locals*\n arguments specify the environment in which the code is executed; by\n default the dictionary of the module "__main__" is used. (See the\n explanation of the built-in "exec()" or "eval()" functions.)\n\npdb.runeval(expression, globals=None, locals=None)\n\n Evaluate the *expression* (given as a string or a code object)\n under debugger control. When "runeval()" returns, it returns the\n value of the expression. Otherwise this function is similar to\n "run()".\n\npdb.runcall(function, *args, **kwds)\n\n Call the *function* (a function or method object, not a string)\n with the given arguments. When "runcall()" returns, it returns\n whatever the function call returned. The debugger prompt appears\n as soon as the function is entered.\n\npdb.set_trace()\n\n Enter the debugger at the calling stack frame. This is useful to\n hard-code a breakpoint at a given point in a program, even if the\n code is not otherwise being debugged (e.g. when an assertion\n fails).\n\npdb.post_mortem(traceback=None)\n\n Enter post-mortem debugging of the given *traceback* object. If no\n *traceback* is given, it uses the one of the exception that is\n currently being handled (an exception must be being handled if the\n default is to be used).\n\npdb.pm()\n\n Enter post-mortem debugging of the traceback found in\n "sys.last_traceback".\n\nThe "run*" functions and "set_trace()" are aliases for instantiating\nthe "Pdb" class and calling the method of the same name. If you want\nto access further features, you have to do this yourself:\n\nclass class pdb.Pdb(completekey=\'tab\', stdin=None, stdout=None, skip=None, nosigint=False)\n\n "Pdb" is the debugger class.\n\n The *completekey*, *stdin* and *stdout* arguments are passed to the\n underlying "cmd.Cmd" class; see the description there.\n\n The *skip* argument, if given, must be an iterable of glob-style\n module name patterns. The debugger will not step into frames that\n originate in a module that matches one of these patterns. [1]\n\n By default, Pdb sets a handler for the SIGINT signal (which is sent\n when the user presses Ctrl-C on the console) when you give a\n "continue" command. This allows you to break into the debugger\n again by pressing Ctrl-C. If you want Pdb not to touch the SIGINT\n handler, set *nosigint* tot true.\n\n Example call to enable tracing with *skip*:\n\n import pdb; pdb.Pdb(skip=[\'django.*\']).set_trace()\n\n New in version 3.1: The *skip* argument.\n\n New in version 3.2: The *nosigint* argument. Previously, a SIGINT\n handler was never set by Pdb.\n\n run(statement, globals=None, locals=None)\n runeval(expression, globals=None, locals=None)\n runcall(function, *args, **kwds)\n set_trace()\n\n See the documentation for the functions explained above.\n\n\nDebugger Commands\n=================\n\nThe commands recognized by the debugger are listed below. Most\ncommands can be abbreviated to one or two letters as indicated; e.g.\n"h(elp)" means that either "h" or "help" can be used to enter the help\ncommand (but not "he" or "hel", nor "H" or "Help" or "HELP").\nArguments to commands must be separated by whitespace (spaces or\ntabs). Optional arguments are enclosed in square brackets ("[]") in\nthe command syntax; the square brackets must not be typed.\nAlternatives in the command syntax are separated by a vertical bar\n("|").\n\nEntering a blank line repeats the last command entered. Exception: if\nthe last command was a "list" command, the next 11 lines are listed.\n\nCommands that the debugger doesn\'t recognize are assumed to be Python\nstatements and are executed in the context of the program being\ndebugged. Python statements can also be prefixed with an exclamation\npoint ("!"). This is a powerful way to inspect the program being\ndebugged; it is even possible to change a variable or call a function.\nWhen an exception occurs in such a statement, the exception name is\nprinted but the debugger\'s state is not changed.\n\nThe debugger supports *aliases*. Aliases can have parameters which\nallows one a certain level of adaptability to the context under\nexamination.\n\nMultiple commands may be entered on a single line, separated by ";;".\n(A single ";" is not used as it is the separator for multiple commands\nin a line that is passed to the Python parser.) No intelligence is\napplied to separating the commands; the input is split at the first\n";;" pair, even if it is in the middle of a quoted string.\n\nIf a file ".pdbrc" exists in the user\'s home directory or in the\ncurrent directory, it is read in and executed as if it had been typed\nat the debugger prompt. This is particularly useful for aliases. If\nboth files exist, the one in the home directory is read first and\naliases defined there can be overridden by the local file.\n\nChanged in version 3.2: ".pdbrc" can now contain commands that\ncontinue debugging, such as "continue" or "next". Previously, these\ncommands had no effect.\n\nh(elp) [command]\n\n Without argument, print the list of available commands. With a\n *command* as argument, print help about that command. "help pdb"\n displays the full documentation (the docstring of the "pdb"\n module). Since the *command* argument must be an identifier, "help\n exec" must be entered to get help on the "!" command.\n\nw(here)\n\n Print a stack trace, with the most recent frame at the bottom. An\n arrow indicates the current frame, which determines the context of\n most commands.\n\nd(own) [count]\n\n Move the current frame *count* (default one) levels down in the\n stack trace (to a newer frame).\n\nu(p) [count]\n\n Move the current frame *count* (default one) levels up in the stack\n trace (to an older frame).\n\nb(reak) [([filename:]lineno | function) [, condition]]\n\n With a *lineno* argument, set a break there in the current file.\n With a *function* argument, set a break at the first executable\n statement within that function. The line number may be prefixed\n with a filename and a colon, to specify a breakpoint in another\n file (probably one that hasn\'t been loaded yet). The file is\n searched on "sys.path". Note that each breakpoint is assigned a\n number to which all the other breakpoint commands refer.\n\n If a second argument is present, it is an expression which must\n evaluate to true before the breakpoint is honored.\n\n Without argument, list all breaks, including for each breakpoint,\n the number of times that breakpoint has been hit, the current\n ignore count, and the associated condition if any.\n\ntbreak [([filename:]lineno | function) [, condition]]\n\n Temporary breakpoint, which is removed automatically when it is\n first hit. The arguments are the same as for "break".\n\ncl(ear) [filename:lineno | bpnumber [bpnumber ...]]\n\n With a *filename:lineno* argument, clear all the breakpoints at\n this line. With a space separated list of breakpoint numbers, clear\n those breakpoints. Without argument, clear all breaks (but first\n ask confirmation).\n\ndisable [bpnumber [bpnumber ...]]\n\n Disable the breakpoints given as a space separated list of\n breakpoint numbers. Disabling a breakpoint means it cannot cause\n the program to stop execution, but unlike clearing a breakpoint, it\n remains in the list of breakpoints and can be (re-)enabled.\n\nenable [bpnumber [bpnumber ...]]\n\n Enable the breakpoints specified.\n\nignore bpnumber [count]\n\n Set the ignore count for the given breakpoint number. If count is\n omitted, the ignore count is set to 0. A breakpoint becomes active\n when the ignore count is zero. When non-zero, the count is\n decremented each time the breakpoint is reached and the breakpoint\n is not disabled and any associated condition evaluates to true.\n\ncondition bpnumber [condition]\n\n Set a new *condition* for the breakpoint, an expression which must\n evaluate to true before the breakpoint is honored. If *condition*\n is absent, any existing condition is removed; i.e., the breakpoint\n is made unconditional.\n\ncommands [bpnumber]\n\n Specify a list of commands for breakpoint number *bpnumber*. The\n commands themselves appear on the following lines. Type a line\n containing just "end" to terminate the commands. An example:\n\n (Pdb) commands 1\n (com) p some_variable\n (com) end\n (Pdb)\n\n To remove all commands from a breakpoint, type commands and follow\n it immediately with "end"; that is, give no commands.\n\n With no *bpnumber* argument, commands refers to the last breakpoint\n set.\n\n You can use breakpoint commands to start your program up again.\n Simply use the continue command, or step, or any other command that\n resumes execution.\n\n Specifying any command resuming execution (currently continue,\n step, next, return, jump, quit and their abbreviations) terminates\n the command list (as if that command was immediately followed by\n end). This is because any time you resume execution (even with a\n simple next or step), you may encounter another breakpoint--which\n could have its own command list, leading to ambiguities about which\n list to execute.\n\n If you use the \'silent\' command in the command list, the usual\n message about stopping at a breakpoint is not printed. This may be\n desirable for breakpoints that are to print a specific message and\n then continue. If none of the other commands print anything, you\n see no sign that the breakpoint was reached.\n\ns(tep)\n\n Execute the current line, stop at the first possible occasion\n (either in a function that is called or on the next line in the\n current function).\n\nn(ext)\n\n Continue execution until the next line in the current function is\n reached or it returns. (The difference between "next" and "step"\n is that "step" stops inside a called function, while "next"\n executes called functions at (nearly) full speed, only stopping at\n the next line in the current function.)\n\nunt(il) [lineno]\n\n Without argument, continue execution until the line with a number\n greater than the current one is reached.\n\n With a line number, continue execution until a line with a number\n greater or equal to that is reached. In both cases, also stop when\n the current frame returns.\n\n Changed in version 3.2: Allow giving an explicit line number.\n\nr(eturn)\n\n Continue execution until the current function returns.\n\nc(ont(inue))\n\n Continue execution, only stop when a breakpoint is encountered.\n\nj(ump) lineno\n\n Set the next line that will be executed. Only available in the\n bottom-most frame. This lets you jump back and execute code again,\n or jump forward to skip code that you don\'t want to run.\n\n It should be noted that not all jumps are allowed -- for instance\n it is not possible to jump into the middle of a "for" loop or out\n of a "finally" clause.\n\nl(ist) [first[, last]]\n\n List source code for the current file. Without arguments, list 11\n lines around the current line or continue the previous listing.\n With "." as argument, list 11 lines around the current line. With\n one argument, list 11 lines around at that line. With two\n arguments, list the given range; if the second argument is less\n than the first, it is interpreted as a count.\n\n The current line in the current frame is indicated by "->". If an\n exception is being debugged, the line where the exception was\n originally raised or propagated is indicated by ">>", if it differs\n from the current line.\n\n New in version 3.2: The ">>" marker.\n\nll | longlist\n\n List all source code for the current function or frame.\n Interesting lines are marked as for "list".\n\n New in version 3.2.\n\na(rgs)\n\n Print the argument list of the current function.\n\np expression\n\n Evaluate the *expression* in the current context and print its\n value.\n\n Note: "print()" can also be used, but is not a debugger command\n --- this executes the Python "print()" function.\n\npp expression\n\n Like the "p" command, except the value of the expression is pretty-\n printed using the "pprint" module.\n\nwhatis expression\n\n Print the type of the *expression*.\n\nsource expression\n\n Try to get source code for the given object and display it.\n\n New in version 3.2.\n\ndisplay [expression]\n\n Display the value of the expression if it changed, each time\n execution stops in the current frame.\n\n Without expression, list all display expressions for the current\n frame.\n\n New in version 3.2.\n\nundisplay [expression]\n\n Do not display the expression any more in the current frame.\n Without expression, clear all display expressions for the current\n frame.\n\n New in version 3.2.\n\ninteract\n\n Start an interative interpreter (using the "code" module) whose\n global namespace contains all the (global and local) names found in\n the current scope.\n\n New in version 3.2.\n\nalias [name [command]]\n\n Create an alias called *name* that executes *command*. The command\n must *not* be enclosed in quotes. Replaceable parameters can be\n indicated by "%1", "%2", and so on, while "%*" is replaced by all\n the parameters. If no command is given, the current alias for\n *name* is shown. If no arguments are given, all aliases are listed.\n\n Aliases may be nested and can contain anything that can be legally\n typed at the pdb prompt. Note that internal pdb commands *can* be\n overridden by aliases. Such a command is then hidden until the\n alias is removed. Aliasing is recursively applied to the first\n word of the command line; all other words in the line are left\n alone.\n\n As an example, here are two useful aliases (especially when placed\n in the ".pdbrc" file):\n\n # Print instance variables (usage "pi classInst")\n alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k])\n # Print instance variables in self\n alias ps pi self\n\nunalias name\n\n Delete the specified alias.\n\n! statement\n\n Execute the (one-line) *statement* in the context of the current\n stack frame. The exclamation point can be omitted unless the first\n word of the statement resembles a debugger command. To set a\n global variable, you can prefix the assignment command with a\n "global" statement on the same line, e.g.:\n\n (Pdb) global list_options; list_options = [\'-l\']\n (Pdb)\n\nrun [args ...]\nrestart [args ...]\n\n Restart the debugged Python program. If an argument is supplied,\n it is split with "shlex" and the result is used as the new\n "sys.argv". History, breakpoints, actions and debugger options are\n preserved. "restart" is an alias for "run".\n\nq(uit)\n\n Quit from the debugger. The program being executed is aborted.\n\n-[ Footnotes ]-\n\n[1] Whether a frame is considered to originate in a certain module\n is determined by the "__name__" in the frame globals.\n', 'del': '\nThe "del" statement\n*******************\n\n del_stmt ::= "del" target_list\n\nDeletion is recursively defined very similar to the way assignment is\ndefined. Rather than spelling it out in full details, here are some\nhints.\n\nDeletion of a target list recursively deletes each target, from left\nto right.\n\nDeletion of a name removes the binding of that name from the local or\nglobal namespace, depending on whether the name occurs in a "global"\nstatement in the same code block. If the name is unbound, a\n"NameError" exception will be raised.\n\nDeletion of attribute references, subscriptions and slicings is passed\nto the primary object involved; deletion of a slicing is in general\nequivalent to assignment of an empty slice of the right type (but even\nthis is determined by the sliced object).\n\nChanged in version 3.2: Previously it was illegal to delete a name\nfrom the local namespace if it occurs as a free variable in a nested\nblock.\n', 'dict': '\nDictionary displays\n*******************\n\nA dictionary display is a possibly empty series of key/datum pairs\nenclosed in curly braces:\n\n dict_display ::= "{" [key_datum_list | dict_comprehension] "}"\n key_datum_list ::= key_datum ("," key_datum)* [","]\n key_datum ::= expression ":" expression\n dict_comprehension ::= expression ":" expression comp_for\n\nA dictionary display yields a new dictionary object.\n\nIf a comma-separated sequence of key/datum pairs is given, they are\nevaluated from left to right to define the entries of the dictionary:\neach key object is used as a key into the dictionary to store the\ncorresponding datum. This means that you can specify the same key\nmultiple times in the key/datum list, and the final dictionary\'s value\nfor that key will be the last one given.\n\nA dict comprehension, in contrast to list and set comprehensions,\nneeds two expressions separated with a colon followed by the usual\n"for" and "if" clauses. When the comprehension is run, the resulting\nkey and value elements are inserted in the new dictionary in the order\nthey are produced.\n\nRestrictions on the types of the key values are listed earlier in\nsection *The standard type hierarchy*. (To summarize, the key type\nshould be *hashable*, which excludes all mutable objects.) Clashes\nbetween duplicate keys are not detected; the last datum (textually\nrightmost in the display) stored for a given key value prevails.\n', @@ -49,7 +49,7 @@ 'naming': '\nNaming and binding\n******************\n\n*Names* refer to objects. Names are introduced by name binding\noperations. Each occurrence of a name in the program text refers to\nthe *binding* of that name established in the innermost function block\ncontaining the use.\n\nA *block* is a piece of Python program text that is executed as a\nunit. The following are blocks: a module, a function body, and a class\ndefinition. Each command typed interactively is a block. A script\nfile (a file given as standard input to the interpreter or specified\non the interpreter command line the first argument) is a code block.\nA script command (a command specified on the interpreter command line\nwith the \'**-c**\' option) is a code block. The string argument passed\nto the built-in functions "eval()" and "exec()" is a code block.\n\nA code block is executed in an *execution frame*. A frame contains\nsome administrative information (used for debugging) and determines\nwhere and how execution continues after the code block\'s execution has\ncompleted.\n\nA *scope* defines the visibility of a name within a block. If a local\nvariable is defined in a block, its scope includes that block. If the\ndefinition occurs in a function block, the scope extends to any blocks\ncontained within the defining one, unless a contained block introduces\na different binding for the name. The scope of names defined in a\nclass block is limited to the class block; it does not extend to the\ncode blocks of methods -- this includes comprehensions and generator\nexpressions since they are implemented using a function scope. This\nmeans that the following will fail:\n\n class A:\n a = 42\n b = list(a + i for i in range(10))\n\nWhen a name is used in a code block, it is resolved using the nearest\nenclosing scope. The set of all such scopes visible to a code block\nis called the block\'s *environment*.\n\nIf a name is bound in a block, it is a local variable of that block,\nunless declared as "nonlocal". If a name is bound at the module\nlevel, it is a global variable. (The variables of the module code\nblock are local and global.) If a variable is used in a code block\nbut not defined there, it is a *free variable*.\n\nWhen a name is not found at all, a "NameError" exception is raised.\nIf the name refers to a local variable that has not been bound, a\n"UnboundLocalError" exception is raised. "UnboundLocalError" is a\nsubclass of "NameError".\n\nThe following constructs bind names: formal parameters to functions,\n"import" statements, class and function definitions (these bind the\nclass or function name in the defining block), and targets that are\nidentifiers if occurring in an assignment, "for" loop header, or after\n"as" in a "with" statement or "except" clause. The "import" statement\nof the form "from ... import *" binds all names defined in the\nimported module, except those beginning with an underscore. This form\nmay only be used at the module level.\n\nA target occurring in a "del" statement is also considered bound for\nthis purpose (though the actual semantics are to unbind the name).\n\nEach assignment or import statement occurs within a block defined by a\nclass or function definition or at the module level (the top-level\ncode block).\n\nIf a name binding operation occurs anywhere within a code block, all\nuses of the name within the block are treated as references to the\ncurrent block. This can lead to errors when a name is used within a\nblock before it is bound. This rule is subtle. Python lacks\ndeclarations and allows name binding operations to occur anywhere\nwithin a code block. The local variables of a code block can be\ndetermined by scanning the entire text of the block for name binding\noperations.\n\nIf the "global" statement occurs within a block, all uses of the name\nspecified in the statement refer to the binding of that name in the\ntop-level namespace. Names are resolved in the top-level namespace by\nsearching the global namespace, i.e. the namespace of the module\ncontaining the code block, and the builtins namespace, the namespace\nof the module "builtins". The global namespace is searched first. If\nthe name is not found there, the builtins namespace is searched. The\nglobal statement must precede all uses of the name.\n\nThe builtins namespace associated with the execution of a code block\nis actually found by looking up the name "__builtins__" in its global\nnamespace; this should be a dictionary or a module (in the latter case\nthe module\'s dictionary is used). By default, when in the "__main__"\nmodule, "__builtins__" is the built-in module "builtins"; when in any\nother module, "__builtins__" is an alias for the dictionary of the\n"builtins" module itself. "__builtins__" can be set to a user-created\ndictionary to create a weak form of restricted execution.\n\n**CPython implementation detail:** Users should not touch\n"__builtins__"; it is strictly an implementation detail. Users\nwanting to override values in the builtins namespace should "import"\nthe "builtins" module and modify its attributes appropriately.\n\nThe namespace for a module is automatically created the first time a\nmodule is imported. The main module for a script is always called\n"__main__".\n\nThe "global" statement has the same scope as a name binding operation\nin the same block. If the nearest enclosing scope for a free variable\ncontains a global statement, the free variable is treated as a global.\n\nA class definition is an executable statement that may use and define\nnames. These references follow the normal rules for name resolution.\nThe namespace of the class definition becomes the attribute dictionary\nof the class. Names defined at the class scope are not visible in\nmethods.\n\n\nInteraction with dynamic features\n=================================\n\nThere are several cases where Python statements are illegal when used\nin conjunction with nested scopes that contain free variables.\n\nIf a variable is referenced in an enclosing scope, it is illegal to\ndelete the name. An error will be reported at compile time.\n\nIf the wild card form of import --- "import *" --- is used in a\nfunction and the function contains or is a nested block with free\nvariables, the compiler will raise a "SyntaxError".\n\nThe "eval()" and "exec()" functions do not have access to the full\nenvironment for resolving names. Names may be resolved in the local\nand global namespaces of the caller. Free variables are not resolved\nin the nearest enclosing namespace, but in the global namespace. [1]\nThe "exec()" and "eval()" functions have optional arguments to\noverride the global and local namespace. If only one namespace is\nspecified, it is used for both.\n', 'nonlocal': '\nThe "nonlocal" statement\n************************\n\n nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*\n\nThe "nonlocal" statement causes the listed identifiers to refer to\npreviously bound variables in the nearest enclosing scope. This is\nimportant because the default behavior for binding is to search the\nlocal namespace first. The statement allows encapsulated code to\nrebind variables outside of the local scope besides the global\n(module) scope.\n\nNames listed in a "nonlocal" statement, unlike to those listed in a\n"global" statement, must refer to pre-existing bindings in an\nenclosing scope (the scope in which a new binding should be created\ncannot be determined unambiguously).\n\nNames listed in a "nonlocal" statement must not collide with pre-\nexisting bindings in the local scope.\n\nSee also: **PEP 3104** - Access to Names in Outer Scopes\n\n The specification for the "nonlocal" statement.\n', 'numbers': '\nNumeric literals\n****************\n\nThere are three types of numeric literals: integers, floating point\nnumbers, and imaginary numbers. There are no complex literals\n(complex numbers can be formed by adding a real number and an\nimaginary number).\n\nNote that numeric literals do not include a sign; a phrase like "-1"\nis actually an expression composed of the unary operator \'"-"\' and the\nliteral "1".\n', - 'numeric-types': '\nEmulating numeric types\n***********************\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|"). For instance, to evaluate the\n expression "x + y", where *x* is an instance of a class that has an\n "__add__()" method, "x.__add__(y)" is called. The "__divmod__()"\n method should be the equivalent to using "__floordiv__()" and\n "__mod__()"; it should not be related to "__truediv__()". Note\n that "__pow__()" should be defined to accept an optional third\n argument if the ternary version of the built-in "pow()" function is\n to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return "NotImplemented".\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|") with reflected (swapped) operands.\n These functions are only called if the left operand does not\n support the corresponding operation and the operands are of\n different types. [2] For instance, to evaluate the expression "x -\n y", where *y* is an instance of a class that has an "__rsub__()"\n method, "y.__rsub__(x)" is called if "x.__sub__(y)" returns\n *NotImplemented*.\n\n Note that ternary "pow()" will not try calling "__rpow__()" (the\n coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left\n operand\'s type and that subclass provides the reflected method\n for the operation, this method will be called before the left\n operand\'s non-reflected method. This behavior allows subclasses\n to override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments ("+=", "-=", "*=", "/=", "//=", "%=", "**=", "<<=",\n ">>=", "&=", "^=", "|="). These methods should attempt to do the\n operation in-place (modifying *self*) and return the result (which\n could be, but does not have to be, *self*). If a specific method\n is not defined, the augmented assignment falls back to the normal\n methods. For instance, to execute the statement "x += y", where\n *x* is an instance of a class that has an "__iadd__()" method,\n "x.__iadd__(y)" is called. If *x* is an instance of a class that\n does not define a "__iadd__()" method, "x.__add__(y)" and\n "y.__radd__(x)" are considered, as with the evaluation of "x + y".\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations ("-", "+",\n "abs()" and "~").\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions "complex()", "int()",\n "float()" and "round()". Should return a value of the appropriate\n type.\n\nobject.__index__(self)\n\n Called to implement "operator.index()", and whenever Python needs\n to losslessly convert the numeric object to an integer object (such\n as in slicing, or in the built-in "bin()", "hex()" and "oct()"\n functions). Presence of this method indicates that the numeric\n object is an integer type. Must return an integer.\n\n Note: When "__index__()" is defined, "__int__()" should also be\n defined, and both shuld return the same value, in order to have a\n coherent integer type class.\n', + 'numeric-types': '\nEmulating numeric types\n***********************\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|"). For instance, to evaluate the\n expression "x + y", where *x* is an instance of a class that has an\n "__add__()" method, "x.__add__(y)" is called. The "__divmod__()"\n method should be the equivalent to using "__floordiv__()" and\n "__mod__()"; it should not be related to "__truediv__()". Note\n that "__pow__()" should be defined to accept an optional third\n argument if the ternary version of the built-in "pow()" function is\n to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return "NotImplemented".\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|") with reflected (swapped) operands.\n These functions are only called if the left operand does not\n support the corresponding operation and the operands are of\n different types. [2] For instance, to evaluate the expression "x -\n y", where *y* is an instance of a class that has an "__rsub__()"\n method, "y.__rsub__(x)" is called if "x.__sub__(y)" returns\n *NotImplemented*.\n\n Note that ternary "pow()" will not try calling "__rpow__()" (the\n coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left\n operand\'s type and that subclass provides the reflected method\n for the operation, this method will be called before the left\n operand\'s non-reflected method. This behavior allows subclasses\n to override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments ("+=", "-=", "*=", "/=", "//=", "%=", "**=", "<<=",\n ">>=", "&=", "^=", "|="). These methods should attempt to do the\n operation in-place (modifying *self*) and return the result (which\n could be, but does not have to be, *self*). If a specific method\n is not defined, the augmented assignment falls back to the normal\n methods. For instance, if *x* is an instance of a class with an\n "__iadd__()" method, "x += y" is equivalent to "x = x.__iadd__(y)"\n . Otherwise, "x.__add__(y)" and "y.__radd__(x)" are considered, as\n with the evaluation of "x + y". In certain situations, augmented\n assignment can result in unexpected errors (see *Why does\n a_tuple[i] += [\'item\'] raise an exception when the addition\n works?*), but this behavior is in fact part of the data model.\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations ("-", "+",\n "abs()" and "~").\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions "complex()", "int()",\n "float()" and "round()". Should return a value of the appropriate\n type.\n\nobject.__index__(self)\n\n Called to implement "operator.index()", and whenever Python needs\n to losslessly convert the numeric object to an integer object (such\n as in slicing, or in the built-in "bin()", "hex()" and "oct()"\n functions). Presence of this method indicates that the numeric\n object is an integer type. Must return an integer.\n\n Note: When "__index__()" is defined, "__int__()" should also be\n defined, and both shuld return the same value, in order to have a\n coherent integer type class.\n', 'objects': '\nObjects, values and types\n*************************\n\n*Objects* are Python\'s abstraction for data. All data in a Python\nprogram is represented by objects or by relations between objects. (In\na sense, and in conformance to Von Neumann\'s model of a "stored\nprogram computer," code is also represented by objects.)\n\nEvery object has an identity, a type and a value. An object\'s\n*identity* never changes once it has been created; you may think of it\nas the object\'s address in memory. The \'"is"\' operator compares the\nidentity of two objects; the "id()" function returns an integer\nrepresenting its identity.\n\n**CPython implementation detail:** For CPython, "id(x)" is the memory\naddress where "x" is stored.\n\nAn object\'s type determines the operations that the object supports\n(e.g., "does it have a length?") and also defines the possible values\nfor objects of that type. The "type()" function returns an object\'s\ntype (which is an object itself). Like its identity, an object\'s\n*type* is also unchangeable. [1]\n\nThe *value* of some objects can change. Objects whose value can\nchange are said to be *mutable*; objects whose value is unchangeable\nonce they are created are called *immutable*. (The value of an\nimmutable container object that contains a reference to a mutable\nobject can change when the latter\'s value is changed; however the\ncontainer is still considered immutable, because the collection of\nobjects it contains cannot be changed. So, immutability is not\nstrictly the same as having an unchangeable value, it is more subtle.)\nAn object\'s mutability is determined by its type; for instance,\nnumbers, strings and tuples are immutable, while dictionaries and\nlists are mutable.\n\nObjects are never explicitly destroyed; however, when they become\nunreachable they may be garbage-collected. An implementation is\nallowed to postpone garbage collection or omit it altogether --- it is\na matter of implementation quality how garbage collection is\nimplemented, as long as no objects are collected that are still\nreachable.\n\n**CPython implementation detail:** CPython currently uses a reference-\ncounting scheme with (optional) delayed detection of cyclically linked\ngarbage, which collects most objects as soon as they become\nunreachable, but is not guaranteed to collect garbage containing\ncircular references. See the documentation of the "gc" module for\ninformation on controlling the collection of cyclic garbage. Other\nimplementations act differently and CPython may change. Do not depend\non immediate finalization of objects when they become unreachable (ex:\nalways close files).\n\nNote that the use of the implementation\'s tracing or debugging\nfacilities may keep objects alive that would normally be collectable.\nAlso note that catching an exception with a \'"try"..."except"\'\nstatement may keep objects alive.\n\nSome objects contain references to "external" resources such as open\nfiles or windows. It is understood that these resources are freed\nwhen the object is garbage-collected, but since garbage collection is\nnot guaranteed to happen, such objects also provide an explicit way to\nrelease the external resource, usually a "close()" method. Programs\nare strongly recommended to explicitly close such objects. The\n\'"try"..."finally"\' statement and the \'"with"\' statement provide\nconvenient ways to do this.\n\nSome objects contain references to other objects; these are called\n*containers*. Examples of containers are tuples, lists and\ndictionaries. The references are part of a container\'s value. In\nmost cases, when we talk about the value of a container, we imply the\nvalues, not the identities of the contained objects; however, when we\ntalk about the mutability of a container, only the identities of the\nimmediately contained objects are implied. So, if an immutable\ncontainer (like a tuple) contains a reference to a mutable object, its\nvalue changes if that mutable object is changed.\n\nTypes affect almost all aspects of object behavior. Even the\nimportance of object identity is affected in some sense: for immutable\ntypes, operations that compute new values may actually return a\nreference to any existing object with the same type and value, while\nfor mutable objects this is not allowed. E.g., after "a = 1; b = 1",\n"a" and "b" may or may not refer to the same object with the value\none, depending on the implementation, but after "c = []; d = []", "c"\nand "d" are guaranteed to refer to two different, unique, newly\ncreated empty lists. (Note that "c = d = []" assigns the same object\nto both "c" and "d".)\n', 'operator-summary': '\nOperator precedence\n*******************\n\nThe following table summarizes the operator precedences in Python,\nfrom lowest precedence (least binding) to highest precedence (most\nbinding). Operators in the same box have the same precedence. Unless\nthe syntax is explicitly given, operators are binary. Operators in\nthe same box group left to right (except for comparisons, including\ntests, which all have the same precedence and chain from left to right\n--- see section *Comparisons* --- and exponentiation, which groups\nfrom right to left).\n\n+-------------------------------------------------+---------------------------------------+\n| Operator | Description |\n+=================================================+=======================================+\n| "lambda" | Lambda expression |\n+-------------------------------------------------+---------------------------------------+\n| "if" -- "else" | Conditional expression |\n+-------------------------------------------------+---------------------------------------+\n| "or" | Boolean OR |\n+-------------------------------------------------+---------------------------------------+\n| "and" | Boolean AND |\n+-------------------------------------------------+---------------------------------------+\n| "not" "x" | Boolean NOT |\n+-------------------------------------------------+---------------------------------------+\n| "in", "not in", "is", "is not", "<", "<=", ">", | Comparisons, including membership |\n| ">=", "!=", "==" | tests and identity tests |\n+-------------------------------------------------+---------------------------------------+\n| "|" | Bitwise OR |\n+-------------------------------------------------+---------------------------------------+\n| "^" | Bitwise XOR |\n+-------------------------------------------------+---------------------------------------+\n| "&" | Bitwise AND |\n+-------------------------------------------------+---------------------------------------+\n| "<<", ">>" | Shifts |\n+-------------------------------------------------+---------------------------------------+\n| "+", "-" | Addition and subtraction |\n+-------------------------------------------------+---------------------------------------+\n| "*", "/", "//", "%" | Multiplication, division, remainder |\n+-------------------------------------------------+---------------------------------------+\n| "+x", "-x", "~x" | Positive, negative, bitwise NOT |\n+-------------------------------------------------+---------------------------------------+\n| "**" | Exponentiation [6] |\n+-------------------------------------------------+---------------------------------------+\n| "x[index]", "x[index:index]", | Subscription, slicing, call, |\n| "x(arguments...)", "x.attribute" | attribute reference |\n+-------------------------------------------------+---------------------------------------+\n| "(expressions...)", "[expressions...]", "{key: | Binding or tuple display, list |\n| value...}", "{expressions...}" | display, dictionary display, set |\n+-------------------------------------------------+---------------------------------------+\n\n-[ Footnotes ]-\n\n[1] While "abs(x%y) < abs(y)" is true mathematically, for floats\n it may not be true numerically due to roundoff. For example, and\n assuming a platform on which a Python float is an IEEE 754 double-\n precision number, in order that "-1e-100 % 1e100" have the same\n sign as "1e100", the computed result is "-1e-100 + 1e100", which\n is numerically exactly equal to "1e100". The function\n "math.fmod()" returns a result whose sign matches the sign of the\n first argument instead, and so returns "-1e-100" in this case.\n Which approach is more appropriate depends on the application.\n\n[2] If x is very close to an exact integer multiple of y, it\'s\n possible for "x//y" to be one larger than "(x-x%y)//y" due to\n rounding. In such cases, Python returns the latter result, in\n order to preserve that "divmod(x,y)[0] * y + x % y" be very close\n to "x".\n\n[3] While comparisons between strings make sense at the byte\n level, they may be counter-intuitive to users. For example, the\n strings ""\\u00C7"" and ""\\u0327\\u0043"" compare differently, even\n though they both represent the same unicode character (LATIN\n CAPITAL LETTER C WITH CEDILLA). To compare strings in a human\n recognizable way, compare using "unicodedata.normalize()".\n\n[4] Due to automatic garbage-collection, free lists, and the\n dynamic nature of descriptors, you may notice seemingly unusual\n behaviour in certain uses of the "is" operator, like those\n involving comparisons between instance methods, or constants.\n Check their documentation for more info.\n\n[5] The "%" operator is also used for string formatting; the same\n precedence applies.\n\n[6] The power operator "**" binds less tightly than an arithmetic\n or bitwise unary operator on its right, that is, "2**-1" is "0.5".\n', 'pass': '\nThe "pass" statement\n********************\n\n pass_stmt ::= "pass"\n\n"pass" is a null operation --- when it is executed, nothing happens.\nIt is useful as a placeholder when a statement is required\nsyntactically, but no code needs to be executed, for example:\n\n def f(arg): pass # a function that does nothing (yet)\n\n class C: pass # a class with no methods (yet)\n', @@ -60,7 +60,7 @@ 'shifting': '\nShifting operations\n*******************\n\nThe shifting operations have lower priority than the arithmetic\noperations:\n\n shift_expr ::= a_expr | shift_expr ( "<<" | ">>" ) a_expr\n\nThese operators accept integers as arguments. They shift the first\nargument to the left or right by the number of bits given by the\nsecond argument.\n\nA right shift by *n* bits is defined as floor division by "pow(2,n)".\nA left shift by *n* bits is defined as multiplication with "pow(2,n)".\n\nNote: In the current implementation, the right-hand operand is\n required to be at most "sys.maxsize". If the right-hand operand is\n larger than "sys.maxsize" an "OverflowError" exception is raised.\n', 'slicings': '\nSlicings\n********\n\nA slicing selects a range of items in a sequence object (e.g., a\nstring, tuple or list). Slicings may be used as expressions or as\ntargets in assignment or "del" statements. The syntax for a slicing:\n\n slicing ::= primary "[" slice_list "]"\n slice_list ::= slice_item ("," slice_item)* [","]\n slice_item ::= expression | proper_slice\n proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]\n lower_bound ::= expression\n upper_bound ::= expression\n stride ::= expression\n\nThere is ambiguity in the formal syntax here: anything that looks like\nan expression list also looks like a slice list, so any subscription\ncan be interpreted as a slicing. Rather than further complicating the\nsyntax, this is disambiguated by defining that in this case the\ninterpretation as a subscription takes priority over the\ninterpretation as a slicing (this is the case if the slice list\ncontains no proper slice).\n\nThe semantics for a slicing are as follows. The primary must evaluate\nto a mapping object, and it is indexed (using the same "__getitem__()"\nmethod as normal subscription) with a key that is constructed from the\nslice list, as follows. If the slice list contains at least one\ncomma, the key is a tuple containing the conversion of the slice\nitems; otherwise, the conversion of the lone slice item is the key.\nThe conversion of a slice item that is an expression is that\nexpression. The conversion of a proper slice is a slice object (see\nsection *The standard type hierarchy*) whose "start", "stop" and\n"step" attributes are the values of the expressions given as lower\nbound, upper bound and stride, respectively, substituting "None" for\nmissing expressions.\n', 'specialattrs': '\nSpecial Attributes\n******************\n\nThe implementation adds a few special read-only attributes to several\nobject types, where they are relevant. Some of these are not reported\nby the "dir()" built-in function.\n\nobject.__dict__\n\n A dictionary or other mapping object used to store an object\'s\n (writable) attributes.\n\ninstance.__class__\n\n The class to which a class instance belongs.\n\nclass.__bases__\n\n The tuple of base classes of a class object.\n\nclass.__name__\n\n The name of the class or type.\n\nclass.__qualname__\n\n The *qualified name* of the class or type.\n\n New in version 3.3.\n\nclass.__mro__\n\n This attribute is a tuple of classes that are considered when\n looking for base classes during method resolution.\n\nclass.mro()\n\n This method can be overridden by a metaclass to customize the\n method resolution order for its instances. It is called at class\n instantiation, and its result is stored in "__mro__".\n\nclass.__subclasses__()\n\n Each class keeps a list of weak references to its immediate\n subclasses. This method returns a list of all those references\n still alive. Example:\n\n >>> int.__subclasses__()\n []\n\n-[ Footnotes ]-\n\n[1] Additional information on these special methods may be found\n in the Python Reference Manual (*Basic customization*).\n\n[2] As a consequence, the list "[1, 2]" is considered equal to\n "[1.0, 2.0]", and similarly for tuples.\n\n[3] They must have since the parser can\'t tell the type of the\n operands.\n\n[4] Cased characters are those with general category property\n being one of "Lu" (Letter, uppercase), "Ll" (Letter, lowercase),\n or "Lt" (Letter, titlecase).\n\n[5] To format only a tuple you should therefore provide a\n singleton tuple whose only element is the tuple to be formatted.\n', - 'specialnames': '\nSpecial method names\n********************\n\nA class can implement certain operations that are invoked by special\nsyntax (such as arithmetic operations or subscripting and slicing) by\ndefining methods with special names. This is Python\'s approach to\n*operator overloading*, allowing classes to define their own behavior\nwith respect to language operators. For instance, if a class defines\na method named "__getitem__()", and "x" is an instance of this class,\nthen "x[i]" is roughly equivalent to "type(x).__getitem__(x, i)".\nExcept where mentioned, attempts to execute an operation raise an\nexception when no appropriate method is defined (typically\n"AttributeError" or "TypeError").\n\nWhen implementing a class that emulates any built-in type, it is\nimportant that the emulation only be implemented to the degree that it\nmakes sense for the object being modelled. For example, some\nsequences may work well with retrieval of individual elements, but\nextracting a slice may not make sense. (One example of this is the\n"NodeList" interface in the W3C\'s Document Object Model.)\n\n\nBasic customization\n===================\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. "__new__()" is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of "__new__()" should be the new object instance (usually an\n instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s "__new__()" method using\n "super(currentclass, cls).__new__(cls[, ...])" with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If "__new__()" returns an instance of *cls*, then the new\n instance\'s "__init__()" method will be invoked like\n "__init__(self[, ...])", where *self* is the new instance and the\n remaining arguments are the same as were passed to "__new__()".\n\n If "__new__()" does not return an instance of *cls*, then the new\n instance\'s "__init__()" method will not be invoked.\n\n "__new__()" is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n "__init__()" method, the derived class\'s "__init__()" method, if\n any, must explicitly call it to ensure proper initialization of the\n base class part of the instance; for example:\n "BaseClass.__init__(self, [args...])". As a special constraint on\n constructors, no value may be returned; doing so will cause a\n "TypeError" to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a "__del__()" method, the\n derived class\'s "__del__()" method, if any, must explicitly call it\n to ensure proper deletion of the base class part of the instance.\n Note that it is possible (though not recommended!) for the\n "__del__()" method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n "__del__()" methods are called for objects that still exist when\n the interpreter exits.\n\n Note: "del x" doesn\'t directly call "x.__del__()" --- the former\n decrements the reference count for "x" by one, and the latter is\n only called when "x"\'s reference count reaches zero. Some common\n situations that may prevent the reference count of an object from\n going to zero include: circular references between objects (e.g.,\n a doubly-linked list or a tree data structure with parent and\n child pointers); a reference to the object on the stack frame of\n a function that caught an exception (the traceback stored in\n "sys.exc_info()[2]" keeps the stack frame alive); or a reference\n to the object on the stack frame that raised an unhandled\n exception in interactive mode (the traceback stored in\n "sys.last_traceback" keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing "None" in\n "sys.last_traceback". Circular references which are garbage are\n detected and cleaned up when the cyclic garbage collector is\n enabled (it\'s on by default). Refer to the documentation for the\n "gc" module for more information about this topic.\n\n Warning: Due to the precarious circumstances under which\n "__del__()" methods are invoked, exceptions that occur during\n their execution are ignored, and a warning is printed to\n "sys.stderr" instead. Also, when "__del__()" is invoked in\n response to a module being deleted (e.g., when execution of the\n program is done), other globals referenced by the "__del__()"\n method may already have been deleted or in the process of being\n torn down (e.g. the import machinery shutting down). For this\n reason, "__del__()" methods should do the absolute minimum needed\n to maintain external invariants. Starting with version 1.5,\n Python guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the "__del__()" method is called.\n\nobject.__repr__(self)\n\n Called by the "repr()" built-in function to compute the "official"\n string representation of an object. If at all possible, this\n should look like a valid Python expression that could be used to\n recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n "<...some useful description...>" should be returned. The return\n value must be a string object. If a class defines "__repr__()" but\n not "__str__()", then "__repr__()" is also used when an "informal"\n string representation of instances of that class is required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by "str(object)" and the built-in functions "format()" and\n "print()" to compute the "informal" or nicely printable string\n representation of an object. The return value must be a *string*\n object.\n\n This method differs from "object.__repr__()" in that there is no\n expectation that "__str__()" return a valid Python expression: a\n more convenient or concise representation can be used.\n\n The default implementation defined by the built-in type "object"\n calls "object.__repr__()".\n\nobject.__bytes__(self)\n\n Called by "bytes()" to compute a byte-string representation of an\n object. This should return a "bytes" object.\n\nobject.__format__(self, format_spec)\n\n Called by the "format()" built-in function (and by extension, the\n "str.format()" method of class "str") to produce a "formatted"\n string representation of an object. The "format_spec" argument is a\n string that contains a description of the formatting options\n desired. The interpretation of the "format_spec" argument is up to\n the type implementing "__format__()", however most classes will\n either delegate formatting to one of the built-in types, or use a\n similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: "xy" calls\n "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n\n A rich comparison method may return the singleton "NotImplemented"\n if it does not implement the operation for a given pair of\n arguments. By convention, "False" and "True" are returned for a\n successful comparison. However, these methods can return any value,\n so if the comparison operator is used in a Boolean context (e.g.,\n in the condition of an "if" statement), Python will call "bool()"\n on the value to determine if the result is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of "x==y" does not imply that "x!=y" is false.\n Accordingly, when defining "__eq__()", one should also define\n "__ne__()" so that the operators will behave as expected. See the\n paragraph on "__hash__()" for some important notes on creating\n *hashable* objects which support custom comparison operations and\n are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, "__lt__()" and "__gt__()" are each other\'s\n reflection, "__le__()" and "__ge__()" are each other\'s reflection,\n and "__eq__()" and "__ne__()" are their own reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see "functools.total_ordering()".\n\nobject.__hash__(self)\n\n Called by built-in function "hash()" and for operations on members\n of hashed collections including "set", "frozenset", and "dict".\n "__hash__()" should return an integer. The only required property\n is that objects which compare equal have the same hash value; it is\n advised to somehow mix together (e.g. using exclusive or) the hash\n values for the components of the object that also play a part in\n comparison of objects.\n\n Note: "hash()" truncates the value returned from an object\'s\n custom "__hash__()" method to the size of a "Py_ssize_t". This\n is typically 8 bytes on 64-bit builds and 4 bytes on 32-bit\n builds. If an object\'s "__hash__()" must interoperate on builds\n of different bit sizes, be sure to check the width on all\n supported builds. An easy way to do this is with "python -c\n "import sys; print(sys.hash_info.width)""\n\n If a class does not define an "__eq__()" method it should not\n define a "__hash__()" operation either; if it defines "__eq__()"\n but not "__hash__()", its instances will not be usable as items in\n hashable collections. If a class defines mutable objects and\n implements an "__eq__()" method, it should not implement\n "__hash__()", since the implementation of hashable collections\n requires that a key\'s hash value is immutable (if the object\'s hash\n value changes, it will be in the wrong hash bucket).\n\n User-defined classes have "__eq__()" and "__hash__()" methods by\n default; with them, all objects compare unequal (except with\n themselves) and "x.__hash__()" returns an appropriate value such\n that "x == y" implies both that "x is y" and "hash(x) == hash(y)".\n\n A class that overrides "__eq__()" and does not define "__hash__()"\n will have its "__hash__()" implicitly set to "None". When the\n "__hash__()" method of a class is "None", instances of the class\n will raise an appropriate "TypeError" when a program attempts to\n retrieve their hash value, and will also be correctly identified as\n unhashable when checking "isinstance(obj, collections.Hashable").\n\n If a class that overrides "__eq__()" needs to retain the\n implementation of "__hash__()" from a parent class, the interpreter\n must be told this explicitly by setting "__hash__ =\n .__hash__".\n\n If a class that does not override "__eq__()" wishes to suppress\n hash support, it should include "__hash__ = None" in the class\n definition. A class which defines its own "__hash__()" that\n explicitly raises a "TypeError" would be incorrectly identified as\n hashable by an "isinstance(obj, collections.Hashable)" call.\n\n Note: By default, the "__hash__()" values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the iteration order of\n dicts, sets and other mappings. Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also "PYTHONHASHSEED".\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n "bool()"; should return "False" or "True". When this method is not\n defined, "__len__()" is called, if it is defined, and the object is\n considered true if its result is nonzero. If a class defines\n neither "__len__()" nor "__bool__()", all its instances are\n considered true.\n\n\nCustomizing attribute access\n============================\n\nThe following methods can be defined to customize the meaning of\nattribute access (use of, assignment to, or deletion of "x.name") for\nclass instances.\n\nobject.__getattr__(self, name)\n\n Called when an attribute lookup has not found the attribute in the\n usual places (i.e. it is not an instance attribute nor is it found\n in the class tree for "self"). "name" is the attribute name. This\n method should return the (computed) attribute value or raise an\n "AttributeError" exception.\n\n Note that if the attribute is found through the normal mechanism,\n "__getattr__()" is not called. (This is an intentional asymmetry\n between "__getattr__()" and "__setattr__()".) This is done both for\n efficiency reasons and because otherwise "__getattr__()" would have\n no way to access other attributes of the instance. Note that at\n least for instance variables, you can fake total control by not\n inserting any values in the instance attribute dictionary (but\n instead inserting them in another object). See the\n "__getattribute__()" method below for a way to actually get total\n control over attribute access.\n\nobject.__getattribute__(self, name)\n\n Called unconditionally to implement attribute accesses for\n instances of the class. If the class also defines "__getattr__()",\n the latter will not be called unless "__getattribute__()" either\n calls it explicitly or raises an "AttributeError". This method\n should return the (computed) attribute value or raise an\n "AttributeError" exception. In order to avoid infinite recursion in\n this method, its implementation should always call the base class\n method with the same name to access any attributes it needs, for\n example, "object.__getattribute__(self, name)".\n\n Note: This method may still be bypassed when looking up special\n methods as the result of implicit invocation via language syntax\n or built-in functions. See *Special method lookup*.\n\nobject.__setattr__(self, name, value)\n\n Called when an attribute assignment is attempted. This is called\n instead of the normal mechanism (i.e. store the value in the\n instance dictionary). *name* is the attribute name, *value* is the\n value to be assigned to it.\n\n If "__setattr__()" wants to assign to an instance attribute, it\n should call the base class method with the same name, for example,\n "object.__setattr__(self, name, value)".\n\nobject.__delattr__(self, name)\n\n Like "__setattr__()" but for attribute deletion instead of\n assignment. This should only be implemented if "del obj.name" is\n meaningful for the object.\n\nobject.__dir__(self)\n\n Called when "dir()" is called on the object. A sequence must be\n returned. "dir()" converts the returned sequence to a list and\n sorts it.\n\n\nImplementing Descriptors\n------------------------\n\nThe following methods only apply when an instance of the class\ncontaining the method (a so-called *descriptor* class) appears in an\n*owner* class (the descriptor must be in either the owner\'s class\ndictionary or in the class dictionary for one of its parents). In the\nexamples below, "the attribute" refers to the attribute whose name is\nthe key of the property in the owner class\' "__dict__".\n\nobject.__get__(self, instance, owner)\n\n Called to get the attribute of the owner class (class attribute\n access) or of an instance of that class (instance attribute\n access). *owner* is always the owner class, while *instance* is the\n instance that the attribute was accessed through, or "None" when\n the attribute is accessed through the *owner*. This method should\n return the (computed) attribute value or raise an "AttributeError"\n exception.\n\nobject.__set__(self, instance, value)\n\n Called to set the attribute on an instance *instance* of the owner\n class to a new value, *value*.\n\nobject.__delete__(self, instance)\n\n Called to delete the attribute on an instance *instance* of the\n owner class.\n\n\nInvoking Descriptors\n--------------------\n\nIn general, a descriptor is an object attribute with "binding\nbehavior", one whose attribute access has been overridden by methods\nin the descriptor protocol: "__get__()", "__set__()", and\n"__delete__()". If any of those methods are defined for an object, it\nis said to be a descriptor.\n\nThe default behavior for attribute access is to get, set, or delete\nthe attribute from an object\'s dictionary. For instance, "a.x" has a\nlookup chain starting with "a.__dict__[\'x\']", then\n"type(a).__dict__[\'x\']", and continuing through the base classes of\n"type(a)" excluding metaclasses.\n\nHowever, if the looked-up value is an object defining one of the\ndescriptor methods, then Python may override the default behavior and\ninvoke the descriptor method instead. Where this occurs in the\nprecedence chain depends on which descriptor methods were defined and\nhow they were called.\n\nThe starting point for descriptor invocation is a binding, "a.x". How\nthe arguments are assembled depends on "a":\n\nDirect Call\n The simplest and least common call is when user code directly\n invokes a descriptor method: "x.__get__(a)".\n\nInstance Binding\n If binding to an object instance, "a.x" is transformed into the\n call: "type(a).__dict__[\'x\'].__get__(a, type(a))".\n\nClass Binding\n If binding to a class, "A.x" is transformed into the call:\n "A.__dict__[\'x\'].__get__(None, A)".\n\nSuper Binding\n If "a" is an instance of "super", then the binding "super(B,\n obj).m()" searches "obj.__class__.__mro__" for the base class "A"\n immediately preceding "B" and then invokes the descriptor with the\n call: "A.__dict__[\'m\'].__get__(obj, obj.__class__)".\n\nFor instance bindings, the precedence of descriptor invocation depends\non the which descriptor methods are defined. A descriptor can define\nany combination of "__get__()", "__set__()" and "__delete__()". If it\ndoes not define "__get__()", then accessing the attribute will return\nthe descriptor object itself unless there is a value in the object\'s\ninstance dictionary. If the descriptor defines "__set__()" and/or\n"__delete__()", it is a data descriptor; if it defines neither, it is\na non-data descriptor. Normally, data descriptors define both\n"__get__()" and "__set__()", while non-data descriptors have just the\n"__get__()" method. Data descriptors with "__set__()" and "__get__()"\ndefined always override a redefinition in an instance dictionary. In\ncontrast, non-data descriptors can be overridden by instances.\n\nPython methods (including "staticmethod()" and "classmethod()") are\nimplemented as non-data descriptors. Accordingly, instances can\nredefine and override methods. This allows individual instances to\nacquire behaviors that differ from other instances of the same class.\n\nThe "property()" function is implemented as a data descriptor.\nAccordingly, instances cannot override the behavior of a property.\n\n\n__slots__\n---------\n\nBy default, instances of classes have a dictionary for attribute\nstorage. This wastes space for objects having very few instance\nvariables. The space consumption can become acute when creating large\nnumbers of instances.\n\nThe default can be overridden by defining *__slots__* in a class\ndefinition. The *__slots__* declaration takes a sequence of instance\nvariables and reserves just enough space in each instance to hold a\nvalue for each variable. Space is saved because *__dict__* is not\ncreated for each instance.\n\nobject.__slots__\n\n This class variable can be assigned a string, iterable, or sequence\n of strings with variable names used by instances. If defined in a\n class, *__slots__* reserves space for the declared variables and\n prevents the automatic creation of *__dict__* and *__weakref__* for\n each instance.\n\n\nNotes on using *__slots__*\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n* When inheriting from a class without *__slots__*, the *__dict__*\n attribute of that class will always be accessible, so a *__slots__*\n definition in the subclass is meaningless.\n\n* Without a *__dict__* variable, instances cannot be assigned new\n variables not listed in the *__slots__* definition. Attempts to\n assign to an unlisted variable name raises "AttributeError". If\n dynamic assignment of new variables is desired, then add\n "\'__dict__\'" to the sequence of strings in the *__slots__*\n declaration.\n\n* Without a *__weakref__* variable for each instance, classes\n defining *__slots__* do not support weak references to its\n instances. If weak reference support is needed, then add\n "\'__weakref__\'" to the sequence of strings in the *__slots__*\n declaration.\n\n* *__slots__* are implemented at the class level by creating\n descriptors (*Implementing Descriptors*) for each variable name. As\n a result, class attributes cannot be used to set default values for\n instance variables defined by *__slots__*; otherwise, the class\n attribute would overwrite the descriptor assignment.\n\n* The action of a *__slots__* declaration is limited to the class\n where it is defined. As a result, subclasses will have a *__dict__*\n unless they also define *__slots__* (which must only contain names\n of any *additional* slots).\n\n* If a class defines a slot also defined in a base class, the\n instance variable defined by the base class slot is inaccessible\n (except by retrieving its descriptor directly from the base class).\n This renders the meaning of the program undefined. In the future, a\n check may be added to prevent this.\n\n* Nonempty *__slots__* does not work for classes derived from\n "variable-length" built-in types such as "int", "bytes" and "tuple".\n\n* Any non-string iterable may be assigned to *__slots__*. Mappings\n may also be used; however, in the future, special meaning may be\n assigned to the values corresponding to each key.\n\n* *__class__* assignment works only if both classes have the same\n *__slots__*.\n\n\nCustomizing class creation\n==========================\n\nBy default, classes are constructed using "type()". The class body is\nexecuted in a new namespace and the class name is bound locally to the\nresult of "type(name, bases, namespace)".\n\nThe class creation process can be customised by passing the\n"metaclass" keyword argument in the class definition line, or by\ninheriting from an existing class that included such an argument. In\nthe following example, both "MyClass" and "MySubclass" are instances\nof "Meta":\n\n class Meta(type):\n pass\n\n class MyClass(metaclass=Meta):\n pass\n\n class MySubclass(MyClass):\n pass\n\nAny other keyword arguments that are specified in the class definition\nare passed through to all metaclass operations described below.\n\nWhen a class definition is executed, the following steps occur:\n\n* the appropriate metaclass is determined\n\n* the class namespace is prepared\n\n* the class body is executed\n\n* the class object is created\n\n\nDetermining the appropriate metaclass\n-------------------------------------\n\nThe appropriate metaclass for a class definition is determined as\nfollows:\n\n* if no bases and no explicit metaclass are given, then "type()" is\n used\n\n* if an explicit metaclass is given and it is *not* an instance of\n "type()", then it is used directly as the metaclass\n\n* if an instance of "type()" is given as the explicit metaclass, or\n bases are defined, then the most derived metaclass is used\n\nThe most derived metaclass is selected from the explicitly specified\nmetaclass (if any) and the metaclasses (i.e. "type(cls)") of all\nspecified base classes. The most derived metaclass is one which is a\nsubtype of *all* of these candidate metaclasses. If none of the\ncandidate metaclasses meets that criterion, then the class definition\nwill fail with "TypeError".\n\n\nPreparing the class namespace\n-----------------------------\n\nOnce the appropriate metaclass has been identified, then the class\nnamespace is prepared. If the metaclass has a "__prepare__" attribute,\nit is called as "namespace = metaclass.__prepare__(name, bases,\n**kwds)" (where the additional keyword arguments, if any, come from\nthe class definition).\n\nIf the metaclass has no "__prepare__" attribute, then the class\nnamespace is initialised as an empty "dict()" instance.\n\nSee also: **PEP 3115** - Metaclasses in Python 3000\n\n Introduced the "__prepare__" namespace hook\n\n\nExecuting the class body\n------------------------\n\nThe class body is executed (approximately) as "exec(body, globals(),\nnamespace)". The key difference from a normal call to "exec()" is that\nlexical scoping allows the class body (including any methods) to\nreference names from the current and outer scopes when the class\ndefinition occurs inside a function.\n\nHowever, even when the class definition occurs inside the function,\nmethods defined inside the class still cannot see names defined at the\nclass scope. Class variables must be accessed through the first\nparameter of instance or class methods, and cannot be accessed at all\nfrom static methods.\n\n\nCreating the class object\n-------------------------\n\nOnce the class namespace has been populated by executing the class\nbody, the class object is created by calling "metaclass(name, bases,\nnamespace, **kwds)" (the additional keywords passed here are the same\nas those passed to "__prepare__").\n\nThis class object is the one that will be referenced by the zero-\nargument form of "super()". "__class__" is an implicit closure\nreference created by the compiler if any methods in a class body refer\nto either "__class__" or "super". This allows the zero argument form\nof "super()" to correctly identify the class being defined based on\nlexical scoping, while the class or instance that was used to make the\ncurrent call is identified based on the first argument passed to the\nmethod.\n\nAfter the class object is created, it is passed to the class\ndecorators included in the class definition (if any) and the resulting\nobject is bound in the local namespace as the defined class.\n\nSee also: **PEP 3135** - New super\n\n Describes the implicit "__class__" closure reference\n\n\nMetaclass example\n-----------------\n\nThe potential uses for metaclasses are boundless. Some ideas that have\nbeen explored include logging, interface checking, automatic\ndelegation, automatic property creation, proxies, frameworks, and\nautomatic resource locking/synchronization.\n\nHere is an example of a metaclass that uses an\n"collections.OrderedDict" to remember the order that class members\nwere defined:\n\n class OrderedClass(type):\n\n @classmethod\n def __prepare__(metacls, name, bases, **kwds):\n return collections.OrderedDict()\n\n def __new__(cls, name, bases, namespace, **kwds):\n result = type.__new__(cls, name, bases, dict(namespace))\n result.members = tuple(namespace)\n return result\n\n class A(metaclass=OrderedClass):\n def one(self): pass\n def two(self): pass\n def three(self): pass\n def four(self): pass\n\n >>> A.members\n (\'__module__\', \'one\', \'two\', \'three\', \'four\')\n\nWhen the class definition for *A* gets executed, the process begins\nwith calling the metaclass\'s "__prepare__()" method which returns an\nempty "collections.OrderedDict". That mapping records the methods and\nattributes of *A* as they are defined within the body of the class\nstatement. Once those definitions are executed, the ordered dictionary\nis fully populated and the metaclass\'s "__new__()" method gets\ninvoked. That method builds the new type and it saves the ordered\ndictionary keys in an attribute called "members".\n\n\nCustomizing instance and subclass checks\n========================================\n\nThe following methods are used to override the default behavior of the\n"isinstance()" and "issubclass()" built-in functions.\n\nIn particular, the metaclass "abc.ABCMeta" implements these methods in\norder to allow the addition of Abstract Base Classes (ABCs) as\n"virtual base classes" to any class or type (including built-in\ntypes), including other ABCs.\n\nclass.__instancecheck__(self, instance)\n\n Return true if *instance* should be considered a (direct or\n indirect) instance of *class*. If defined, called to implement\n "isinstance(instance, class)".\n\nclass.__subclasscheck__(self, subclass)\n\n Return true if *subclass* should be considered a (direct or\n indirect) subclass of *class*. If defined, called to implement\n "issubclass(subclass, class)".\n\nNote that these methods are looked up on the type (metaclass) of a\nclass. They cannot be defined as class methods in the actual class.\nThis is consistent with the lookup of special methods that are called\non instances, only in this case the instance is itself a class.\n\nSee also: **PEP 3119** - Introducing Abstract Base Classes\n\n Includes the specification for customizing "isinstance()" and\n "issubclass()" behavior through "__instancecheck__()" and\n "__subclasscheck__()", with motivation for this functionality in\n the context of adding Abstract Base Classes (see the "abc"\n module) to the language.\n\n\nEmulating callable objects\n==========================\n\nobject.__call__(self[, args...])\n\n Called when the instance is "called" as a function; if this method\n is defined, "x(arg1, arg2, ...)" is a shorthand for\n "x.__call__(arg1, arg2, ...)".\n\n\nEmulating container types\n=========================\n\nThe following methods can be defined to implement container objects.\nContainers usually are sequences (such as lists or tuples) or mappings\n(like dictionaries), but can represent other containers as well. The\nfirst set of methods is used either to emulate a sequence or to\nemulate a mapping; the difference is that for a sequence, the\nallowable keys should be the integers *k* for which "0 <= k < N" where\n*N* is the length of the sequence, or slice objects, which define a\nrange of items. It is also recommended that mappings provide the\nmethods "keys()", "values()", "items()", "get()", "clear()",\n"setdefault()", "pop()", "popitem()", "copy()", and "update()"\nbehaving similar to those for Python\'s standard dictionary objects.\nThe "collections" module provides a "MutableMapping" abstract base\nclass to help create those methods from a base set of "__getitem__()",\n"__setitem__()", "__delitem__()", and "keys()". Mutable sequences\nshould provide methods "append()", "count()", "index()", "extend()",\n"insert()", "pop()", "remove()", "reverse()" and "sort()", like Python\nstandard list objects. Finally, sequence types should implement\naddition (meaning concatenation) and multiplication (meaning\nrepetition) by defining the methods "__add__()", "__radd__()",\n"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" described\nbelow; they should not define other numerical operators. It is\nrecommended that both mappings and sequences implement the\n"__contains__()" method to allow efficient use of the "in" operator;\nfor mappings, "in" should search the mapping\'s keys; for sequences, it\nshould search through the values. It is further recommended that both\nmappings and sequences implement the "__iter__()" method to allow\nefficient iteration through the container; for mappings, "__iter__()"\nshould be the same as "keys()"; for sequences, it should iterate\nthrough the values.\n\nobject.__len__(self)\n\n Called to implement the built-in function "len()". Should return\n the length of the object, an integer ">=" 0. Also, an object that\n doesn\'t define a "__bool__()" method and whose "__len__()" method\n returns zero is considered to be false in a Boolean context.\n\nobject.__length_hint__(self)\n\n Called to implement "operator.length_hint()". Should return an\n estimated length for the object (which may be greater or less than\n the actual length). The length must be an integer ">=" 0. This\n method is purely an optimization and is never required for\n correctness.\n\n New in version 3.4.\n\nNote: Slicing is done exclusively with the following three methods.\n A call like\n\n a[1:2] = b\n\n is translated to\n\n a[slice(1, 2, None)] = b\n\n and so forth. Missing slice items are always filled in with "None".\n\nobject.__getitem__(self, key)\n\n Called to implement evaluation of "self[key]". For sequence types,\n the accepted keys should be integers and slice objects. Note that\n the special interpretation of negative indexes (if the class wishes\n to emulate a sequence type) is up to the "__getitem__()" method. If\n *key* is of an inappropriate type, "TypeError" may be raised; if of\n a value outside the set of indexes for the sequence (after any\n special interpretation of negative values), "IndexError" should be\n raised. For mapping types, if *key* is missing (not in the\n container), "KeyError" should be raised.\n\n Note: "for" loops expect that an "IndexError" will be raised for\n illegal indexes to allow proper detection of the end of the\n sequence.\n\nobject.__setitem__(self, key, value)\n\n Called to implement assignment to "self[key]". Same note as for\n "__getitem__()". This should only be implemented for mappings if\n the objects support changes to the values for keys, or if new keys\n can be added, or for sequences if elements can be replaced. The\n same exceptions should be raised for improper *key* values as for\n the "__getitem__()" method.\n\nobject.__delitem__(self, key)\n\n Called to implement deletion of "self[key]". Same note as for\n "__getitem__()". This should only be implemented for mappings if\n the objects support removal of keys, or for sequences if elements\n can be removed from the sequence. The same exceptions should be\n raised for improper *key* values as for the "__getitem__()" method.\n\nobject.__iter__(self)\n\n This method is called when an iterator is required for a container.\n This method should return a new iterator object that can iterate\n over all the objects in the container. For mappings, it should\n iterate over the keys of the container, and should also be made\n available as the method "keys()".\n\n Iterator objects also need to implement this method; they are\n required to return themselves. For more information on iterator\n objects, see *Iterator Types*.\n\nobject.__reversed__(self)\n\n Called (if present) by the "reversed()" built-in to implement\n reverse iteration. It should return a new iterator object that\n iterates over all the objects in the container in reverse order.\n\n If the "__reversed__()" method is not provided, the "reversed()"\n built-in will fall back to using the sequence protocol ("__len__()"\n and "__getitem__()"). Objects that support the sequence protocol\n should only provide "__reversed__()" if they can provide an\n implementation that is more efficient than the one provided by\n "reversed()".\n\nThe membership test operators ("in" and "not in") are normally\nimplemented as an iteration through a sequence. However, container\nobjects can supply the following special method with a more efficient\nimplementation, which also does not require the object be a sequence.\n\nobject.__contains__(self, item)\n\n Called to implement membership test operators. Should return true\n if *item* is in *self*, false otherwise. For mapping objects, this\n should consider the keys of the mapping rather than the values or\n the key-item pairs.\n\n For objects that don\'t define "__contains__()", the membership test\n first tries iteration via "__iter__()", then the old sequence\n iteration protocol via "__getitem__()", see *this section in the\n language reference*.\n\n\nEmulating numeric types\n=======================\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|"). For instance, to evaluate the\n expression "x + y", where *x* is an instance of a class that has an\n "__add__()" method, "x.__add__(y)" is called. The "__divmod__()"\n method should be the equivalent to using "__floordiv__()" and\n "__mod__()"; it should not be related to "__truediv__()". Note\n that "__pow__()" should be defined to accept an optional third\n argument if the ternary version of the built-in "pow()" function is\n to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return "NotImplemented".\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|") with reflected (swapped) operands.\n These functions are only called if the left operand does not\n support the corresponding operation and the operands are of\n different types. [2] For instance, to evaluate the expression "x -\n y", where *y* is an instance of a class that has an "__rsub__()"\n method, "y.__rsub__(x)" is called if "x.__sub__(y)" returns\n *NotImplemented*.\n\n Note that ternary "pow()" will not try calling "__rpow__()" (the\n coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left\n operand\'s type and that subclass provides the reflected method\n for the operation, this method will be called before the left\n operand\'s non-reflected method. This behavior allows subclasses\n to override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments ("+=", "-=", "*=", "/=", "//=", "%=", "**=", "<<=",\n ">>=", "&=", "^=", "|="). These methods should attempt to do the\n operation in-place (modifying *self*) and return the result (which\n could be, but does not have to be, *self*). If a specific method\n is not defined, the augmented assignment falls back to the normal\n methods. For instance, to execute the statement "x += y", where\n *x* is an instance of a class that has an "__iadd__()" method,\n "x.__iadd__(y)" is called. If *x* is an instance of a class that\n does not define a "__iadd__()" method, "x.__add__(y)" and\n "y.__radd__(x)" are considered, as with the evaluation of "x + y".\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations ("-", "+",\n "abs()" and "~").\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions "complex()", "int()",\n "float()" and "round()". Should return a value of the appropriate\n type.\n\nobject.__index__(self)\n\n Called to implement "operator.index()", and whenever Python needs\n to losslessly convert the numeric object to an integer object (such\n as in slicing, or in the built-in "bin()", "hex()" and "oct()"\n functions). Presence of this method indicates that the numeric\n object is an integer type. Must return an integer.\n\n Note: When "__index__()" is defined, "__int__()" should also be\n defined, and both shuld return the same value, in order to have a\n coherent integer type class.\n\n\nWith Statement Context Managers\n===============================\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a "with" statement. The context manager\nhandles the entry into, and the exit from, the desired runtime context\nfor the execution of the block of code. Context managers are normally\ninvoked using the "with" statement (described in section *The with\nstatement*), but can also be used by directly invoking their methods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The "with"\n statement will bind this method\'s return value to the target(s)\n specified in the "as" clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be "None".\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that "__exit__()" methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also: **PEP 0343** - The "with" statement\n\n The specification, background, and examples for the Python "with"\n statement.\n\n\nSpecial method lookup\n=====================\n\nFor custom classes, implicit invocations of special methods are only\nguaranteed to work correctly if defined on an object\'s type, not in\nthe object\'s instance dictionary. That behaviour is the reason why\nthe following code raises an exception:\n\n >>> class C:\n ... pass\n ...\n >>> c = C()\n >>> c.__len__ = lambda: 5\n >>> len(c)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: object of type \'C\' has no len()\n\nThe rationale behind this behaviour lies with a number of special\nmethods such as "__hash__()" and "__repr__()" that are implemented by\nall objects, including type objects. If the implicit lookup of these\nmethods used the conventional lookup process, they would fail when\ninvoked on the type object itself:\n\n >>> 1 .__hash__() == hash(1)\n True\n >>> int.__hash__() == hash(int)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: descriptor \'__hash__\' of \'int\' object needs an argument\n\nIncorrectly attempting to invoke an unbound method of a class in this\nway is sometimes referred to as \'metaclass confusion\', and is avoided\nby bypassing the instance when looking up special methods:\n\n >>> type(1).__hash__(1) == hash(1)\n True\n >>> type(int).__hash__(int) == hash(int)\n True\n\nIn addition to bypassing any instance attributes in the interest of\ncorrectness, implicit special method lookup generally also bypasses\nthe "__getattribute__()" method even of the object\'s metaclass:\n\n >>> class Meta(type):\n ... def __getattribute__(*args):\n ... print("Metaclass getattribute invoked")\n ... return type.__getattribute__(*args)\n ...\n >>> class C(object, metaclass=Meta):\n ... def __len__(self):\n ... return 10\n ... def __getattribute__(*args):\n ... print("Class getattribute invoked")\n ... return object.__getattribute__(*args)\n ...\n >>> c = C()\n >>> c.__len__() # Explicit lookup via instance\n Class getattribute invoked\n 10\n >>> type(c).__len__(c) # Explicit lookup via type\n Metaclass getattribute invoked\n 10\n >>> len(c) # Implicit lookup\n 10\n\nBypassing the "__getattribute__()" machinery in this fashion provides\nsignificant scope for speed optimisations within the interpreter, at\nthe cost of some flexibility in the handling of special methods (the\nspecial method *must* be set on the class object itself in order to be\nconsistently invoked by the interpreter).\n\n-[ Footnotes ]-\n\n[1] It *is* possible in some cases to change an object\'s type,\n under certain controlled conditions. It generally isn\'t a good\n idea though, since it can lead to some very strange behaviour if\n it is handled incorrectly.\n\n[2] For operands of the same type, it is assumed that if the non-\n reflected method (such as "__add__()") fails the operation is not\n supported, which is why the reflected method is not called.\n', + 'specialnames': '\nSpecial method names\n********************\n\nA class can implement certain operations that are invoked by special\nsyntax (such as arithmetic operations or subscripting and slicing) by\ndefining methods with special names. This is Python\'s approach to\n*operator overloading*, allowing classes to define their own behavior\nwith respect to language operators. For instance, if a class defines\na method named "__getitem__()", and "x" is an instance of this class,\nthen "x[i]" is roughly equivalent to "type(x).__getitem__(x, i)".\nExcept where mentioned, attempts to execute an operation raise an\nexception when no appropriate method is defined (typically\n"AttributeError" or "TypeError").\n\nWhen implementing a class that emulates any built-in type, it is\nimportant that the emulation only be implemented to the degree that it\nmakes sense for the object being modelled. For example, some\nsequences may work well with retrieval of individual elements, but\nextracting a slice may not make sense. (One example of this is the\n"NodeList" interface in the W3C\'s Document Object Model.)\n\n\nBasic customization\n===================\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. "__new__()" is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of "__new__()" should be the new object instance (usually an\n instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s "__new__()" method using\n "super(currentclass, cls).__new__(cls[, ...])" with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If "__new__()" returns an instance of *cls*, then the new\n instance\'s "__init__()" method will be invoked like\n "__init__(self[, ...])", where *self* is the new instance and the\n remaining arguments are the same as were passed to "__new__()".\n\n If "__new__()" does not return an instance of *cls*, then the new\n instance\'s "__init__()" method will not be invoked.\n\n "__new__()" is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n "__init__()" method, the derived class\'s "__init__()" method, if\n any, must explicitly call it to ensure proper initialization of the\n base class part of the instance; for example:\n "BaseClass.__init__(self, [args...])". As a special constraint on\n constructors, no value may be returned; doing so will cause a\n "TypeError" to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a "__del__()" method, the\n derived class\'s "__del__()" method, if any, must explicitly call it\n to ensure proper deletion of the base class part of the instance.\n Note that it is possible (though not recommended!) for the\n "__del__()" method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n "__del__()" methods are called for objects that still exist when\n the interpreter exits.\n\n Note: "del x" doesn\'t directly call "x.__del__()" --- the former\n decrements the reference count for "x" by one, and the latter is\n only called when "x"\'s reference count reaches zero. Some common\n situations that may prevent the reference count of an object from\n going to zero include: circular references between objects (e.g.,\n a doubly-linked list or a tree data structure with parent and\n child pointers); a reference to the object on the stack frame of\n a function that caught an exception (the traceback stored in\n "sys.exc_info()[2]" keeps the stack frame alive); or a reference\n to the object on the stack frame that raised an unhandled\n exception in interactive mode (the traceback stored in\n "sys.last_traceback" keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing "None" in\n "sys.last_traceback". Circular references which are garbage are\n detected and cleaned up when the cyclic garbage collector is\n enabled (it\'s on by default). Refer to the documentation for the\n "gc" module for more information about this topic.\n\n Warning: Due to the precarious circumstances under which\n "__del__()" methods are invoked, exceptions that occur during\n their execution are ignored, and a warning is printed to\n "sys.stderr" instead. Also, when "__del__()" is invoked in\n response to a module being deleted (e.g., when execution of the\n program is done), other globals referenced by the "__del__()"\n method may already have been deleted or in the process of being\n torn down (e.g. the import machinery shutting down). For this\n reason, "__del__()" methods should do the absolute minimum needed\n to maintain external invariants. Starting with version 1.5,\n Python guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the "__del__()" method is called.\n\nobject.__repr__(self)\n\n Called by the "repr()" built-in function to compute the "official"\n string representation of an object. If at all possible, this\n should look like a valid Python expression that could be used to\n recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n "<...some useful description...>" should be returned. The return\n value must be a string object. If a class defines "__repr__()" but\n not "__str__()", then "__repr__()" is also used when an "informal"\n string representation of instances of that class is required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by "str(object)" and the built-in functions "format()" and\n "print()" to compute the "informal" or nicely printable string\n representation of an object. The return value must be a *string*\n object.\n\n This method differs from "object.__repr__()" in that there is no\n expectation that "__str__()" return a valid Python expression: a\n more convenient or concise representation can be used.\n\n The default implementation defined by the built-in type "object"\n calls "object.__repr__()".\n\nobject.__bytes__(self)\n\n Called by "bytes()" to compute a byte-string representation of an\n object. This should return a "bytes" object.\n\nobject.__format__(self, format_spec)\n\n Called by the "format()" built-in function (and by extension, the\n "str.format()" method of class "str") to produce a "formatted"\n string representation of an object. The "format_spec" argument is a\n string that contains a description of the formatting options\n desired. The interpretation of the "format_spec" argument is up to\n the type implementing "__format__()", however most classes will\n either delegate formatting to one of the built-in types, or use a\n similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\n Changed in version 3.4: The __format__ method of "object" itself\n raises a "TypeError" if passed any non-empty string.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: "xy" calls\n "x.__gt__(y)", and "x>=y" calls "x.__ge__(y)".\n\n A rich comparison method may return the singleton "NotImplemented"\n if it does not implement the operation for a given pair of\n arguments. By convention, "False" and "True" are returned for a\n successful comparison. However, these methods can return any value,\n so if the comparison operator is used in a Boolean context (e.g.,\n in the condition of an "if" statement), Python will call "bool()"\n on the value to determine if the result is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of "x==y" does not imply that "x!=y" is false.\n Accordingly, when defining "__eq__()", one should also define\n "__ne__()" so that the operators will behave as expected. See the\n paragraph on "__hash__()" for some important notes on creating\n *hashable* objects which support custom comparison operations and\n are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, "__lt__()" and "__gt__()" are each other\'s\n reflection, "__le__()" and "__ge__()" are each other\'s reflection,\n and "__eq__()" and "__ne__()" are their own reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see "functools.total_ordering()".\n\nobject.__hash__(self)\n\n Called by built-in function "hash()" and for operations on members\n of hashed collections including "set", "frozenset", and "dict".\n "__hash__()" should return an integer. The only required property\n is that objects which compare equal have the same hash value; it is\n advised to somehow mix together (e.g. using exclusive or) the hash\n values for the components of the object that also play a part in\n comparison of objects.\n\n Note: "hash()" truncates the value returned from an object\'s\n custom "__hash__()" method to the size of a "Py_ssize_t". This\n is typically 8 bytes on 64-bit builds and 4 bytes on 32-bit\n builds. If an object\'s "__hash__()" must interoperate on builds\n of different bit sizes, be sure to check the width on all\n supported builds. An easy way to do this is with "python -c\n "import sys; print(sys.hash_info.width)""\n\n If a class does not define an "__eq__()" method it should not\n define a "__hash__()" operation either; if it defines "__eq__()"\n but not "__hash__()", its instances will not be usable as items in\n hashable collections. If a class defines mutable objects and\n implements an "__eq__()" method, it should not implement\n "__hash__()", since the implementation of hashable collections\n requires that a key\'s hash value is immutable (if the object\'s hash\n value changes, it will be in the wrong hash bucket).\n\n User-defined classes have "__eq__()" and "__hash__()" methods by\n default; with them, all objects compare unequal (except with\n themselves) and "x.__hash__()" returns an appropriate value such\n that "x == y" implies both that "x is y" and "hash(x) == hash(y)".\n\n A class that overrides "__eq__()" and does not define "__hash__()"\n will have its "__hash__()" implicitly set to "None". When the\n "__hash__()" method of a class is "None", instances of the class\n will raise an appropriate "TypeError" when a program attempts to\n retrieve their hash value, and will also be correctly identified as\n unhashable when checking "isinstance(obj, collections.Hashable").\n\n If a class that overrides "__eq__()" needs to retain the\n implementation of "__hash__()" from a parent class, the interpreter\n must be told this explicitly by setting "__hash__ =\n .__hash__".\n\n If a class that does not override "__eq__()" wishes to suppress\n hash support, it should include "__hash__ = None" in the class\n definition. A class which defines its own "__hash__()" that\n explicitly raises a "TypeError" would be incorrectly identified as\n hashable by an "isinstance(obj, collections.Hashable)" call.\n\n Note: By default, the "__hash__()" values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the iteration order of\n dicts, sets and other mappings. Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also "PYTHONHASHSEED".\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n "bool()"; should return "False" or "True". When this method is not\n defined, "__len__()" is called, if it is defined, and the object is\n considered true if its result is nonzero. If a class defines\n neither "__len__()" nor "__bool__()", all its instances are\n considered true.\n\n\nCustomizing attribute access\n============================\n\nThe following methods can be defined to customize the meaning of\nattribute access (use of, assignment to, or deletion of "x.name") for\nclass instances.\n\nobject.__getattr__(self, name)\n\n Called when an attribute lookup has not found the attribute in the\n usual places (i.e. it is not an instance attribute nor is it found\n in the class tree for "self"). "name" is the attribute name. This\n method should return the (computed) attribute value or raise an\n "AttributeError" exception.\n\n Note that if the attribute is found through the normal mechanism,\n "__getattr__()" is not called. (This is an intentional asymmetry\n between "__getattr__()" and "__setattr__()".) This is done both for\n efficiency reasons and because otherwise "__getattr__()" would have\n no way to access other attributes of the instance. Note that at\n least for instance variables, you can fake total control by not\n inserting any values in the instance attribute dictionary (but\n instead inserting them in another object). See the\n "__getattribute__()" method below for a way to actually get total\n control over attribute access.\n\nobject.__getattribute__(self, name)\n\n Called unconditionally to implement attribute accesses for\n instances of the class. If the class also defines "__getattr__()",\n the latter will not be called unless "__getattribute__()" either\n calls it explicitly or raises an "AttributeError". This method\n should return the (computed) attribute value or raise an\n "AttributeError" exception. In order to avoid infinite recursion in\n this method, its implementation should always call the base class\n method with the same name to access any attributes it needs, for\n example, "object.__getattribute__(self, name)".\n\n Note: This method may still be bypassed when looking up special\n methods as the result of implicit invocation via language syntax\n or built-in functions. See *Special method lookup*.\n\nobject.__setattr__(self, name, value)\n\n Called when an attribute assignment is attempted. This is called\n instead of the normal mechanism (i.e. store the value in the\n instance dictionary). *name* is the attribute name, *value* is the\n value to be assigned to it.\n\n If "__setattr__()" wants to assign to an instance attribute, it\n should call the base class method with the same name, for example,\n "object.__setattr__(self, name, value)".\n\nobject.__delattr__(self, name)\n\n Like "__setattr__()" but for attribute deletion instead of\n assignment. This should only be implemented if "del obj.name" is\n meaningful for the object.\n\nobject.__dir__(self)\n\n Called when "dir()" is called on the object. A sequence must be\n returned. "dir()" converts the returned sequence to a list and\n sorts it.\n\n\nImplementing Descriptors\n------------------------\n\nThe following methods only apply when an instance of the class\ncontaining the method (a so-called *descriptor* class) appears in an\n*owner* class (the descriptor must be in either the owner\'s class\ndictionary or in the class dictionary for one of its parents). In the\nexamples below, "the attribute" refers to the attribute whose name is\nthe key of the property in the owner class\' "__dict__".\n\nobject.__get__(self, instance, owner)\n\n Called to get the attribute of the owner class (class attribute\n access) or of an instance of that class (instance attribute\n access). *owner* is always the owner class, while *instance* is the\n instance that the attribute was accessed through, or "None" when\n the attribute is accessed through the *owner*. This method should\n return the (computed) attribute value or raise an "AttributeError"\n exception.\n\nobject.__set__(self, instance, value)\n\n Called to set the attribute on an instance *instance* of the owner\n class to a new value, *value*.\n\nobject.__delete__(self, instance)\n\n Called to delete the attribute on an instance *instance* of the\n owner class.\n\n\nInvoking Descriptors\n--------------------\n\nIn general, a descriptor is an object attribute with "binding\nbehavior", one whose attribute access has been overridden by methods\nin the descriptor protocol: "__get__()", "__set__()", and\n"__delete__()". If any of those methods are defined for an object, it\nis said to be a descriptor.\n\nThe default behavior for attribute access is to get, set, or delete\nthe attribute from an object\'s dictionary. For instance, "a.x" has a\nlookup chain starting with "a.__dict__[\'x\']", then\n"type(a).__dict__[\'x\']", and continuing through the base classes of\n"type(a)" excluding metaclasses.\n\nHowever, if the looked-up value is an object defining one of the\ndescriptor methods, then Python may override the default behavior and\ninvoke the descriptor method instead. Where this occurs in the\nprecedence chain depends on which descriptor methods were defined and\nhow they were called.\n\nThe starting point for descriptor invocation is a binding, "a.x". How\nthe arguments are assembled depends on "a":\n\nDirect Call\n The simplest and least common call is when user code directly\n invokes a descriptor method: "x.__get__(a)".\n\nInstance Binding\n If binding to an object instance, "a.x" is transformed into the\n call: "type(a).__dict__[\'x\'].__get__(a, type(a))".\n\nClass Binding\n If binding to a class, "A.x" is transformed into the call:\n "A.__dict__[\'x\'].__get__(None, A)".\n\nSuper Binding\n If "a" is an instance of "super", then the binding "super(B,\n obj).m()" searches "obj.__class__.__mro__" for the base class "A"\n immediately preceding "B" and then invokes the descriptor with the\n call: "A.__dict__[\'m\'].__get__(obj, obj.__class__)".\n\nFor instance bindings, the precedence of descriptor invocation depends\non the which descriptor methods are defined. A descriptor can define\nany combination of "__get__()", "__set__()" and "__delete__()". If it\ndoes not define "__get__()", then accessing the attribute will return\nthe descriptor object itself unless there is a value in the object\'s\ninstance dictionary. If the descriptor defines "__set__()" and/or\n"__delete__()", it is a data descriptor; if it defines neither, it is\na non-data descriptor. Normally, data descriptors define both\n"__get__()" and "__set__()", while non-data descriptors have just the\n"__get__()" method. Data descriptors with "__set__()" and "__get__()"\ndefined always override a redefinition in an instance dictionary. In\ncontrast, non-data descriptors can be overridden by instances.\n\nPython methods (including "staticmethod()" and "classmethod()") are\nimplemented as non-data descriptors. Accordingly, instances can\nredefine and override methods. This allows individual instances to\nacquire behaviors that differ from other instances of the same class.\n\nThe "property()" function is implemented as a data descriptor.\nAccordingly, instances cannot override the behavior of a property.\n\n\n__slots__\n---------\n\nBy default, instances of classes have a dictionary for attribute\nstorage. This wastes space for objects having very few instance\nvariables. The space consumption can become acute when creating large\nnumbers of instances.\n\nThe default can be overridden by defining *__slots__* in a class\ndefinition. The *__slots__* declaration takes a sequence of instance\nvariables and reserves just enough space in each instance to hold a\nvalue for each variable. Space is saved because *__dict__* is not\ncreated for each instance.\n\nobject.__slots__\n\n This class variable can be assigned a string, iterable, or sequence\n of strings with variable names used by instances. If defined in a\n class, *__slots__* reserves space for the declared variables and\n prevents the automatic creation of *__dict__* and *__weakref__* for\n each instance.\n\n\nNotes on using *__slots__*\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n* When inheriting from a class without *__slots__*, the *__dict__*\n attribute of that class will always be accessible, so a *__slots__*\n definition in the subclass is meaningless.\n\n* Without a *__dict__* variable, instances cannot be assigned new\n variables not listed in the *__slots__* definition. Attempts to\n assign to an unlisted variable name raises "AttributeError". If\n dynamic assignment of new variables is desired, then add\n "\'__dict__\'" to the sequence of strings in the *__slots__*\n declaration.\n\n* Without a *__weakref__* variable for each instance, classes\n defining *__slots__* do not support weak references to its\n instances. If weak reference support is needed, then add\n "\'__weakref__\'" to the sequence of strings in the *__slots__*\n declaration.\n\n* *__slots__* are implemented at the class level by creating\n descriptors (*Implementing Descriptors*) for each variable name. As\n a result, class attributes cannot be used to set default values for\n instance variables defined by *__slots__*; otherwise, the class\n attribute would overwrite the descriptor assignment.\n\n* The action of a *__slots__* declaration is limited to the class\n where it is defined. As a result, subclasses will have a *__dict__*\n unless they also define *__slots__* (which must only contain names\n of any *additional* slots).\n\n* If a class defines a slot also defined in a base class, the\n instance variable defined by the base class slot is inaccessible\n (except by retrieving its descriptor directly from the base class).\n This renders the meaning of the program undefined. In the future, a\n check may be added to prevent this.\n\n* Nonempty *__slots__* does not work for classes derived from\n "variable-length" built-in types such as "int", "bytes" and "tuple".\n\n* Any non-string iterable may be assigned to *__slots__*. Mappings\n may also be used; however, in the future, special meaning may be\n assigned to the values corresponding to each key.\n\n* *__class__* assignment works only if both classes have the same\n *__slots__*.\n\n\nCustomizing class creation\n==========================\n\nBy default, classes are constructed using "type()". The class body is\nexecuted in a new namespace and the class name is bound locally to the\nresult of "type(name, bases, namespace)".\n\nThe class creation process can be customised by passing the\n"metaclass" keyword argument in the class definition line, or by\ninheriting from an existing class that included such an argument. In\nthe following example, both "MyClass" and "MySubclass" are instances\nof "Meta":\n\n class Meta(type):\n pass\n\n class MyClass(metaclass=Meta):\n pass\n\n class MySubclass(MyClass):\n pass\n\nAny other keyword arguments that are specified in the class definition\nare passed through to all metaclass operations described below.\n\nWhen a class definition is executed, the following steps occur:\n\n* the appropriate metaclass is determined\n\n* the class namespace is prepared\n\n* the class body is executed\n\n* the class object is created\n\n\nDetermining the appropriate metaclass\n-------------------------------------\n\nThe appropriate metaclass for a class definition is determined as\nfollows:\n\n* if no bases and no explicit metaclass are given, then "type()" is\n used\n\n* if an explicit metaclass is given and it is *not* an instance of\n "type()", then it is used directly as the metaclass\n\n* if an instance of "type()" is given as the explicit metaclass, or\n bases are defined, then the most derived metaclass is used\n\nThe most derived metaclass is selected from the explicitly specified\nmetaclass (if any) and the metaclasses (i.e. "type(cls)") of all\nspecified base classes. The most derived metaclass is one which is a\nsubtype of *all* of these candidate metaclasses. If none of the\ncandidate metaclasses meets that criterion, then the class definition\nwill fail with "TypeError".\n\n\nPreparing the class namespace\n-----------------------------\n\nOnce the appropriate metaclass has been identified, then the class\nnamespace is prepared. If the metaclass has a "__prepare__" attribute,\nit is called as "namespace = metaclass.__prepare__(name, bases,\n**kwds)" (where the additional keyword arguments, if any, come from\nthe class definition).\n\nIf the metaclass has no "__prepare__" attribute, then the class\nnamespace is initialised as an empty "dict()" instance.\n\nSee also: **PEP 3115** - Metaclasses in Python 3000\n\n Introduced the "__prepare__" namespace hook\n\n\nExecuting the class body\n------------------------\n\nThe class body is executed (approximately) as "exec(body, globals(),\nnamespace)". The key difference from a normal call to "exec()" is that\nlexical scoping allows the class body (including any methods) to\nreference names from the current and outer scopes when the class\ndefinition occurs inside a function.\n\nHowever, even when the class definition occurs inside the function,\nmethods defined inside the class still cannot see names defined at the\nclass scope. Class variables must be accessed through the first\nparameter of instance or class methods, and cannot be accessed at all\nfrom static methods.\n\n\nCreating the class object\n-------------------------\n\nOnce the class namespace has been populated by executing the class\nbody, the class object is created by calling "metaclass(name, bases,\nnamespace, **kwds)" (the additional keywords passed here are the same\nas those passed to "__prepare__").\n\nThis class object is the one that will be referenced by the zero-\nargument form of "super()". "__class__" is an implicit closure\nreference created by the compiler if any methods in a class body refer\nto either "__class__" or "super". This allows the zero argument form\nof "super()" to correctly identify the class being defined based on\nlexical scoping, while the class or instance that was used to make the\ncurrent call is identified based on the first argument passed to the\nmethod.\n\nAfter the class object is created, it is passed to the class\ndecorators included in the class definition (if any) and the resulting\nobject is bound in the local namespace as the defined class.\n\nSee also: **PEP 3135** - New super\n\n Describes the implicit "__class__" closure reference\n\n\nMetaclass example\n-----------------\n\nThe potential uses for metaclasses are boundless. Some ideas that have\nbeen explored include logging, interface checking, automatic\ndelegation, automatic property creation, proxies, frameworks, and\nautomatic resource locking/synchronization.\n\nHere is an example of a metaclass that uses an\n"collections.OrderedDict" to remember the order that class members\nwere defined:\n\n class OrderedClass(type):\n\n @classmethod\n def __prepare__(metacls, name, bases, **kwds):\n return collections.OrderedDict()\n\n def __new__(cls, name, bases, namespace, **kwds):\n result = type.__new__(cls, name, bases, dict(namespace))\n result.members = tuple(namespace)\n return result\n\n class A(metaclass=OrderedClass):\n def one(self): pass\n def two(self): pass\n def three(self): pass\n def four(self): pass\n\n >>> A.members\n (\'__module__\', \'one\', \'two\', \'three\', \'four\')\n\nWhen the class definition for *A* gets executed, the process begins\nwith calling the metaclass\'s "__prepare__()" method which returns an\nempty "collections.OrderedDict". That mapping records the methods and\nattributes of *A* as they are defined within the body of the class\nstatement. Once those definitions are executed, the ordered dictionary\nis fully populated and the metaclass\'s "__new__()" method gets\ninvoked. That method builds the new type and it saves the ordered\ndictionary keys in an attribute called "members".\n\n\nCustomizing instance and subclass checks\n========================================\n\nThe following methods are used to override the default behavior of the\n"isinstance()" and "issubclass()" built-in functions.\n\nIn particular, the metaclass "abc.ABCMeta" implements these methods in\norder to allow the addition of Abstract Base Classes (ABCs) as\n"virtual base classes" to any class or type (including built-in\ntypes), including other ABCs.\n\nclass.__instancecheck__(self, instance)\n\n Return true if *instance* should be considered a (direct or\n indirect) instance of *class*. If defined, called to implement\n "isinstance(instance, class)".\n\nclass.__subclasscheck__(self, subclass)\n\n Return true if *subclass* should be considered a (direct or\n indirect) subclass of *class*. If defined, called to implement\n "issubclass(subclass, class)".\n\nNote that these methods are looked up on the type (metaclass) of a\nclass. They cannot be defined as class methods in the actual class.\nThis is consistent with the lookup of special methods that are called\non instances, only in this case the instance is itself a class.\n\nSee also: **PEP 3119** - Introducing Abstract Base Classes\n\n Includes the specification for customizing "isinstance()" and\n "issubclass()" behavior through "__instancecheck__()" and\n "__subclasscheck__()", with motivation for this functionality in\n the context of adding Abstract Base Classes (see the "abc"\n module) to the language.\n\n\nEmulating callable objects\n==========================\n\nobject.__call__(self[, args...])\n\n Called when the instance is "called" as a function; if this method\n is defined, "x(arg1, arg2, ...)" is a shorthand for\n "x.__call__(arg1, arg2, ...)".\n\n\nEmulating container types\n=========================\n\nThe following methods can be defined to implement container objects.\nContainers usually are sequences (such as lists or tuples) or mappings\n(like dictionaries), but can represent other containers as well. The\nfirst set of methods is used either to emulate a sequence or to\nemulate a mapping; the difference is that for a sequence, the\nallowable keys should be the integers *k* for which "0 <= k < N" where\n*N* is the length of the sequence, or slice objects, which define a\nrange of items. It is also recommended that mappings provide the\nmethods "keys()", "values()", "items()", "get()", "clear()",\n"setdefault()", "pop()", "popitem()", "copy()", and "update()"\nbehaving similar to those for Python\'s standard dictionary objects.\nThe "collections" module provides a "MutableMapping" abstract base\nclass to help create those methods from a base set of "__getitem__()",\n"__setitem__()", "__delitem__()", and "keys()". Mutable sequences\nshould provide methods "append()", "count()", "index()", "extend()",\n"insert()", "pop()", "remove()", "reverse()" and "sort()", like Python\nstandard list objects. Finally, sequence types should implement\naddition (meaning concatenation) and multiplication (meaning\nrepetition) by defining the methods "__add__()", "__radd__()",\n"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" described\nbelow; they should not define other numerical operators. It is\nrecommended that both mappings and sequences implement the\n"__contains__()" method to allow efficient use of the "in" operator;\nfor mappings, "in" should search the mapping\'s keys; for sequences, it\nshould search through the values. It is further recommended that both\nmappings and sequences implement the "__iter__()" method to allow\nefficient iteration through the container; for mappings, "__iter__()"\nshould be the same as "keys()"; for sequences, it should iterate\nthrough the values.\n\nobject.__len__(self)\n\n Called to implement the built-in function "len()". Should return\n the length of the object, an integer ">=" 0. Also, an object that\n doesn\'t define a "__bool__()" method and whose "__len__()" method\n returns zero is considered to be false in a Boolean context.\n\nobject.__length_hint__(self)\n\n Called to implement "operator.length_hint()". Should return an\n estimated length for the object (which may be greater or less than\n the actual length). The length must be an integer ">=" 0. This\n method is purely an optimization and is never required for\n correctness.\n\n New in version 3.4.\n\nNote: Slicing is done exclusively with the following three methods.\n A call like\n\n a[1:2] = b\n\n is translated to\n\n a[slice(1, 2, None)] = b\n\n and so forth. Missing slice items are always filled in with "None".\n\nobject.__getitem__(self, key)\n\n Called to implement evaluation of "self[key]". For sequence types,\n the accepted keys should be integers and slice objects. Note that\n the special interpretation of negative indexes (if the class wishes\n to emulate a sequence type) is up to the "__getitem__()" method. If\n *key* is of an inappropriate type, "TypeError" may be raised; if of\n a value outside the set of indexes for the sequence (after any\n special interpretation of negative values), "IndexError" should be\n raised. For mapping types, if *key* is missing (not in the\n container), "KeyError" should be raised.\n\n Note: "for" loops expect that an "IndexError" will be raised for\n illegal indexes to allow proper detection of the end of the\n sequence.\n\nobject.__setitem__(self, key, value)\n\n Called to implement assignment to "self[key]". Same note as for\n "__getitem__()". This should only be implemented for mappings if\n the objects support changes to the values for keys, or if new keys\n can be added, or for sequences if elements can be replaced. The\n same exceptions should be raised for improper *key* values as for\n the "__getitem__()" method.\n\nobject.__delitem__(self, key)\n\n Called to implement deletion of "self[key]". Same note as for\n "__getitem__()". This should only be implemented for mappings if\n the objects support removal of keys, or for sequences if elements\n can be removed from the sequence. The same exceptions should be\n raised for improper *key* values as for the "__getitem__()" method.\n\nobject.__iter__(self)\n\n This method is called when an iterator is required for a container.\n This method should return a new iterator object that can iterate\n over all the objects in the container. For mappings, it should\n iterate over the keys of the container, and should also be made\n available as the method "keys()".\n\n Iterator objects also need to implement this method; they are\n required to return themselves. For more information on iterator\n objects, see *Iterator Types*.\n\nobject.__reversed__(self)\n\n Called (if present) by the "reversed()" built-in to implement\n reverse iteration. It should return a new iterator object that\n iterates over all the objects in the container in reverse order.\n\n If the "__reversed__()" method is not provided, the "reversed()"\n built-in will fall back to using the sequence protocol ("__len__()"\n and "__getitem__()"). Objects that support the sequence protocol\n should only provide "__reversed__()" if they can provide an\n implementation that is more efficient than the one provided by\n "reversed()".\n\nThe membership test operators ("in" and "not in") are normally\nimplemented as an iteration through a sequence. However, container\nobjects can supply the following special method with a more efficient\nimplementation, which also does not require the object be a sequence.\n\nobject.__contains__(self, item)\n\n Called to implement membership test operators. Should return true\n if *item* is in *self*, false otherwise. For mapping objects, this\n should consider the keys of the mapping rather than the values or\n the key-item pairs.\n\n For objects that don\'t define "__contains__()", the membership test\n first tries iteration via "__iter__()", then the old sequence\n iteration protocol via "__getitem__()", see *this section in the\n language reference*.\n\n\nEmulating numeric types\n=======================\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|"). For instance, to evaluate the\n expression "x + y", where *x* is an instance of a class that has an\n "__add__()" method, "x.__add__(y)" is called. The "__divmod__()"\n method should be the equivalent to using "__floordiv__()" and\n "__mod__()"; it should not be related to "__truediv__()". Note\n that "__pow__()" should be defined to accept an optional third\n argument if the ternary version of the built-in "pow()" function is\n to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return "NotImplemented".\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations ("+", "-", "*", "/", "//", "%", "divmod()", "pow()",\n "**", "<<", ">>", "&", "^", "|") with reflected (swapped) operands.\n These functions are only called if the left operand does not\n support the corresponding operation and the operands are of\n different types. [2] For instance, to evaluate the expression "x -\n y", where *y* is an instance of a class that has an "__rsub__()"\n method, "y.__rsub__(x)" is called if "x.__sub__(y)" returns\n *NotImplemented*.\n\n Note that ternary "pow()" will not try calling "__rpow__()" (the\n coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left\n operand\'s type and that subclass provides the reflected method\n for the operation, this method will be called before the left\n operand\'s non-reflected method. This behavior allows subclasses\n to override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments ("+=", "-=", "*=", "/=", "//=", "%=", "**=", "<<=",\n ">>=", "&=", "^=", "|="). These methods should attempt to do the\n operation in-place (modifying *self*) and return the result (which\n could be, but does not have to be, *self*). If a specific method\n is not defined, the augmented assignment falls back to the normal\n methods. For instance, if *x* is an instance of a class with an\n "__iadd__()" method, "x += y" is equivalent to "x = x.__iadd__(y)"\n . Otherwise, "x.__add__(y)" and "y.__radd__(x)" are considered, as\n with the evaluation of "x + y". In certain situations, augmented\n assignment can result in unexpected errors (see *Why does\n a_tuple[i] += [\'item\'] raise an exception when the addition\n works?*), but this behavior is in fact part of the data model.\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations ("-", "+",\n "abs()" and "~").\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions "complex()", "int()",\n "float()" and "round()". Should return a value of the appropriate\n type.\n\nobject.__index__(self)\n\n Called to implement "operator.index()", and whenever Python needs\n to losslessly convert the numeric object to an integer object (such\n as in slicing, or in the built-in "bin()", "hex()" and "oct()"\n functions). Presence of this method indicates that the numeric\n object is an integer type. Must return an integer.\n\n Note: When "__index__()" is defined, "__int__()" should also be\n defined, and both shuld return the same value, in order to have a\n coherent integer type class.\n\n\nWith Statement Context Managers\n===============================\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a "with" statement. The context manager\nhandles the entry into, and the exit from, the desired runtime context\nfor the execution of the block of code. Context managers are normally\ninvoked using the "with" statement (described in section *The with\nstatement*), but can also be used by directly invoking their methods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The "with"\n statement will bind this method\'s return value to the target(s)\n specified in the "as" clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be "None".\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that "__exit__()" methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also: **PEP 0343** - The "with" statement\n\n The specification, background, and examples for the Python "with"\n statement.\n\n\nSpecial method lookup\n=====================\n\nFor custom classes, implicit invocations of special methods are only\nguaranteed to work correctly if defined on an object\'s type, not in\nthe object\'s instance dictionary. That behaviour is the reason why\nthe following code raises an exception:\n\n >>> class C:\n ... pass\n ...\n >>> c = C()\n >>> c.__len__ = lambda: 5\n >>> len(c)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: object of type \'C\' has no len()\n\nThe rationale behind this behaviour lies with a number of special\nmethods such as "__hash__()" and "__repr__()" that are implemented by\nall objects, including type objects. If the implicit lookup of these\nmethods used the conventional lookup process, they would fail when\ninvoked on the type object itself:\n\n >>> 1 .__hash__() == hash(1)\n True\n >>> int.__hash__() == hash(int)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: descriptor \'__hash__\' of \'int\' object needs an argument\n\nIncorrectly attempting to invoke an unbound method of a class in this\nway is sometimes referred to as \'metaclass confusion\', and is avoided\nby bypassing the instance when looking up special methods:\n\n >>> type(1).__hash__(1) == hash(1)\n True\n >>> type(int).__hash__(int) == hash(int)\n True\n\nIn addition to bypassing any instance attributes in the interest of\ncorrectness, implicit special method lookup generally also bypasses\nthe "__getattribute__()" method even of the object\'s metaclass:\n\n >>> class Meta(type):\n ... def __getattribute__(*args):\n ... print("Metaclass getattribute invoked")\n ... return type.__getattribute__(*args)\n ...\n >>> class C(object, metaclass=Meta):\n ... def __len__(self):\n ... return 10\n ... def __getattribute__(*args):\n ... print("Class getattribute invoked")\n ... return object.__getattribute__(*args)\n ...\n >>> c = C()\n >>> c.__len__() # Explicit lookup via instance\n Class getattribute invoked\n 10\n >>> type(c).__len__(c) # Explicit lookup via type\n Metaclass getattribute invoked\n 10\n >>> len(c) # Implicit lookup\n 10\n\nBypassing the "__getattribute__()" machinery in this fashion provides\nsignificant scope for speed optimisations within the interpreter, at\nthe cost of some flexibility in the handling of special methods (the\nspecial method *must* be set on the class object itself in order to be\nconsistently invoked by the interpreter).\n\n-[ Footnotes ]-\n\n[1] It *is* possible in some cases to change an object\'s type,\n under certain controlled conditions. It generally isn\'t a good\n idea though, since it can lead to some very strange behaviour if\n it is handled incorrectly.\n\n[2] For operands of the same type, it is assumed that if the non-\n reflected method (such as "__add__()") fails the operation is not\n supported, which is why the reflected method is not called.\n', 'string-methods': '\nString Methods\n**************\n\nStrings implement all of the *common* sequence operations, along with\nthe additional methods described below.\n\nStrings also support two styles of string formatting, one providing a\nlarge degree of flexibility and customization (see "str.format()",\n*Format String Syntax* and *String Formatting*) and the other based on\nC "printf" style formatting that handles a narrower range of types and\nis slightly harder to use correctly, but is often faster for the cases\nit can handle (*printf-style String Formatting*).\n\nThe *Text Processing Services* section of the standard library covers\na number of other modules that provide various text related utilities\n(including regular expression support in the "re" module).\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter "\'\xc3\x9f\'" is equivalent to ""ss"".\n Since it is already lowercase, "lower()" would do nothing to "\'\xc3\x9f\'";\n "casefold()" converts it to ""ss"".\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is "\'utf-8\'". *errors* may be given to set a different\n error handling scheme. The default for *errors* is "\'strict\'",\n meaning that encoding errors raise a "UnicodeError". Other possible\n values are "\'ignore\'", "\'replace\'", "\'xmlcharrefreplace\'",\n "\'backslashreplace\'" and any other name registered via\n "codecs.register_error()", see section *Codec Base Classes*. For a\n list of possible encodings, see section *Standard Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return "True" if the string ends with the specified *suffix*,\n otherwise return "False". *suffix* can also be a tuple of suffixes\n to look for. With optional *start*, test beginning at that\n position. With optional *end*, stop comparing at that position.\n\nstr.expandtabs(tabsize=8)\n\n Return a copy of the string where all tab characters are replaced\n by one or more spaces, depending on the current column and the\n given tab size. Tab positions occur every *tabsize* characters\n (default is 8, giving tab positions at columns 0, 8, 16 and so on).\n To expand the string, the current column is set to zero and the\n string is examined character by character. If the character is a\n tab ("\\t"), one or more space characters are inserted in the result\n until the current column is equal to the next tab position. (The\n tab character itself is not copied.) If the character is a newline\n ("\\n") or return ("\\r"), it is copied and the current column is\n reset to zero. Any other character is copied unchanged and the\n current column is incremented by one regardless of how the\n character is represented when printed.\n\n >>> \'01\\t012\\t0123\\t01234\'.expandtabs()\n \'01 012 0123 01234\'\n >>> \'01\\t012\\t0123\\t01234\'.expandtabs(4)\n \'01 012 0123 01234\'\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice "s[start:end]".\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return "-1" if *sub* is not found.\n\n Note: The "find()" method should be used only if you need to know\n the position of *sub*. To check if *sub* is a substring or not,\n use the "in" operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces "{}". Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to "str.format(**mapping)", except that "mapping" is used\n directly and not copied to a "dict". This is useful if for example\n "mapping" is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like "find()", but raise "ValueError" when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character "c"\n is alphanumeric if one of the following returns "True":\n "c.isalpha()", "c.isdecimal()", "c.isdigit()", or "c.isnumeric()".\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\n Use "keyword.iskeyword()" to test for reserved identifiers such as\n "def" and "class".\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when "repr()" is\n invoked on a string. It has no bearing on the handling of strings\n written to "sys.stdout" or "sys.stderr".)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A "TypeError" will be raised if there are\n any non-string values in *iterable*, including "bytes" objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to "len(s)".\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or "None", the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n "str.translate()".\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within "s[start:end]".\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return "-1" on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like "rfind()" but raises "ValueError" when the substring *sub* is\n not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to "len(s)".\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n "None", any whitespace string is a separator. Except for splitting\n from the right, "rsplit()" behaves like "split()" which is\n described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or "None", the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most "maxsplit+1"\n elements). If *maxsplit* is not specified or "-1", then there is\n no limit on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n "\'1,,2\'.split(\',\')" returns "[\'1\', \'\', \'2\']"). The *sep* argument\n may consist of multiple characters (for example,\n "\'1<>2<>3\'.split(\'<>\')" returns "[\'1\', \'2\', \'3\']"). Splitting an\n empty string with a specified separator returns "[\'\']".\n\n If *sep* is not specified or is "None", a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a "None" separator returns "[]".\n\n For example, "\' 1 2 3 \'.split()" returns "[\'1\', \'2\', \'3\']", and\n "\' 1 2 3 \'.split(None, 1)" returns "[\'1\', \'2 3 \']".\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. This method uses the *universal newlines* approach to\n splitting lines. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\n For example, "\'ab c\\n\\nde fg\\rkl\\r\\n\'.splitlines()" returns "[\'ab\n c\', \'\', \'de fg\', \'kl\']", while the same call with\n "splitlines(True)" returns "[\'ab c\\n\', \'\\n\', \'de fg\\r\', \'kl\\r\\n\']".\n\n Unlike "split()" when a delimiter string *sep* is given, this\n method returns an empty list for the empty string, and a terminal\n line break does not result in an extra line.\n\nstr.startswith(prefix[, start[, end]])\n\n Return "True" if string starts with the *prefix*, otherwise return\n "False". *prefix* can also be a tuple of prefixes to look for.\n With optional *start*, test string beginning at that position.\n With optional *end*, stop comparing string at that position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or "None", the *chars*\n argument defaults to removing whitespace. The *chars* argument is\n not a prefix or suffix; rather, all combinations of its values are\n stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n "s.swapcase().swapcase() == s".\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n ... return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n ... lambda mo: mo.group(0)[0].upper() +\n ... mo.group(0)[1:].lower(),\n ... s)\n ...\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or "None". Unmapped\n characters are left untouched. Characters mapped to "None" are\n deleted.\n\n You can use "str.maketrans()" to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom\n character mapping codec using the "codecs" module (see\n "encodings.cp1251" for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that "str.upper().isupper()" might be\n "False" if "s" contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to "len(s)".\n', 'strings': '\nString and Bytes literals\n*************************\n\nString literals are described by the following lexical definitions:\n\n stringliteral ::= [stringprefix](shortstring | longstring)\n stringprefix ::= "r" | "u" | "R" | "U"\n shortstring ::= "\'" shortstringitem* "\'" | \'"\' shortstringitem* \'"\'\n longstring ::= "\'\'\'" longstringitem* "\'\'\'" | \'"""\' longstringitem* \'"""\'\n shortstringitem ::= shortstringchar | stringescapeseq\n longstringitem ::= longstringchar | stringescapeseq\n shortstringchar ::= \n longstringchar ::= \n stringescapeseq ::= "\\" \n\n bytesliteral ::= bytesprefix(shortbytes | longbytes)\n bytesprefix ::= "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB"\n shortbytes ::= "\'" shortbytesitem* "\'" | \'"\' shortbytesitem* \'"\'\n longbytes ::= "\'\'\'" longbytesitem* "\'\'\'" | \'"""\' longbytesitem* \'"""\'\n shortbytesitem ::= shortbyteschar | bytesescapeseq\n longbytesitem ::= longbyteschar | bytesescapeseq\n shortbyteschar ::= \n longbyteschar ::= \n bytesescapeseq ::= "\\" \n\nOne syntactic restriction not indicated by these productions is that\nwhitespace is not allowed between the "stringprefix" or "bytesprefix"\nand the rest of the literal. The source character set is defined by\nthe encoding declaration; it is UTF-8 if no encoding declaration is\ngiven in the source file; see section *Encoding declarations*.\n\nIn plain English: Both types of literals can be enclosed in matching\nsingle quotes ("\'") or double quotes ("""). They can also be enclosed\nin matching groups of three single or double quotes (these are\ngenerally referred to as *triple-quoted strings*). The backslash\n("\\") character is used to escape characters that otherwise have a\nspecial meaning, such as newline, backslash itself, or the quote\ncharacter.\n\nBytes literals are always prefixed with "\'b\'" or "\'B\'"; they produce\nan instance of the "bytes" type instead of the "str" type. They may\nonly contain ASCII characters; bytes with a numeric value of 128 or\ngreater must be expressed with escapes.\n\nAs of Python 3.3 it is possible again to prefix unicode strings with a\n"u" prefix to simplify maintenance of dual 2.x and 3.x codebases.\n\nBoth string and bytes literals may optionally be prefixed with a\nletter "\'r\'" or "\'R\'"; such strings are called *raw strings* and treat\nbackslashes as literal characters. As a result, in string literals,\n"\'\\U\'" and "\'\\u\'" escapes in raw strings are not treated specially.\nGiven that Python 2.x\'s raw unicode literals behave differently than\nPython 3.x\'s the "\'ur\'" syntax is not supported.\n\n New in version 3.3: The "\'rb\'" prefix of raw bytes literals has\n been added as a synonym of "\'br\'".\n\n New in version 3.3: Support for the unicode legacy literal\n ("u\'value\'") was reintroduced to simplify the maintenance of dual\n Python 2.x and 3.x codebases. See **PEP 414** for more information.\n\nIn triple-quoted strings, unescaped newlines and quotes are allowed\n(and are retained), except that three unescaped quotes in a row\nterminate the string. (A "quote" is the character used to open the\nstring, i.e. either "\'" or """.)\n\nUnless an "\'r\'" or "\'R\'" prefix is present, escape sequences in\nstrings are interpreted according to rules similar to those used by\nStandard C. The recognized escape sequences are:\n\n+-------------------+-----------------------------------+---------+\n| Escape Sequence | Meaning | Notes |\n+===================+===================================+=========+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n+-------------------+-----------------------------------+---------+\n| "\\ooo" | Character with octal value *ooo* | (1,3) |\n+-------------------+-----------------------------------+---------+\n| "\\xhh" | Character with hex value *hh* | (2,3) |\n+-------------------+-----------------------------------+---------+\n\nEscape sequences only recognized in string literals are:\n\n+-------------------+-----------------------------------+---------+\n| Escape Sequence | Meaning | Notes |\n+===================+===================================+=========+\n| "\\N{name}" | Character named *name* in the | (4) |\n+-------------------+-----------------------------------+---------+\n| "\\uxxxx" | Character with 16-bit hex value | (5) |\n+-------------------+-----------------------------------+---------+\n| "\\Uxxxxxxxx" | Character with 32-bit hex value | (6) |\n+-------------------+-----------------------------------+---------+\n\nNotes:\n\n1. As in Standard C, up to three octal digits are accepted.\n\n2. Unlike in Standard C, exactly two hex digits are required.\n\n3. In a bytes literal, hexadecimal and octal escapes denote the\n byte with the given value. In a string literal, these escapes\n denote a Unicode character with the given value.\n\n4. Changed in version 3.3: Support for name aliases [1] has been\n added.\n\n5. Individual code units which form parts of a surrogate pair can\n be encoded using this escape sequence. Exactly four hex digits are\n required.\n\n6. Any Unicode character can be encoded this way. Exactly eight\n hex digits are required.\n\nUnlike Standard C, all unrecognized escape sequences are left in the\nstring unchanged, i.e., *the backslash is left in the string*. (This\nbehavior is useful when debugging: if an escape sequence is mistyped,\nthe resulting output is more easily recognized as broken.) It is also\nimportant to note that the escape sequences only recognized in string\nliterals fall into the category of unrecognized escapes for bytes\nliterals.\n\nEven in a raw string, string quotes can be escaped with a backslash,\nbut the backslash remains in the string; for example, "r"\\""" is a\nvalid string literal consisting of two characters: a backslash and a\ndouble quote; "r"\\"" is not a valid string literal (even a raw string\ncannot end in an odd number of backslashes). Specifically, *a raw\nstring cannot end in a single backslash* (since the backslash would\nescape the following quote character). Note also that a single\nbackslash followed by a newline is interpreted as those two characters\nas part of the string, *not* as a line continuation.\n', 'subscriptions': '\nSubscriptions\n*************\n\nA subscription selects an item of a sequence (string, tuple or list)\nor mapping (dictionary) object:\n\n subscription ::= primary "[" expression_list "]"\n\nThe primary must evaluate to an object that supports subscription,\ne.g. a list or dictionary. User-defined objects can support\nsubscription by defining a "__getitem__()" method.\n\nFor built-in objects, there are two types of objects that support\nsubscription:\n\nIf the primary is a mapping, the expression list must evaluate to an\nobject whose value is one of the keys of the mapping, and the\nsubscription selects the value in the mapping that corresponds to that\nkey. (The expression list is a tuple except if it has exactly one\nitem.)\n\nIf the primary is a sequence, the expression (list) must evaluate to\nan integer or a slice (as discussed in the following section).\n\nThe formal syntax makes no special provision for negative indices in\nsequences; however, built-in sequences all provide a "__getitem__()"\nmethod that interprets negative indices by adding the length of the\nsequence to the index (so that "x[-1]" selects the last item of "x").\nThe resulting value must be a nonnegative integer less than the number\nof items in the sequence, and the subscription selects the item whose\nindex is that value (counting from zero). Since the support for\nnegative indices and slicing occurs in the object\'s "__getitem__()"\nmethod, subclasses overriding this method will need to explicitly add\nthat support.\n\nA string\'s items are characters. A character is not a separate data\ntype but a string of exactly one character.\n', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:40 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Release_bump_f?= =?utf-8?b?b3IgMy40LjAgZmluYWwu?= Message-ID: <3fnSQD0GJLz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/c7f42d3fb489 changeset: 89799:c7f42d3fb489 branch: 3.4 user: Larry Hastings date: Sat Mar 15 22:34:24 2014 -0700 summary: Release bump for 3.4.0 final. files: Doc/tools/sphinxext/pyspecific.py | 2 +- Include/patchlevel.h | 6 +++--- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/RPM/python-3.4.spec | 2 +- README | 4 ++-- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Doc/tools/sphinxext/pyspecific.py b/Doc/tools/sphinxext/pyspecific.py --- a/Doc/tools/sphinxext/pyspecific.py +++ b/Doc/tools/sphinxext/pyspecific.py @@ -10,7 +10,7 @@ """ ISSUE_URI = 'http://bugs.python.org/issue%s' -SOURCE_URI = 'http://hg.python.org/cpython/file/default/%s' +SOURCE_URI = 'http://hg.python.org/cpython/file/3.4/%s' from docutils import nodes, utils diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 4 #define PY_MICRO_VERSION 0 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 3 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.4.0rc3+" +#define PY_VERSION "3.4.0" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.4.0rc3" +__version__ = "3.4.0" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.4.0rc3" +IDLE_VERSION = "3.4.0" diff --git a/Misc/RPM/python-3.4.spec b/Misc/RPM/python-3.4.spec --- a/Misc/RPM/python-3.4.spec +++ b/Misc/RPM/python-3.4.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.4.0rc3 +%define version 3.4.0 %define libvers 3.4 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.4.0 release candidate 3 -================================================ +This is Python version 3.4.0 +============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:41 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Final_document?= =?utf-8?q?ation_fixes_for_3=2E4=2E0_final=2E?= Message-ID: <3fnSQF2RDgz7LkR@mail.python.org> http://hg.python.org/cpython/rev/04f714765c13 changeset: 89800:04f714765c13 branch: 3.4 tag: v3.4.0 user: Larry Hastings date: Sat Mar 15 22:43:17 2014 -0700 summary: Final documentation fixes for 3.4.0 final. files: Doc/license.rst | 6 +++--- Doc/whatsnew/3.4.rst | 5 +---- LICENSE | 4 ++-- 3 files changed, 6 insertions(+), 9 deletions(-) diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,9 +84,9 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ? 2001-2014 Python Software Foundation; All Rights - Reserved" are retained in Python |release| alone or in any derivative version - prepared by Licensee. + copyright, i.e., "Copyright ? 2001-2014 Python Software Foundation; All + Rights Reserved" are retained in Python |release| alone or in any derivative + version prepared by Licensee. #. In the event Licensee prepares a derivative work that is based on or incorporates Python |release| or any part thereof, and wants to make the diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -61,10 +61,7 @@ while :ref:`~mod.attr` will display as ``attr``. This article explains the new features in Python 3.4, compared to 3.3. - -.. Python 3.4 was released on TBD. - -For full details, see the +Python 3.4 was released on March 16, 2014. For full details, see the `changelog `_. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained -in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are +retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:42 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Added_tag_v3?= =?utf-8?q?=2E4=2E0_for_changeset_04f714765c13?= Message-ID: <3fnSQG4wqFz7LjY@mail.python.org> http://hg.python.org/cpython/rev/8671f89107c8 changeset: 89801:8671f89107c8 branch: 3.4 user: Larry Hastings date: Sat Mar 15 22:43:24 2014 -0700 summary: Added tag v3.4.0 for changeset 04f714765c13 files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -130,3 +130,4 @@ 5e088cea8660677969113741c1313d570d977e02 v3.4.0rc1 a300712ed38c9a242b736c44e806caea25a6dc05 v3.4.0rc2 8a81cdab3e9d521daaef989fade94b16455fc3b8 v3.4.0rc3 +04f714765c13824c3bc2835d7b008908862e083a v3.4.0 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:44 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Post-release_v?= =?utf-8?b?ZXJpb24gYnVtcCBmb3IgMy40LjAoKyku?= Message-ID: <3fnSQJ03vQz7LkV@mail.python.org> http://hg.python.org/cpython/rev/cf3ea100233a changeset: 89802:cf3ea100233a branch: 3.4 user: Larry Hastings date: Sun Mar 16 20:13:07 2014 -0700 summary: Post-release verion bump for 3.4.0(+). files: Include/patchlevel.h | 2 +- Misc/NEWS | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.4.0" +#define PY_VERSION "3.4.0+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,18 @@ Python News +++++++++++ +What's New in Python 3.4.1rc1? +============================== + +Release date: TBA + +Core and Builtins +----------------- + +Library +------- + + What's New in Python 3.4.0? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:47 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_3=2E4?= =?utf-8?q?=29=3A_Merged_default_into_3=2E4_branch=2E__3=2E4_branch_is_now?= =?utf-8?q?_effectively_3=2E4=2E1rc1=2E?= Message-ID: <3fnSQM4mDYz7Lkd@mail.python.org> http://hg.python.org/cpython/rev/3a3a83195159 changeset: 89803:3a3a83195159 branch: 3.4 parent: 89802:cf3ea100233a parent: 89691:5cab0ada97b2 user: Larry Hastings date: Sun Mar 16 22:54:05 2014 -0700 summary: Merged default into 3.4 branch. 3.4 branch is now effectively 3.4.1rc1. files: .hgeol | 2 + .hgignore | 5 - .hgtags | 3 + Doc/Makefile | 36 +- Doc/README.txt | 62 +- Doc/library/asyncio-subprocess.rst | 6 +- Doc/library/http.client.rst | 22 +- Doc/make.bat | 19 - Doc/tools/sphinx-build.py | 28 - Doc/tools/sphinxext/pyspecific.py | 22 +- Include/objimpl.h | 2 +- Include/pytime.h | 17 +- Lib/asyncio/selector_events.py | 11 +- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/test_utils.py | 20 +- Lib/asyncio/windows_events.py | 20 +- Lib/asyncio/windows_utils.py | 17 +- Lib/base64.py | 1 - Lib/copy.py | 2 +- Lib/ctypes/test/test_bitfields.py | 2 +- Lib/ctypes/test/test_structures.py | 18 +- Lib/distutils/core.py | 7 +- Lib/distutils/dir_util.py | 6 +- Lib/distutils/spawn.py | 55 +- Lib/distutils/tests/test_util.py | 10 +- Lib/distutils/util.py | 23 +- Lib/email/message.py | 6 +- Lib/encodings/base64_codec.py | 3 +- Lib/encodings/hex_codec.py | 3 +- Lib/encodings/quopri_codec.py | 3 +- Lib/encodings/rot_13.py | 3 +- Lib/encodings/uu_codec.py | 3 +- Lib/encodings/zlib_codec.py | 3 +- Lib/gzip.py | 2 +- Lib/idlelib/GrepDialog.py | 2 +- Lib/idlelib/MultiCall.py | 6 + Lib/idlelib/idle_test/README.txt | 10 +- Lib/idlelib/idle_test/test_formatparagraph.py | 3 + Lib/idlelib/idle_test/test_idlehistory.py | 1 + Lib/idlelib/idle_test/test_searchengine.py | 3 + Lib/idlelib/idle_test/test_text.py | 1 + Lib/ipaddress.py | 12 + Lib/logging/__init__.py | 16 +- Lib/modulefinder.py | 12 +- Lib/ntpath.py | 60 +- Lib/posixpath.py | 18 +- Lib/pydoc.py | 9 +- Lib/shutil.py | 3 +- Lib/sqlite3/test/factory.py | 1 + Lib/sqlite3/test/hooks.py | 2 +- Lib/subprocess.py | 23 +- Lib/test/_test_multiprocessing.py | 2 +- Lib/test/coding20731.py | 4 + Lib/test/regrtest.py | 9 +- Lib/test/test_asyncio/test_base_events.py | 110 +- Lib/test/test_asyncio/test_events.py | 455 ++++----- Lib/test/test_asyncio/test_futures.py | 16 +- Lib/test/test_asyncio/test_locks.py | 20 +- Lib/test/test_asyncio/test_proactor_events.py | 78 +- Lib/test/test_asyncio/test_queues.py | 4 +- Lib/test/test_asyncio/test_selector_events.py | 273 ++--- Lib/test/test_asyncio/test_streams.py | 5 +- Lib/test/test_asyncio/test_tasks.py | 8 +- Lib/test/test_asyncio/test_transports.py | 6 +- Lib/test/test_asyncio/test_unix_events.py | 200 ++-- Lib/test/test_asyncio/test_windows_events.py | 3 +- Lib/test/test_asyncio/test_windows_utils.py | 31 +- Lib/test/test_copy.py | 1 + Lib/test/test_email/test_email.py | 8 + Lib/test/test_email/test_policy.py | 9 + Lib/test/test_fileinput.py | 41 + Lib/test/test_genericpath.py | 30 +- Lib/test/test_grammar.py | 4 +- Lib/test/test_gzip.py | 7 + Lib/test/test_idle.py | 1 + Lib/test/test_importlib/source/test_file_loader.py | 1 + Lib/test/test_io.py | 38 + Lib/test/test_modulefinder.py | 20 + Lib/test/test_ntpath.py | 27 +- Lib/test/test_pkgutil.py | 5 + Lib/test/test_posix.py | 2 +- Lib/test/test_pydoc.py | 10 + Lib/test/test_range.py | 24 + Lib/test/test_re.py | 18 + Lib/test/test_robotparser.py | 1 + Lib/test/test_shutil.py | 9 + Lib/test/test_socket.py | 10 +- Lib/test/test_source_encoding.py | 10 + Lib/test/test_support.py | 2 +- Lib/test/test_sys.py | 85 +- Lib/test/test_tarfile.py | 18 +- Lib/test/test_tcl.py | 1 + Lib/test/test_threadsignals.py | 6 + Lib/test/test_time.py | 139 ++- Lib/test/test_tokenize.py | 98 +- Lib/test/test_tracemalloc.py | 2 + Lib/test/test_types.py | 26 +- Lib/test/test_urllib2.py | 3 +- Lib/tkinter/__init__.py | 29 +- Lib/tokenize.py | 34 +- Lib/tracemalloc.py | 21 +- Mac/README | 7 + Misc/ACKS | 5 + Misc/NEWS | 131 ++- Modules/_ctypes/cfield.c | 6 +- Modules/_datetimemodule.c | 4 +- Modules/_decimal/_decimal.c | 3 - Modules/_io/bufferedio.c | 9 +- Modules/_math.c | 2 + Modules/_sre.c | 145 +- Modules/_testcapimodule.c | 31 +- Modules/arraymodule.c | 2 + Modules/mathmodule.c | 4 +- Modules/posixmodule.c | 4 +- Modules/selectmodule.c | 27 +- Modules/signalmodule.c | 3 +- Modules/timemodule.c | 15 +- Objects/bytearrayobject.c | 10 +- Objects/bytesobject.c | 10 +- Objects/listobject.c | 6 +- Objects/longobject.c | 4 +- Objects/rangeobject.c | 31 +- Objects/tupleobject.c | 4 +- Objects/typeobject.c | 3 + Objects/unicodeobject.c | 12 +- PC/winreg.c | 5 +- PCbuild/pywlauncher.vcxproj | 5 + Parser/asdl_c.py | 8 +- Parser/tokenizer.c | 14 +- Python/Python-ast.c | 8 +- Python/getargs.c | 14 +- Python/pytime.c | 35 +- Tools/buildbot/test-amd64.bat | 2 +- Tools/hg/hgtouch.py | 2 +- Tools/scripts/run_tests.py | 6 + setup.py | 1 - 136 files changed, 1893 insertions(+), 1275 deletions(-) diff --git a/.hgeol b/.hgeol --- a/.hgeol +++ b/.hgeol @@ -38,6 +38,8 @@ Lib/venv/scripts/nt/* = BIN +Lib/test/coding20731.py = BIN + # All other files (which presumably are human-editable) are "native". # This must be the last rule! diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -24,11 +24,6 @@ reflog.txt$ tags$ Lib/plat-mac/errors.rsrc.df.rsrc -Doc/tools/sphinx/ -Doc/tools/docutils/ -Doc/tools/jinja/ -Doc/tools/jinja2/ -Doc/tools/pygments/ Misc/python.pc Misc/python-config.sh$ Modules/Setup$ diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -120,6 +120,9 @@ c3896275c0f61b2510a6c7e6c458a750359a91b8 v3.3.3 fa92f5f940c6c0d839d7f0611e4b717606504a3c v3.3.4rc1 7ff62415e4263c432c8acf6e424224209211eadb v3.3.4 +9ec811df548ed154a9bf9815383a916d6df31b98 v3.3.5rc1 +ca5635efe090f78806188ac2758f9948596aa8b2 v3.3.5rc2 +62cf4e77f78564714e7ea3d4bf1479ca1fbd0758 v3.3.5 46535f65e7f3bcdcf176f36d34bc1fed719ffd2b v3.4.0a1 9265a2168e2cb2a84785d8717792acc661e6b692 v3.4.0a2 dd9cdf90a5073510877e9dd5112f8e6cf20d5e89 v3.4.0a3 diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -5,7 +5,7 @@ # You can set these variables from the command line. PYTHON = python -SVNROOT = http://svn.python.org/projects +SPHINXBUILD = sphinx-build SPHINXOPTS = PAPER = SOURCES = @@ -14,14 +14,13 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ $(SPHINXOPTS) . build/$(BUILDER) $(SOURCES) -.PHONY: help checkout update build html htmlhelp latex text changes linkcheck \ +.PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ autobuild-dev autobuild-stable help: @echo "Please use \`make ' where is one of" @echo " clean to remove build files" - @echo " update to update build tools" @echo " html to make standalone HTML files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @@ -37,30 +36,8 @@ @echo " check to run a check for frequent markup errors" @echo " serve to serve the documentation on the localhost (8000)" -# Note: if you update versions here, do the same in make.bat and README.txt -checkout: - @if [ ! -d tools/sphinx ]; then \ - echo "Checking out Sphinx..."; \ - svn checkout $(SVNROOT)/external/Sphinx-1.2/sphinx tools/sphinx; \ - fi - @if [ ! -d tools/docutils ]; then \ - echo "Checking out Docutils..."; \ - svn checkout $(SVNROOT)/external/docutils-0.11/docutils tools/docutils; \ - fi - @if [ ! -d tools/jinja2 ]; then \ - echo "Checking out Jinja..."; \ - svn checkout $(SVNROOT)/external/Jinja-2.3.1/jinja2 tools/jinja2; \ - fi - @if [ ! -d tools/pygments ]; then \ - echo "Checking out Pygments..."; \ - svn checkout $(SVNROOT)/external/Pygments-1.6/pygments tools/pygments; \ - fi - -update: clean checkout - -build: checkout - mkdir -p build/$(BUILDER) build/doctrees - $(PYTHON) tools/sphinx-build.py $(ALLSPHINXOPTS) +build: + $(SPHINXBUILD) $(ALLSPHINXOPTS) @echo html: BUILDER = html @@ -120,10 +97,6 @@ clean: -rm -rf build/* - -rm -rf tools/sphinx - -rm -rf tools/pygments - -rm -rf tools/jinja2 - -rm -rf tools/docutils dist: rm -rf dist @@ -184,7 +157,6 @@ # for development releases: always build autobuild-dev: - make update make dist SPHINXOPTS='-A daily=1 -A versionswitcher=1' -make suspicious diff --git a/Doc/README.txt b/Doc/README.txt --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,36 +3,34 @@ This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at http://docs.python.org/download/. +available at . Documentation on the authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the -developers guide (http://docs.python.org/devguide/documenting.html). -There's also a chapter intended to point out differences to -those familiar with the previous docs written in LaTeX. +developers guide . Building the docs ================= -You need to have Python 2.4 or higher installed; the toolset used to build the -docs is written in Python. It is called *Sphinx*, it is not included in this -tree, but maintained separately. Also needed are the docutils, supplying the -base markup that Sphinx uses, Jinja, a templating engine, and optionally -Pygments, a code highlighter. +You need to have Sphinx installed; it is the toolset +used to build the docs. It is not included in this tree, but maintained +separately and available from PyPI . Using make ---------- -Luckily, a Makefile has been prepared so that on Unix, provided you have -installed Python and Subversion, you can just run :: +A Makefile has been prepared so that on Unix, provided you have installed +Sphinx, you can just run :: make html -to check out the necessary toolset in the `tools/` subdirectory and build the -HTML output files. To view the generated HTML, point your favorite browser at -the top-level index `build/html/index.html` after running "make". +to build the HTML output files. To view the generated HTML, point your favorite +browser at the top-level index `build/html/index.html` after running "make". + +On Windows, we try to emulate the Makefile as closely as possible with a +``make.bat`` file. To use a Python interpreter that's not called ``python``, use the standard way to set Makefile variables, using e.g. :: @@ -74,43 +72,21 @@ `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and keyword help. -A "make update" updates the Subversion checkouts in `tools/`. + * "suspicious", which checks the parsed markup for text that looks like + malformed and thus unconverted reST. Without make ------------ -You'll need to install the Sphinx package, either by checking it out via :: +Install the Sphinx package and its dependencies from PyPI. - svn co http://svn.python.org/projects/external/Sphinx-1.0.7/sphinx tools/sphinx +Then, from the ``Docs`` directory, run :: -or by installing it from PyPI. + sphinx-build -b . build/ -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b . build/ - -where `` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). +where ```` is one of html, text, latex, or htmlhelp (for explanations +see the make targets above). Contributing diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -107,6 +107,8 @@ The data read is buffered in memory, so do not use this method if the data size is large or unlimited. + This method is a :ref:`coroutine `. + .. method:: kill() Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to @@ -129,11 +131,13 @@ to the child. On Windows the Win32 API function :c:func:`TerminateProcess` is called to stop the child. - .. method:: wait(self): + .. method:: wait(): Wait for child process to terminate. Set and return :attr:`returncode` attribute. + This method is a :ref:`coroutine `. + Example ------- diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -451,11 +451,25 @@ .. method:: HTTPConnection.set_tunnel(host, port=None, headers=None) - Set the host and the port for HTTP Connect Tunnelling. Normally used when it - is required to a HTTPS Connection through a proxy server. + Set the host and the port for HTTP Connect Tunnelling. This allows running + the connection through a proxy server. - The headers argument should be a mapping of extra HTTP headers to send - with the CONNECT request. + The host and port arguments specify the endpoint of the tunneled connection + (i.e. the address included in the CONNECT request, *not* the address of the + proxy server). + + The headers argument should be a mapping of extra HTTP headers to send with + the CONNECT request. + + For example, to tunnel through a HTTPS proxy server running locally on port + 8080, we would pass the address of the proxy to the :class:`HTTPSConnection` + constructor, and the address of the host that we eventually want to reach to + the :meth:`~HTTPConnection.set_tunnel` method:: + + >>> import http.client + >>> conn = http.client.HTTPSConnection("localhost", 8080) + >>> conn.set_tunnel("www.python.org") + >>> conn.request("HEAD","/index.html") .. versionadded:: 3.2 diff --git a/Doc/make.bat b/Doc/make.bat --- a/Doc/make.bat +++ b/Doc/make.bat @@ -1,7 +1,6 @@ @@echo off setlocal -set SVNROOT=http://svn.python.org/projects if "%PYTHON%" EQU "" set PYTHON=py -2 if "%HTMLHELP%" EQU "" set HTMLHELP=%ProgramFiles%\HTML Help Workshop\hhc.exe if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/sphinxext/patchlevel.py`) do set DISTVERSION=%%v @@ -14,15 +13,11 @@ if "%1" EQU "suspicious" goto build if "%1" EQU "linkcheck" goto build if "%1" EQU "changes" goto build -if "%1" EQU "checkout" goto checkout -if "%1" EQU "update" goto update :help set this=%~n0 echo HELP echo. -echo %this% checkout -echo %this% update echo %this% html echo %this% htmlhelp echo %this% latex @@ -33,20 +28,6 @@ echo. goto end -:checkout -svn co %SVNROOT%/external/Sphinx-1.2/sphinx tools/sphinx -svn co %SVNROOT%/external/docutils-0.11/docutils tools/docutils -svn co %SVNROOT%/external/Jinja-2.3.1/jinja2 tools/jinja2 -svn co %SVNROOT%/external/Pygments-1.6/pygments tools/pygments -goto end - -:update -svn update tools/sphinx -svn update tools/docutils -svn update tools/jinja2 -svn update tools/pygments -goto end - :build if not exist build mkdir build if not exist build\%1 mkdir build\%1 diff --git a/Doc/tools/sphinx-build.py b/Doc/tools/sphinx-build.py deleted file mode 100644 --- a/Doc/tools/sphinx-build.py +++ /dev/null @@ -1,28 +0,0 @@ -# -*- coding: utf-8 -*- -""" - Sphinx - Python documentation toolchain - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - :copyright: 2007-2010 by Georg Brandl. - :license: Python license. -""" - -import sys -import warnings - -# Get rid of UserWarnings reported by pkg_resources. -warnings.filterwarnings('ignore', category=UserWarning, module='jinja2') - -if __name__ == '__main__': - - if sys.version_info[:3] < (2, 4, 0) or sys.version_info[:3] > (3, 0, 0): - sys.stderr.write("""\ -Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.0 though). -(If you run this from the Makefile, you can set the PYTHON variable -to the path of an alternative interpreter executable, e.g., -``make html PYTHON=python2.5``). -""") - sys.exit(1) - - from sphinx import main - sys.exit(main(sys.argv)) diff --git a/Doc/tools/sphinxext/pyspecific.py b/Doc/tools/sphinxext/pyspecific.py --- a/Doc/tools/sphinxext/pyspecific.py +++ b/Doc/tools/sphinxext/pyspecific.py @@ -16,6 +16,7 @@ import sphinx from sphinx.util.nodes import split_explicit_title +from sphinx.util.compat import Directive from sphinx.writers.html import HTMLTranslator from sphinx.writers.latex import LaTeXTranslator from sphinx.locale import versionlabels @@ -27,7 +28,9 @@ Body.enum.converters['lowerroman'] = \ Body.enum.converters['upperroman'] = lambda x: None -if sphinx.__version__[:3] < '1.2': +SPHINX11 = sphinx.__version__[:3] < '1.2' + +if SPHINX11: # monkey-patch HTML translator to give versionmodified paragraphs a class def new_visit_versionmodified(self, node): self.body.append(self.starttag(node, 'p', CLASS=node['type'])) @@ -88,8 +91,6 @@ # Support for marking up implementation details -from sphinx.util.compat import Directive - class ImplementationDetail(Directive): has_content = True @@ -142,10 +143,6 @@ # Support for documenting version of removal in deprecations -from sphinx.locale import versionlabels -from sphinx.util.compat import Directive - - class DeprecatedRemoved(Directive): has_content = True required_arguments = 2 @@ -171,16 +168,16 @@ messages = [] if self.content: self.state.nested_parse(self.content, self.content_offset, node) - if len(node): if isinstance(node[0], nodes.paragraph) and node[0].rawsource: content = nodes.inline(node[0].rawsource, translatable=True) content.source = node[0].source content.line = node[0].line content += node[0].children node[0].replace_self(nodes.paragraph('', '', content)) - node[0].insert(0, nodes.inline('', '%s: ' % text, - classes=['versionmodified'])) - else: + if not SPHINX11: + node[0].insert(0, nodes.inline('', '%s: ' % text, + classes=['versionmodified'])) + elif not SPHINX11: para = nodes.paragraph('', '', nodes.inline('', '%s.' % text, classes=['versionmodified'])) node.append(para) @@ -188,6 +185,9 @@ env.note_versionchange('deprecated', version[0], node, self.lineno) return [node] + messages +# for Sphinx < 1.2 +versionlabels['deprecated-removed'] = DeprecatedRemoved._label + # Support for including Misc/NEWS diff --git a/Include/objimpl.h b/Include/objimpl.h --- a/Include/objimpl.h +++ b/Include/objimpl.h @@ -265,7 +265,7 @@ #define _PyGCHead_REFS(g) ((g)->gc.gc_refs >> _PyGC_REFS_SHIFT) #define _PyGCHead_SET_REFS(g, v) do { \ (g)->gc.gc_refs = ((g)->gc.gc_refs & ~_PyGC_REFS_MASK) \ - | (v << _PyGC_REFS_SHIFT); \ + | (((size_t)(v)) << _PyGC_REFS_SHIFT); \ } while (0) #define _PyGCHead_DECREF(g) ((g)->gc.gc_refs -= 1 << _PyGC_REFS_SHIFT) diff --git a/Include/pytime.h b/Include/pytime.h --- a/Include/pytime.h +++ b/Include/pytime.h @@ -53,10 +53,19 @@ (tv_end.tv_usec - tv_start.tv_usec) * 0.000001) #ifndef Py_LIMITED_API + +typedef enum { + /* Round towards zero. */ + _PyTime_ROUND_DOWN=0, + /* Round away from zero. */ + _PyTime_ROUND_UP +} _PyTime_round_t; + /* Convert a number of seconds, int or float, to time_t. */ PyAPI_FUNC(int) _PyTime_ObjectToTime_t( PyObject *obj, - time_t *sec); + time_t *sec, + _PyTime_round_t); /* Convert a time_t to a PyLong. */ PyAPI_FUNC(PyObject *) _PyLong_FromTime_t( @@ -72,7 +81,8 @@ PyAPI_FUNC(int) _PyTime_ObjectToTimeval( PyObject *obj, time_t *sec, - long *usec); + long *usec, + _PyTime_round_t); /* Convert a number of seconds, int or float, to a timespec structure. nsec is in the range [0; 999999999] and rounded towards zero. @@ -80,7 +90,8 @@ PyAPI_FUNC(int) _PyTime_ObjectToTimespec( PyObject *obj, time_t *sec, - long *nsec); + long *nsec, + _PyTime_round_t); #endif /* Dummy to force linking. */ diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -136,6 +136,8 @@ def add_reader(self, fd, callback, *args): """Add a reader callback.""" + if self._selector is None: + raise RuntimeError('Event loop is closed') handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) @@ -151,6 +153,8 @@ def remove_reader(self, fd): """Remove a reader callback.""" + if self._selector is None: + return False try: key = self._selector.get_key(fd) except KeyError: @@ -171,6 +175,8 @@ def add_writer(self, fd, callback, *args): """Add a writer callback..""" + if self._selector is None: + raise RuntimeError('Event loop is closed') handle = events.Handle(callback, args, self) try: key = self._selector.get_key(fd) @@ -186,6 +192,8 @@ def remove_writer(self, fd): """Remove a writer callback.""" + if self._selector is None: + return False try: key = self._selector.get_key(fd) except KeyError: @@ -702,8 +710,7 @@ if self._buffer: try: n = self._sock.send(self._buffer) - except (BlockingIOError, InterruptedError, - ssl.SSLWantWriteError): + except (BlockingIOError, InterruptedError, ssl.SSLWantWriteError): n = 0 except ssl.SSLWantReadError: n = 0 diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -325,7 +325,7 @@ 'Task got bad yield: {!r}'.format(result))) finally: self.__class__._current_tasks.pop(self._loop) - self = None + self = None # Needed to break cycles when an exception occurs. def _wakeup(self, future): try: diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -11,8 +11,7 @@ import tempfile import threading import time -import unittest -import unittest.mock +from unittest import mock from http.server import HTTPServer from wsgiref.simple_server import WSGIRequestHandler, WSGIServer @@ -22,10 +21,11 @@ except ImportError: # pragma: no cover ssl = None -from . import tasks from . import base_events from . import events +from . import futures from . import selectors +from . import tasks if sys.platform == 'win32': # pragma: no cover @@ -53,18 +53,14 @@ gen.close() -def run_until(loop, pred, timeout=None): - if timeout is not None: - deadline = time.time() + timeout +def run_until(loop, pred, timeout=30): + deadline = time.time() + timeout while not pred(): if timeout is not None: timeout = deadline - time.time() if timeout <= 0: - return False - loop.run_until_complete(tasks.sleep(timeout, loop=loop)) - else: - run_briefly(loop) - return True + raise futures.TimeoutError() + loop.run_until_complete(tasks.sleep(0.001, loop=loop)) def run_once(loop): @@ -362,7 +358,7 @@ def MockCallback(**kwargs): - return unittest.mock.Mock(spec=['__call__'], **kwargs) + return mock.Mock(spec=['__call__'], **kwargs) class MockPattern(str): diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -213,7 +213,7 @@ else: ov.ReadFile(conn.fileno(), nbytes) - def finish(trans, key, ov): + def finish_recv(trans, key, ov): try: return ov.getresult() except OSError as exc: @@ -222,7 +222,7 @@ else: raise - return self._register(ov, conn, finish) + return self._register(ov, conn, finish_recv) def send(self, conn, buf, flags=0): self._register_with_iocp(conn) @@ -232,7 +232,7 @@ else: ov.WriteFile(conn.fileno(), buf) - def finish(trans, key, ov): + def finish_send(trans, key, ov): try: return ov.getresult() except OSError as exc: @@ -241,7 +241,7 @@ else: raise - return self._register(ov, conn, finish) + return self._register(ov, conn, finish_send) def accept(self, listener): self._register_with_iocp(listener) @@ -300,17 +300,17 @@ ov = _overlapped.Overlapped(NULL) ov.ConnectNamedPipe(pipe.fileno()) - def finish(trans, key, ov): + def finish_accept_pipe(trans, key, ov): ov.getresult() return pipe - return self._register(ov, pipe, finish) + return self._register(ov, pipe, finish_accept_pipe) def connect_pipe(self, address): ov = _overlapped.Overlapped(NULL) ov.WaitNamedPipeAndConnect(address, self._iocp, ov.address) - def finish(err, handle, ov): + def finish_connect_pipe(err, handle, ov): # err, handle were arguments passed to PostQueuedCompletionStatus() # in a function run in a thread pool. if err == _overlapped.ERROR_SEM_TIMEOUT: @@ -323,7 +323,7 @@ else: return windows_utils.PipeHandle(handle) - return self._register(ov, None, finish, wait_for_post=True) + return self._register(ov, None, finish_connect_pipe, wait_for_post=True) def wait_for_handle(self, handle, timeout=None): if timeout is None: @@ -339,7 +339,7 @@ handle, self._iocp, ov.address, ms) f = _WaitHandleFuture(wh, loop=self._loop) - def finish(trans, key, ov): + def finish_wait_for_handle(trans, key, ov): if not f.cancelled(): try: _overlapped.UnregisterWait(wh) @@ -355,7 +355,7 @@ return (_winapi.WaitForSingleObject(handle, 0) == _winapi.WAIT_OBJECT_0) - self._cache[ov.address] = (f, ov, None, finish) + self._cache[ov.address] = (f, ov, None, finish_wait_for_handle) return f def _register_with_iocp(self, obj): diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -36,12 +36,25 @@ Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain. """ + if family == socket.AF_INET: + host = '127.0.0.1' + elif family == socket.AF_INET6: + host = '::1' + else: + raise ValueError("Ony AF_INET and AF_INET6 socket address families " + "are supported") + if type != socket.SOCK_STREAM: + raise ValueError("Only SOCK_STREAM socket type is supported") + if proto != 0: + raise ValueError("Only protocol zero is supported") + # We create a connected TCP socket. Note the trick with setblocking(0) # that prevents us from having to create a thread. lsock = socket.socket(family, type, proto) - lsock.bind(('localhost', 0)) + lsock.bind((host, 0)) lsock.listen(1) - addr, port = lsock.getsockname() + # On IPv6, ignore flow_info and scope_id + addr, port = lsock.getsockname()[:2] csock = socket.socket(family, type, proto) csock.setblocking(False) try: diff --git a/Lib/base64.py b/Lib/base64.py --- a/Lib/base64.py +++ b/Lib/base64.py @@ -9,7 +9,6 @@ import re import struct import binascii -import itertools __all__ = [ diff --git a/Lib/copy.py b/Lib/copy.py --- a/Lib/copy.py +++ b/Lib/copy.py @@ -110,7 +110,7 @@ def _copy_immutable(x): return x for t in (type(None), int, float, bool, str, tuple, - frozenset, type, range, + bytes, frozenset, type, range, types.BuiltinFunctionType, type(Ellipsis), types.FunctionType, weakref.ref): d[t] = _copy_immutable diff --git a/Lib/ctypes/test/test_bitfields.py b/Lib/ctypes/test/test_bitfields.py --- a/Lib/ctypes/test/test_bitfields.py +++ b/Lib/ctypes/test/test_bitfields.py @@ -207,7 +207,7 @@ class X(Structure): _fields_ = [("a", c_byte, 4), ("b", c_int, 32)] - self.assertEqual(sizeof(X), sizeof(c_int)*2) + self.assertEqual(sizeof(X), alignment(c_int)+sizeof(c_int)) def test_mixed_3(self): class X(Structure): diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -83,7 +83,7 @@ class Y(Structure): _fields_ = [("x", c_char * 3), ("y", c_int)] - self.assertEqual(alignment(Y), calcsize("i")) + self.assertEqual(alignment(Y), alignment(c_int)) self.assertEqual(sizeof(Y), calcsize("3si")) class SI(Structure): @@ -175,13 +175,6 @@ self.assertEqual(sizeof(X), 10) self.assertEqual(X.b.offset, 2) - class X(Structure): - _fields_ = [("a", c_byte), - ("b", c_longlong)] - _pack_ = 4 - self.assertEqual(sizeof(X), 12) - self.assertEqual(X.b.offset, 4) - import struct longlong_size = struct.calcsize("q") longlong_align = struct.calcsize("bq") - longlong_size @@ -189,9 +182,16 @@ class X(Structure): _fields_ = [("a", c_byte), ("b", c_longlong)] + _pack_ = 4 + self.assertEqual(sizeof(X), min(4, longlong_align) + longlong_size) + self.assertEqual(X.b.offset, min(4, longlong_align)) + + class X(Structure): + _fields_ = [("a", c_byte), + ("b", c_longlong)] _pack_ = 8 - self.assertEqual(sizeof(X), longlong_align + longlong_size) + self.assertEqual(sizeof(X), min(8, longlong_align) + longlong_size) self.assertEqual(X.b.offset, min(8, longlong_align)) diff --git a/Lib/distutils/core.py b/Lib/distutils/core.py --- a/Lib/distutils/core.py +++ b/Lib/distutils/core.py @@ -11,7 +11,6 @@ from distutils.debug import DEBUG from distutils.errors import * -from distutils.util import grok_environment_error # Mainly import these so setup scripts can "from distutils.core import" them. from distutils.dist import Distribution @@ -150,13 +149,11 @@ except KeyboardInterrupt: raise SystemExit("interrupted") except OSError as exc: - error = grok_environment_error(exc) - if DEBUG: - sys.stderr.write(error + "\n") + sys.stderr.write("error: %s\n" % (exc,)) raise else: - raise SystemExit(error) + raise SystemExit("error: %s" % (exc,)) except (DistutilsError, CCompilerError) as msg: diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -2,7 +2,7 @@ Utility functions for manipulating directories and directory trees.""" -import os, sys +import os import errno from distutils.errors import DistutilsFileError, DistutilsInternalError from distutils import log @@ -182,7 +182,6 @@ Any errors are ignored (apart from being reported to stdout if 'verbose' is true). """ - from distutils.util import grok_environment_error global _path_created if verbose >= 1: @@ -199,8 +198,7 @@ if abspath in _path_created: del _path_created[abspath] except OSError as exc: - log.warn(grok_environment_error( - exc, "error removing %s: " % directory)) + log.warn("error removing %s: %s", directory, exc) def ensure_relative(path): """Take the full path 'path', and make it a relative path. diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -10,6 +10,7 @@ import os from distutils.errors import DistutilsPlatformError, DistutilsExecError +from distutils.debug import DEBUG from distutils import log def spawn(cmd, search_path=1, verbose=0, dry_run=0): @@ -28,10 +29,15 @@ Raise DistutilsExecError if running the program fails in any way; just return on success. """ + # cmd is documented as a list, but just in case some code passes a tuple + # in, protect our %-formatting code against horrible death + cmd = list(cmd) if os.name == 'posix': _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': _spawn_nt(cmd, search_path, dry_run=dry_run) + elif os.name == 'os2': + _spawn_os2(cmd, search_path, dry_run=dry_run) else: raise DistutilsPlatformError( "don't know how to spawn programs on platform '%s'" % os.name) @@ -65,12 +71,16 @@ rc = os.spawnv(os.P_WAIT, executable, cmd) except OSError as exc: # this seems to happen when the command isn't found + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if rc != 0: # and this reflects the command running but failing + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" % (cmd[0], rc)) + "command %r failed with exit status %d" % (cmd, rc)) if sys.platform == 'darwin': from distutils import sysconfig @@ -81,8 +91,9 @@ log.info(' '.join(cmd)) if dry_run: return + executable = cmd[0] exec_fn = search_path and os.execvp or os.execv - exec_args = [cmd[0], cmd] + env = None if sys.platform == 'darwin': global _cfg_target, _cfg_target_split if _cfg_target is None: @@ -103,17 +114,23 @@ env = dict(os.environ, MACOSX_DEPLOYMENT_TARGET=cur_target) exec_fn = search_path and os.execvpe or os.execve - exec_args.append(env) pid = os.fork() if pid == 0: # in the child try: - exec_fn(*exec_args) + if env is None: + exec_fn(executable, cmd) + else: + exec_fn(executable, cmd, env) except OSError as e: - sys.stderr.write("unable to execute %s: %s\n" - % (cmd[0], e.strerror)) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r: %s\n" + % (cmd, e.strerror)) os._exit(1) - sys.stderr.write("unable to execute %s for unknown reasons" % cmd[0]) + if not DEBUG: + cmd = executable + sys.stderr.write("unable to execute %r for unknown reasons" % cmd) os._exit(1) else: # in the parent # Loop until the child either exits or is terminated by a signal @@ -125,26 +142,34 @@ import errno if exc.errno == errno.EINTR: continue + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed: %s" % (cmd[0], exc.args[-1])) + "command %r failed: %s" % (cmd, exc.args[-1])) if os.WIFSIGNALED(status): + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' terminated by signal %d" - % (cmd[0], os.WTERMSIG(status))) + "command %r terminated by signal %d" + % (cmd, os.WTERMSIG(status))) elif os.WIFEXITED(status): exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "command '%s' failed with exit status %d" - % (cmd[0], exit_status)) + "command %r failed with exit status %d" + % (cmd, exit_status)) elif os.WIFSTOPPED(status): continue else: + if not DEBUG: + cmd = executable raise DistutilsExecError( - "unknown error executing '%s': termination status %d" - % (cmd[0], status)) + "unknown error executing %r: termination status %d" + % (cmd, status)) def find_executable(executable, path=None): """Tries to find 'executable' in the directories listed in 'path'. diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -8,7 +8,8 @@ from distutils.errors import DistutilsPlatformError, DistutilsByteCompileError from distutils.util import (get_platform, convert_path, change_root, check_environ, split_quoted, strtobool, - rfc822_escape, byte_compile) + rfc822_escape, byte_compile, + grok_environment_error) from distutils import util # used to patch _environ_checked from distutils.sysconfig import get_config_vars from distutils import sysconfig @@ -285,6 +286,13 @@ finally: sys.dont_write_bytecode = old_dont_write_bytecode + def test_grok_environment_error(self): + # test obsolete function to ensure backward compat (#4931) + exc = IOError("Unable to find batch file") + msg = grok_environment_error(exc) + self.assertEqual(msg, "error: Unable to find batch file") + + def test_suite(): return unittest.makeSuite(UtilTestCase) diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -207,25 +207,10 @@ def grok_environment_error (exc, prefix="error: "): - """Generate a useful error message from an OSError - exception object. Handles Python 1.5.1 and 1.5.2 styles, and - does what it can to deal with exception objects that don't have a - filename (which happens when the error is due to a two-file operation, - such as 'rename()' or 'link()'. Returns the error message as a string - prefixed with 'prefix'. - """ - # check for Python 1.5.2-style {IO,OS}Error exception objects - if hasattr(exc, 'filename') and hasattr(exc, 'strerror'): - if exc.filename: - error = prefix + "%s: %s" % (exc.filename, exc.strerror) - else: - # two-argument functions in posix module don't - # include the filename in the exception object! - error = prefix + "%s" % exc.strerror - else: - error = prefix + str(exc.args[-1]) - - return error + # Function kept for backward compatibility. + # Used to try clever things with EnvironmentErrors, + # but nowadays str(exception) produces good messages. + return prefix + str(exc) # Needed by 'split_quoted()' diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -203,7 +203,11 @@ if self._payload is None: self._payload = [payload] else: - self._payload.append(payload) + try: + self._payload.append(payload) + except AttributeError: + raise TypeError("Attach is not valid on a message with a" + " non-multipart payload") def get_payload(self, i=None, decode=False): """Return a reference to the payload. diff --git a/Lib/encodings/base64_codec.py b/Lib/encodings/base64_codec.py --- a/Lib/encodings/base64_codec.py +++ b/Lib/encodings/base64_codec.py @@ -1,7 +1,6 @@ """Python 'base64_codec' Codec - base64 content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/hex_codec.py b/Lib/encodings/hex_codec.py --- a/Lib/encodings/hex_codec.py +++ b/Lib/encodings/hex_codec.py @@ -1,7 +1,6 @@ """Python 'hex_codec' Codec - 2-digit hex content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/quopri_codec.py b/Lib/encodings/quopri_codec.py --- a/Lib/encodings/quopri_codec.py +++ b/Lib/encodings/quopri_codec.py @@ -1,7 +1,6 @@ """Codec for quoted-printable encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. """ import codecs diff --git a/Lib/encodings/rot_13.py b/Lib/encodings/rot_13.py --- a/Lib/encodings/rot_13.py +++ b/Lib/encodings/rot_13.py @@ -1,8 +1,7 @@ #!/usr/bin/env python """ Python Character Mapping Codec for ROT13. -This codec de/encodes from str to str and is therefore usable with -str.transform() and str.untransform(). +This codec de/encodes from str to str. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/encodings/uu_codec.py b/Lib/encodings/uu_codec.py --- a/Lib/encodings/uu_codec.py +++ b/Lib/encodings/uu_codec.py @@ -1,7 +1,6 @@ """Python 'uu_codec' Codec - UU content transfer encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). Some details were adapted from uu.py which was written by Lance Ellinghouse and diff --git a/Lib/encodings/zlib_codec.py b/Lib/encodings/zlib_codec.py --- a/Lib/encodings/zlib_codec.py +++ b/Lib/encodings/zlib_codec.py @@ -1,7 +1,6 @@ """Python 'zlib_codec' Codec - zlib compression encoding. -This codec de/encodes from bytes to bytes and is therefore usable with -bytes.transform() and bytes.untransform(). +This codec de/encodes from bytes to bytes. Written by Marc-Andre Lemburg (mal at lemburg.com). """ diff --git a/Lib/gzip.py b/Lib/gzip.py --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -96,7 +96,7 @@ self._read -= len(prepend) return else: - self._buffer = self._buffer[read:] + prepend + self._buffer = self._buffer[self._read:] + prepend self._length = len(self._buffer) self._read = 0 diff --git a/Lib/idlelib/GrepDialog.py b/Lib/idlelib/GrepDialog.py --- a/Lib/idlelib/GrepDialog.py +++ b/Lib/idlelib/GrepDialog.py @@ -98,7 +98,7 @@ def findfiles(self, dir, base, rec): try: names = os.listdir(dir or os.curdir) - except OSerror as msg: + except OSError as msg: print(msg) return [] list = [] diff --git a/Lib/idlelib/MultiCall.py b/Lib/idlelib/MultiCall.py --- a/Lib/idlelib/MultiCall.py +++ b/Lib/idlelib/MultiCall.py @@ -111,6 +111,8 @@ except tkinter.TclError as e: if e.args[0] == APPLICATION_GONE: pass + else: + raise # An int in range(1 << len(_modifiers)) represents a combination of modifiers # (if the least significent bit is on, _modifiers[0] is on, and so on). @@ -244,6 +246,8 @@ except tkinter.TclError as e: if e.args[0] == APPLICATION_GONE: break + else: + raise # define the list of event types to be handled by MultiEvent. the order is # compatible with the definition of event type constants. @@ -411,6 +415,8 @@ except tkinter.TclError as e: if e.args[0] == APPLICATION_GONE: break + else: + raise _multicall_dict[widget] = MultiCall return MultiCall diff --git a/Lib/idlelib/idle_test/README.txt b/Lib/idlelib/idle_test/README.txt --- a/Lib/idlelib/idle_test/README.txt +++ b/Lib/idlelib/idle_test/README.txt @@ -41,9 +41,10 @@ screen, gui tests must be 'guarded' by "requires('gui')" in a setUp function or method. This will typically be setUpClass. -All gui objects must be destroyed by the end of the test, perhaps in a tearDown -function. Creating the Tk root directly in a setUp allows a reference to be saved -so it can be properly destroyed in the corresponding tearDown. +To avoid interfering with other gui tests, all gui objects must be destroyed +and deleted by the end of the test. If a widget, such as a Tk root, is created +in a setUpX function, destroy it in the corresponding tearDownX. For module +and class attributes, also delete the widget. --- @classmethod def setUpClass(cls): @@ -53,6 +54,7 @@ @classmethod def tearDownClass(cls): cls.root.destroy() + del cls.root --- Support.requires('gui') returns true if it is either called in a main module @@ -105,4 +107,4 @@ To run an individual Testcase or test method, extend the dotted name given to unittest on the command line. (But gui tests will not this way.) -python -m unittest -v idlelib.idle_test.text_xyz.Test_case.test_meth +python -m unittest -v idlelib.idle_test.test_xyz.Test_case.test_meth diff --git a/Lib/idlelib/idle_test/test_formatparagraph.py b/Lib/idlelib/idle_test/test_formatparagraph.py --- a/Lib/idlelib/idle_test/test_formatparagraph.py +++ b/Lib/idlelib/idle_test/test_formatparagraph.py @@ -277,6 +277,9 @@ @classmethod def tearDownClass(cls): cls.root.destroy() + del cls.root + del cls.text + del cls.formatter def test_short_line(self): self.text.insert('1.0', "Short line\n") diff --git a/Lib/idlelib/idle_test/test_idlehistory.py b/Lib/idlelib/idle_test/test_idlehistory.py --- a/Lib/idlelib/idle_test/test_idlehistory.py +++ b/Lib/idlelib/idle_test/test_idlehistory.py @@ -80,6 +80,7 @@ @classmethod def tearDownClass(cls): cls.root.destroy() + del cls.root def fetch_test(self, reverse, line, prefix, index, *, bell=False): # Perform one fetch as invoked by Alt-N or Alt-P diff --git a/Lib/idlelib/idle_test/test_searchengine.py b/Lib/idlelib/idle_test/test_searchengine.py --- a/Lib/idlelib/idle_test/test_searchengine.py +++ b/Lib/idlelib/idle_test/test_searchengine.py @@ -64,6 +64,7 @@ ## @classmethod ## def tearDownClass(cls): ## cls.root.destroy() +## del cls.root def test_get_selection(self): # text = Text(master=self.root) @@ -219,6 +220,7 @@ ## @classmethod ## def tearDownClass(cls): ## cls.root.destroy() +## del cls.root def test_search(self): Equal = self.assertEqual @@ -261,6 +263,7 @@ ## @classmethod ## def tearDownClass(cls): ## cls.root.destroy() +## del cls.root @classmethod def setUpClass(cls): diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py --- a/Lib/idlelib/idle_test/test_text.py +++ b/Lib/idlelib/idle_test/test_text.py @@ -221,6 +221,7 @@ @classmethod def tearDownClass(cls): cls.root.destroy() + del cls.root if __name__ == '__main__': diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -2155,6 +2155,18 @@ if self._prefixlen == (self._max_prefixlen - 1): self.hosts = self.__iter__ + def hosts(self): + """Generate Iterator over usable hosts in a network. + + This is like __iter__ except it doesn't return the + Subnet-Router anycast address. + + """ + network = int(self.network_address) + broadcast = int(self.broadcast_address) + for x in range(network + 1, broadcast + 1): + yield self._address_class(x) + @property def is_site_local(self): """Test if the address is reserved for site-local. diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -18,7 +18,7 @@ Logging package for Python. Based on PEP 282 and comments thereto in comp.lang.python. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ @@ -42,6 +42,7 @@ __author__ = "Vinay Sajip " __status__ = "production" +# The following module attributes are no longer updated. __version__ = "0.5.1.2" __date__ = "07 February 2010" @@ -902,8 +903,15 @@ sys.stderr.write('Logged from file %s, line %s\n' % ( record.filename, record.lineno)) # Issue 18671: output logging message and arguments - sys.stderr.write('Message: %r\n' - 'Arguments: %s\n' % (record.msg, record.args)) + try: + sys.stderr.write('Message: %r\n' + 'Arguments: %s\n' % (record.msg, + record.args)) + except Exception: + sys.stderr.write('Unable to print the message and arguments' + ' - possible formatting error.\nUse the' + ' traceback above to help find the error.\n' + ) except OSError: #pragma: no cover pass # see issue 5971 finally: diff --git a/Lib/modulefinder.py b/Lib/modulefinder.py --- a/Lib/modulefinder.py +++ b/Lib/modulefinder.py @@ -1,6 +1,7 @@ """Find modules used by a script, using introspection.""" import dis +import importlib._bootstrap import importlib.machinery import marshal import os @@ -287,11 +288,12 @@ if type == imp.PY_SOURCE: co = compile(fp.read()+'\n', pathname, 'exec') elif type == imp.PY_COMPILED: - if fp.read(4) != imp.get_magic(): - self.msgout(2, "raise ImportError: Bad magic number", pathname) - raise ImportError("Bad magic number in %s" % pathname) - fp.read(4) - co = marshal.load(fp) + try: + marshal_data = importlib._bootstrap._validate_bytecode_header(fp.read()) + except ImportError as exc: + self.msgout(2, "raise ImportError: " + str(exc), pathname) + raise + co = marshal.loads(marshal_data) else: co = None m = self.add_module(fqname) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -377,6 +377,7 @@ percent = b'%' brace = b'{' dollar = b'$' + environ = getattr(os, 'environb', None) else: if '$' not in path and '%' not in path: return path @@ -386,6 +387,7 @@ percent = '%' brace = '{' dollar = '$' + environ = os.environ res = path[:0] index = 0 pathlen = len(path) @@ -414,14 +416,13 @@ index = pathlen - 1 else: var = path[:index] - if isinstance(path, bytes): - var = var.decode('ascii') - if var in os.environ: - value = os.environ[var] - else: - value = '%' + var + '%' - if isinstance(path, bytes): - value = value.encode('ascii') + try: + if environ is None: + value = os.fsencode(os.environ[os.fsdecode(var)]) + else: + value = environ[var] + except KeyError: + value = percent + var + percent res += value elif c == dollar: # variable or '$$' if path[index + 1:index + 2] == dollar: @@ -435,39 +436,40 @@ index = path.index(b'}') else: index = path.index('}') - var = path[:index] - if isinstance(path, bytes): - var = var.decode('ascii') - if var in os.environ: - value = os.environ[var] - else: - value = '${' + var + '}' - if isinstance(path, bytes): - value = value.encode('ascii') - res += value except ValueError: if isinstance(path, bytes): res += b'${' + path else: res += '${' + path index = pathlen - 1 + else: + var = path[:index] + try: + if environ is None: + value = os.fsencode(os.environ[os.fsdecode(var)]) + else: + value = environ[var] + except KeyError: + if isinstance(path, bytes): + value = b'${' + var + b'}' + else: + value = '${' + var + '}' + res += value else: - var = '' + var = path[:0] index += 1 c = path[index:index + 1] while c and c in varchars: - if isinstance(path, bytes): - var += c.decode('ascii') - else: - var += c + var += c index += 1 c = path[index:index + 1] - if var in os.environ: - value = os.environ[var] - else: - value = '$' + var - if isinstance(path, bytes): - value = value.encode('ascii') + try: + if environ is None: + value = os.fsencode(os.environ[os.fsdecode(var)]) + else: + value = environ[var] + except KeyError: + value = dollar + var res += value if c: index -= 1 diff --git a/Lib/posixpath.py b/Lib/posixpath.py --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -279,6 +279,7 @@ search = _varprogb.search start = b'{' end = b'}' + environ = getattr(os, 'environb', None) else: if '$' not in path: return path @@ -288,6 +289,7 @@ search = _varprog.search start = '{' end = '}' + environ = os.environ i = 0 while True: m = search(path, i) @@ -297,18 +299,18 @@ name = m.group(1) if name.startswith(start) and name.endswith(end): name = name[1:-1] - if isinstance(name, bytes): - name = str(name, 'ASCII') - if name in os.environ: + try: + if environ is None: + value = os.fsencode(os.environ[os.fsdecode(name)]) + else: + value = environ[name] + except KeyError: + i = j + else: tail = path[j:] - value = os.environ[name] - if isinstance(path, bytes): - value = value.encode('ASCII') path = path[:i] + value i = len(path) path += tail - else: - i = j return path diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -1256,9 +1256,12 @@ doc = getdoc(value) else: doc = None - push(self.docother( - getattr(object, name, None) or homecls.__dict__[name], - name, mod, maxlen=70, doc=doc) + '\n') + try: + obj = getattr(object, name) + except AttributeError: + obj = homecls.__dict__[name] + push(self.docother(obj, name, mod, maxlen=70, doc=doc) + + '\n') return attrs attrs = [(name, kind, cls, value) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -483,7 +483,8 @@ def _basename(path): # A basename() variant which first strips the trailing slash, if present. # Thus we always get the last component of the path, even for directories. - return os.path.basename(path.rstrip(os.path.sep)) + sep = os.path.sep + (os.path.altsep or '') + return os.path.basename(path.rstrip(sep)) def move(src, dst): """Recursively move a file or directory to another location. This is diff --git a/Lib/sqlite3/test/factory.py b/Lib/sqlite3/test/factory.py --- a/Lib/sqlite3/test/factory.py +++ b/Lib/sqlite3/test/factory.py @@ -112,6 +112,7 @@ self.con.row_factory = sqlite.Row row = self.con.execute("select 1 as a, 2 as b").fetchone() t = tuple(row) + self.assertEqual(t, (row['a'], row['b'])) def CheckSqliteRowAsDict(self): """Checks if the row object can be correctly converted to a dictionary""" diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -162,7 +162,7 @@ create table bar (a, b) """) second_count = len(progress_calls) - self.assertGreater(first_count, second_count) + self.assertGreaterEqual(first_count, second_count) def CheckCancelOperation(self): """ diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -738,6 +738,9 @@ class Popen(object): + + _child_created = False # Set here since __del__ checks it + def __init__(self, args, bufsize=-1, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS, @@ -748,7 +751,6 @@ """Create new Popen instance.""" _cleanup() - self._child_created = False self._input = None self._communication_started = False if bufsize is None: @@ -890,11 +892,8 @@ # Wait for the process to terminate, to avoid zombies. self.wait() - def __del__(self, _maxsize=sys.maxsize, _active=_active): - # If __init__ hasn't had a chance to execute (e.g. if it - # was passed an undeclared keyword argument), we don't - # have a _child_created attribute at all. - if not getattr(self, '_child_created', False): + def __del__(self, _maxsize=sys.maxsize): + if not self._child_created: # We didn't get to successfully create a child process. return # In case the child hasn't been waited on, check if it's done. @@ -1187,7 +1186,15 @@ try: self.stdin.write(input) except OSError as e: - if e.errno != errno.EPIPE: + if e.errno == errno.EPIPE: + # communicate() should ignore pipe full error + pass + elif (e.errno == errno.EINVAL + and self.poll() is not None): + # Issue #19612: stdin.write() fails with EINVAL + # if the process already exited before the write + pass + else: raise self.stdin.close() @@ -1446,7 +1453,7 @@ _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED, _WEXITSTATUS=os.WEXITSTATUS): # This method is called (indirectly) by __del__, so it cannot - # refer to anything outside of its local scope.""" + # refer to anything outside of its local scope. if _WIFSIGNALED(sts): self.returncode = -_WTERMSIG(sts) elif _WIFEXITED(sts): diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3651,7 +3651,7 @@ _multiprocessing.sem_unlink(name1) p.terminate() p.wait() - time.sleep(1.0) + time.sleep(2.0) with self.assertRaises(OSError) as ctx: _multiprocessing.sem_unlink(name2) # docs say it should be ENOENT, but OSX seems to give EINVAL diff --git a/Lib/test/coding20731.py b/Lib/test/coding20731.py new file mode 100644 --- /dev/null +++ b/Lib/test/coding20731.py @@ -0,0 +1,4 @@ +#coding:latin1 + + + diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -1373,10 +1373,9 @@ try: import zipimport except ImportError: - zsc = zdc = None # Run unmodified on platforms without zipimport support + zdc = None # Run unmodified on platforms without zipimport support else: zdc = zipimport._zip_directory_cache.copy() - zsc = zipimport._zip_stat_cache.copy() abcs = {} for abc in [getattr(collections.abc, a) for a in collections.abc.__all__]: if not isabstract(abc): @@ -1395,7 +1394,7 @@ sys.stderr.flush() for i in range(repcount): indirect_test() - alloc_after, rc_after = dash_R_cleanup(fs, ps, pic, zdc, zsc, abcs) + alloc_after, rc_after = dash_R_cleanup(fs, ps, pic, zdc, abcs) sys.stderr.write('.') sys.stderr.flush() if i >= nwarmup: @@ -1429,7 +1428,7 @@ failed = True return failed -def dash_R_cleanup(fs, ps, pic, zdc, zsc, abcs): +def dash_R_cleanup(fs, ps, pic, zdc, abcs): import gc, copyreg import _strptime, linecache import urllib.parse, urllib.request, mimetypes, doctest @@ -1455,8 +1454,6 @@ else: zipimport._zip_directory_cache.clear() zipimport._zip_directory_cache.update(zdc) - zipimport._zip_stat_cache.clear() - zipimport._zip_stat_cache.update(zsc) # clear type cache sys._clear_type_cache() diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -6,8 +6,8 @@ import sys import time import unittest -import unittest.mock -from test.support import find_unused_port, IPV6_ENABLED +from unittest import mock +from test.support import IPV6_ENABLED import asyncio from asyncio import base_events @@ -15,7 +15,7 @@ from asyncio import test_utils -MOCK_ANY = unittest.mock.ANY +MOCK_ANY = mock.ANY PY34 = sys.version_info >= (3, 4) @@ -23,11 +23,11 @@ def setUp(self): self.loop = base_events.BaseEventLoop() - self.loop._selector = unittest.mock.Mock() + self.loop._selector = mock.Mock() asyncio.set_event_loop(None) def test_not_implemented(self): - m = unittest.mock.Mock() + m = mock.Mock() self.assertRaises( NotImplementedError, self.loop._make_socket_transport, m, m) @@ -75,13 +75,13 @@ self.assertFalse(self.loop._ready) def test_set_default_executor(self): - executor = unittest.mock.Mock() + executor = mock.Mock() self.loop.set_default_executor(executor) self.assertIs(executor, self.loop._default_executor) def test_getnameinfo(self): - sockaddr = unittest.mock.Mock() - self.loop.run_in_executor = unittest.mock.Mock() + sockaddr = mock.Mock() + self.loop.run_in_executor = mock.Mock() self.loop.getnameinfo(sockaddr) self.assertEqual( (None, socket.getnameinfo, sockaddr, 0), @@ -111,7 +111,7 @@ def cb(arg): calls.append(arg) - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() self.loop.call_later(-1, cb, 'a') self.loop.call_later(-2, cb, 'b') test_utils.run_briefly(self.loop) @@ -121,7 +121,7 @@ def cb(): self.loop.stop() - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() delay = 0.1 when = self.loop.time() + delay @@ -163,7 +163,7 @@ pass h = asyncio.Handle(cb, (), self.loop) f = asyncio.Future(loop=self.loop) - executor = unittest.mock.Mock() + executor = mock.Mock() executor.submit.return_value = f self.loop.set_default_executor(executor) @@ -171,7 +171,7 @@ res = self.loop.run_in_executor(None, h) self.assertIs(f, res) - executor = unittest.mock.Mock() + executor = mock.Mock() executor.submit.return_value = f res = self.loop.run_in_executor(executor, h) self.assertIs(f, res) @@ -187,7 +187,7 @@ h1.cancel() - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() self.loop._scheduled.append(h1) self.loop._scheduled.append(h2) self.loop._run_once() @@ -203,8 +203,8 @@ self.loop.set_debug(False) self.assertFalse(self.loop.get_debug()) - @unittest.mock.patch('asyncio.base_events.time') - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.time') + @mock.patch('asyncio.base_events.logger') def test__run_once_logging(self, m_logger, m_time): # Log to INFO level if timeout > 1.0 sec. idx = -1 @@ -219,7 +219,7 @@ self.loop._scheduled.append( asyncio.TimerHandle(11.0, lambda: True, (), self.loop)) - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() self.loop._run_once() self.assertEqual(logging.INFO, m_logger.log.call_args[0][0]) @@ -242,7 +242,7 @@ h = asyncio.TimerHandle(time.monotonic() - 1, cb, (self.loop,), self.loop) - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() self.loop._scheduled.append(h) self.loop._run_once() @@ -303,14 +303,14 @@ asyncio.SubprocessProtocol, 'exit 0', bufsize=4096) def test_default_exc_handler_callback(self): - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() def zero_error(fut): fut.set_result(True) 1/0 # Test call_soon (events.Handle) - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: fut = asyncio.Future(loop=self.loop) self.loop.call_soon(zero_error, fut) fut.add_done_callback(lambda fut: self.loop.stop()) @@ -320,7 +320,7 @@ exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) # Test call_later (events.TimerHandle) - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: fut = asyncio.Future(loop=self.loop) self.loop.call_later(0.01, zero_error, fut) fut.add_done_callback(lambda fut: self.loop.stop()) @@ -330,7 +330,7 @@ exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY)) def test_default_exc_handler_coro(self): - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() @asyncio.coroutine def zero_error_coro(): @@ -338,7 +338,7 @@ 1/0 # Test Future.__del__ - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: fut = asyncio.async(zero_error_coro(), loop=self.loop) fut.add_done_callback(lambda *args: self.loop.stop()) self.loop.run_forever() @@ -368,9 +368,9 @@ self.loop.call_soon(zero_error) self.loop._run_once() - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() - mock_handler = unittest.mock.Mock() + mock_handler = mock.Mock() self.loop.set_exception_handler(mock_handler) run_loop() mock_handler.assert_called_with(self.loop, { @@ -382,7 +382,7 @@ mock_handler.reset_mock() self.loop.set_exception_handler(None) - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: run_loop() log.error.assert_called_with( test_utils.MockPattern( @@ -401,11 +401,11 @@ def handler(loop, context): raise AttributeError('spam') - self.loop._process_events = unittest.mock.Mock() + self.loop._process_events = mock.Mock() self.loop.set_exception_handler(handler) - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: run_loop() log.error.assert_called_with( test_utils.MockPattern( @@ -417,8 +417,8 @@ class Loop(base_events.BaseEventLoop): - _selector = unittest.mock.Mock() - _process_events = unittest.mock.Mock() + _selector = mock.Mock() + _process_events = mock.Mock() def default_exception_handler(self, context): nonlocal _context @@ -435,7 +435,7 @@ loop.call_soon(zero_error) loop._run_once() - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: run_loop() log.error.assert_called_with( 'Exception in default exception handler', @@ -446,7 +446,7 @@ _context = None loop.set_exception_handler(custom_handler) - with unittest.mock.patch('asyncio.base_events.logger') as log: + with mock.patch('asyncio.base_events.logger') as log: run_loop() log.error.assert_called_with( test_utils.MockPattern('Exception in default exception.*' @@ -527,7 +527,7 @@ def tearDown(self): self.loop.close() - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_connection_multiple_errors(self, m_socket): class MyProto(asyncio.Protocol): @@ -592,7 +592,7 @@ return asyncio.Task(getaddrinfo(*args, **kwds), loop=self.loop) self.loop.getaddrinfo = getaddrinfo_task - self.loop.sock_connect = unittest.mock.Mock() + self.loop.sock_connect = mock.Mock() self.loop.sock_connect.side_effect = OSError coro = self.loop.create_connection(MyProto, 'example.com', 80) @@ -609,7 +609,7 @@ return asyncio.Task(getaddrinfo(*args, **kwds), loop=self.loop) self.loop.getaddrinfo = getaddrinfo_task - self.loop.sock_connect = unittest.mock.Mock() + self.loop.sock_connect = mock.Mock() self.loop.sock_connect.side_effect = OSError coro = self.loop.create_connection( @@ -617,7 +617,7 @@ with self.assertRaises(OSError): self.loop.run_until_complete(coro) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_connection_multiple_errors_local_addr(self, m_socket): def bind(addr): @@ -637,7 +637,7 @@ return asyncio.Task(getaddrinfo(*args, **kwds), loop=self.loop) self.loop.getaddrinfo = getaddrinfo_task - self.loop.sock_connect = unittest.mock.Mock() + self.loop.sock_connect = mock.Mock() self.loop.sock_connect.side_effect = OSError('Err2') coro = self.loop.create_connection( @@ -669,7 +669,7 @@ OSError, self.loop.run_until_complete, coro) def test_create_connection_ssl_server_hostname_default(self): - self.loop.getaddrinfo = unittest.mock.Mock() + self.loop.getaddrinfo = mock.Mock() def mock_getaddrinfo(*args, **kwds): f = asyncio.Future(loop=self.loop) @@ -678,9 +678,9 @@ return f self.loop.getaddrinfo.side_effect = mock_getaddrinfo - self.loop.sock_connect = unittest.mock.Mock() + self.loop.sock_connect = mock.Mock() self.loop.sock_connect.return_value = () - self.loop._make_ssl_transport = unittest.mock.Mock() + self.loop._make_ssl_transport = mock.Mock() class _SelectorTransportMock: _sock = None @@ -696,7 +696,7 @@ return transport self.loop._make_ssl_transport.side_effect = mock_make_ssl_transport - ANY = unittest.mock.ANY + ANY = mock.ANY # First try the default server_hostname. self.loop._make_ssl_transport.reset_mock() coro = self.loop.create_connection(MyProto, 'python.org', 80, ssl=True) @@ -775,13 +775,13 @@ self.assertRaises(ValueError, self.loop.run_until_complete, fut) def test_create_server_no_getaddrinfo(self): - getaddrinfo = self.loop.getaddrinfo = unittest.mock.Mock() + getaddrinfo = self.loop.getaddrinfo = mock.Mock() getaddrinfo.return_value = [] f = self.loop.create_server(MyProto, '0.0.0.0', 0) self.assertRaises(OSError, self.loop.run_until_complete, f) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_server_cant_bind(self, m_socket): class Err(OSError): @@ -790,14 +790,14 @@ m_socket.getaddrinfo.return_value = [ (2, 1, 6, '', ('127.0.0.1', 10100))] m_socket.getaddrinfo._is_coroutine = False - m_sock = m_socket.socket.return_value = unittest.mock.Mock() + m_sock = m_socket.socket.return_value = mock.Mock() m_sock.bind.side_effect = Err fut = self.loop.create_server(MyProto, '0.0.0.0', 0) self.assertRaises(OSError, self.loop.run_until_complete, fut) self.assertTrue(m_sock.close.called) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_datagram_endpoint_no_addrinfo(self, m_socket): m_socket.getaddrinfo.return_value = [] m_socket.getaddrinfo._is_coroutine = False @@ -818,7 +818,7 @@ AssertionError, self.loop.run_until_complete, coro) def test_create_datagram_endpoint_connect_err(self): - self.loop.sock_connect = unittest.mock.Mock() + self.loop.sock_connect = mock.Mock() self.loop.sock_connect.side_effect = OSError coro = self.loop.create_datagram_endpoint( @@ -826,7 +826,7 @@ self.assertRaises( OSError, self.loop.run_until_complete, coro) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_datagram_endpoint_socket_err(self, m_socket): m_socket.getaddrinfo = socket.getaddrinfo m_socket.socket.side_effect = OSError @@ -849,7 +849,7 @@ self.assertRaises( ValueError, self.loop.run_until_complete, coro) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_datagram_endpoint_setblk_err(self, m_socket): m_socket.socket.return_value.setblocking.side_effect = OSError @@ -865,14 +865,14 @@ asyncio.DatagramProtocol) self.assertRaises(ValueError, self.loop.run_until_complete, coro) - @unittest.mock.patch('asyncio.base_events.socket') + @mock.patch('asyncio.base_events.socket') def test_create_datagram_endpoint_cant_bind(self, m_socket): class Err(OSError): pass m_socket.AF_INET6 = socket.AF_INET6 m_socket.getaddrinfo = socket.getaddrinfo - m_sock = m_socket.socket.return_value = unittest.mock.Mock() + m_sock = m_socket.socket.return_value = mock.Mock() m_sock.bind.side_effect = Err fut = self.loop.create_datagram_endpoint( @@ -882,19 +882,19 @@ self.assertTrue(m_sock.close.called) def test_accept_connection_retry(self): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.accept.side_effect = BlockingIOError() self.loop._accept_connection(MyProto, sock) self.assertFalse(sock.close.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_accept_connection_exception(self, m_log): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.accept.side_effect = OSError(errno.EMFILE, 'Too many open files') - self.loop.remove_reader = unittest.mock.Mock() - self.loop.call_later = unittest.mock.Mock() + self.loop.remove_reader = mock.Mock() + self.loop.call_later = mock.Mock() self.loop._accept_connection(MyProto, sock) self.assertTrue(m_log.error.called) @@ -902,7 +902,7 @@ self.loop.remove_reader.assert_called_with(10) self.loop.call_later.assert_called_with(constants.ACCEPT_RETRY_DELAY, # self.loop._start_serving - unittest.mock.ANY, + mock.ANY, MyProto, sock, None, None) def test_call_coroutine(self): diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -20,12 +20,11 @@ import time import errno import unittest -import unittest.mock +from unittest import mock from test import support # find_unused_port, IPV6_ENABLED, TEST_HOME_DIR import asyncio -from asyncio import events from asyncio import selector_events from asyncio import test_utils @@ -57,6 +56,7 @@ class MyBaseProto(asyncio.Protocol): + connected = None done = None def __init__(self, loop=None): @@ -64,12 +64,15 @@ self.state = 'INITIAL' self.nbytes = 0 if loop is not None: + self.connected = asyncio.Future(loop=loop) self.done = asyncio.Future(loop=loop) def connection_made(self, transport): self.transport = transport assert self.state == 'INITIAL', self.state self.state = 'CONNECTED' + if self.connected: + self.connected.set_result(None) def data_received(self, data): assert self.state == 'CONNECTED', self.state @@ -331,7 +334,8 @@ def test_reader_callback(self): r, w = test_utils.socketpair() - bytes_read = [] + r.setblocking(False) + bytes_read = bytearray() def reader(): try: @@ -341,37 +345,40 @@ # at least on Linux -- see man select. return if data: - bytes_read.append(data) + bytes_read.extend(data) else: self.assertTrue(self.loop.remove_reader(r.fileno())) r.close() self.loop.add_reader(r.fileno(), reader) self.loop.call_soon(w.send, b'abc') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 3) self.loop.call_soon(w.send, b'def') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: len(bytes_read) >= 6) self.loop.call_soon(w.close) self.loop.call_soon(self.loop.stop) self.loop.run_forever() - self.assertEqual(b''.join(bytes_read), b'abcdef') + self.assertEqual(bytes_read, b'abcdef') def test_writer_callback(self): r, w = test_utils.socketpair() w.setblocking(False) - self.loop.add_writer(w.fileno(), w.send, b'x'*(256*1024)) - test_utils.run_briefly(self.loop) - def remove_writer(): - self.assertTrue(self.loop.remove_writer(w.fileno())) + def writer(data): + w.send(data) + self.loop.stop() - self.loop.call_soon(remove_writer) - self.loop.call_soon(self.loop.stop) + data = b'x' * 1024 + self.loop.add_writer(w.fileno(), writer, data) self.loop.run_forever() + + self.assertTrue(self.loop.remove_writer(w.fileno())) + self.assertFalse(self.loop.remove_writer(w.fileno())) + w.close() - data = r.recv(256*1024) + read = r.recv(len(data) * 2) r.close() - self.assertGreaterEqual(len(data), 200) + self.assertEqual(read, data) def _basetest_sock_client_ops(self, httpd, sock): sock.setblocking(False) @@ -465,10 +472,10 @@ self.assertFalse(self.loop.remove_signal_handler(signal.SIGKILL)) # Now set a handler and handle it. self.loop.add_signal_handler(signal.SIGINT, my_handler) - test_utils.run_briefly(self.loop) + os.kill(os.getpid(), signal.SIGINT) - test_utils.run_briefly(self.loop) - self.assertEqual(caught, 1) + test_utils.run_until(self.loop, lambda: caught) + # Removing it should restore the default handler. self.assertTrue(self.loop.remove_signal_handler(signal.SIGINT)) self.assertEqual(signal.getsignal(signal.SIGINT), @@ -624,7 +631,7 @@ self.assertIn(str(httpd.address), cm.exception.strerror) def test_create_server(self): - proto = MyProto() + proto = MyProto(self.loop) f = self.loop.create_server(lambda: proto, '0.0.0.0', 0) server = self.loop.run_until_complete(f) self.assertEqual(len(server.sockets), 1) @@ -634,14 +641,11 @@ client = socket.socket() client.connect(('127.0.0.1', port)) client.sendall(b'xxx') - test_utils.run_briefly(self.loop) - test_utils.run_until(self.loop, lambda: proto is not None, 10) - self.assertIsInstance(proto, MyProto) - self.assertEqual('INITIAL', proto.state) - test_utils.run_briefly(self.loop) + + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # extra info is available @@ -651,7 +655,7 @@ # close connection proto.transport.close() - test_utils.run_briefly(self.loop) # windows iocp + self.loop.run_until_complete(proto.done) self.assertEqual('CLOSED', proto.state) @@ -673,27 +677,22 @@ @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server(self): - proto = MyProto() + proto = MyProto(loop=self.loop) server, path = self._make_unix_server(lambda: proto) self.assertEqual(len(server.sockets), 1) client = socket.socket(socket.AF_UNIX) client.connect(path) client.sendall(b'xxx') - test_utils.run_briefly(self.loop) - test_utils.run_until(self.loop, lambda: proto is not None, 10) - self.assertIsInstance(proto, MyProto) - self.assertEqual('INITIAL', proto.state) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # close connection proto.transport.close() - test_utils.run_briefly(self.loop) # windows iocp + self.loop.run_until_complete(proto.done) self.assertEqual('CLOSED', proto.state) @@ -736,12 +735,10 @@ client, pr = self.loop.run_until_complete(f_c) client.write(b'xxx') - test_utils.run_briefly(self.loop) - self.assertIsInstance(proto, MyProto) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # extra info is available @@ -775,12 +772,9 @@ client, pr = self.loop.run_until_complete(f_c) client.write(b'xxx') - test_utils.run_briefly(self.loop) - self.assertIsInstance(proto, MyProto) - test_utils.run_briefly(self.loop) + self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - test_utils.run_until(self.loop, lambda: proto.nbytes > 0, - timeout=10) + test_utils.run_until(self.loop, lambda: proto.nbytes > 0) self.assertEqual(3, proto.nbytes) # close connection @@ -1045,15 +1039,9 @@ self.assertEqual('INITIALIZED', client.state) transport.sendto(b'xxx') - for _ in range(1000): - if server.nbytes: - break - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: server.nbytes) self.assertEqual(3, server.nbytes) - for _ in range(1000): - if client.nbytes: - break - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: client.nbytes) # received self.assertEqual(8, client.nbytes) @@ -1098,11 +1086,11 @@ self.loop.run_until_complete(connect()) os.write(wpipe, b'1') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: proto.nbytes >= 1) self.assertEqual(1, proto.nbytes) os.write(wpipe, b'2345') - test_utils.run_briefly(self.loop) + test_utils.run_until(self.loop, lambda: proto.nbytes >= 5) self.assertEqual(['INITIAL', 'CONNECTED'], proto.state) self.assertEqual(5, proto.nbytes) @@ -1156,33 +1144,30 @@ @unittest.skipUnless(sys.platform != 'win32', "Don't support pipes for Windows") def test_write_pipe(self): - proto = MyWritePipeProto(loop=self.loop) - transport = None - rpipe, wpipe = os.pipe() pipeobj = io.open(wpipe, 'wb', 1024) - @asyncio.coroutine - def connect(): - nonlocal transport - t, p = yield from self.loop.connect_write_pipe( - lambda: proto, pipeobj) - self.assertIs(p, proto) - self.assertIs(t, proto.transport) - self.assertEqual('CONNECTED', proto.state) - transport = t - - self.loop.run_until_complete(connect()) + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, pipeobj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) + self.assertEqual('CONNECTED', proto.state) transport.write(b'1') - test_utils.run_briefly(self.loop) - data = os.read(rpipe, 1024) + + data = bytearray() + def reader(data): + chunk = os.read(rpipe, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1) self.assertEqual(b'1', data) transport.write(b'2345') - test_utils.run_briefly(self.loop) - data = os.read(rpipe, 1024) - self.assertEqual(b'2345', data) + test_utils.run_until(self.loop, lambda: reader(data) >= 5) + self.assertEqual(b'12345', data) self.assertEqual('CONNECTED', proto.state) os.close(rpipe) @@ -1198,23 +1183,14 @@ @unittest.skipUnless(sys.platform != 'win32', "Don't support pipes for Windows") def test_write_pipe_disconnect_on_close(self): - proto = MyWritePipeProto(loop=self.loop) - transport = None - rsock, wsock = test_utils.socketpair() pipeobj = io.open(wsock.detach(), 'wb', 1024) - @asyncio.coroutine - def connect(): - nonlocal transport - t, p = yield from self.loop.connect_write_pipe(lambda: proto, - pipeobj) - self.assertIs(p, proto) - self.assertIs(t, proto.transport) - self.assertEqual('CONNECTED', proto.state) - transport = t - - self.loop.run_until_complete(connect()) + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, pipeobj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) self.assertEqual('CONNECTED', proto.state) transport.write(b'1') @@ -1232,33 +1208,32 @@ # older than 10.6 (Snow Leopard) @support.requires_mac_ver(10, 6) def test_write_pty(self): - proto = MyWritePipeProto(loop=self.loop) - transport = None - master, slave = os.openpty() slave_write_obj = io.open(slave, 'wb', 0) - @asyncio.coroutine - def connect(): - nonlocal transport - t, p = yield from self.loop.connect_write_pipe(lambda: proto, - slave_write_obj) - self.assertIs(p, proto) - self.assertIs(t, proto.transport) - self.assertEqual('CONNECTED', proto.state) - transport = t - - self.loop.run_until_complete(connect()) + proto = MyWritePipeProto(loop=self.loop) + connect = self.loop.connect_write_pipe(lambda: proto, slave_write_obj) + transport, p = self.loop.run_until_complete(connect) + self.assertIs(p, proto) + self.assertIs(transport, proto.transport) + self.assertEqual('CONNECTED', proto.state) transport.write(b'1') - test_utils.run_briefly(self.loop) - data = os.read(master, 1024) + + data = bytearray() + def reader(data): + chunk = os.read(master, 1024) + data += chunk + return len(data) + + test_utils.run_until(self.loop, lambda: reader(data) >= 1, + timeout=10) self.assertEqual(b'1', data) transport.write(b'2345') - test_utils.run_briefly(self.loop) - data = os.read(master, 1024) - self.assertEqual(b'2345', data) + test_utils.run_until(self.loop, lambda: reader(data) >= 5, + timeout=10) + self.assertEqual(b'12345', data) self.assertEqual('CONNECTED', proto.state) os.close(master) @@ -1352,6 +1327,30 @@ self.assertIn('address must be resolved', str(cm.exception)) + def test_remove_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = test_utils.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.add_reader(r, callback) + loop.add_writer(w, callback) + loop.close() + self.assertFalse(loop.remove_reader(r)) + self.assertFalse(loop.remove_writer(w)) + + def test_add_fds_after_closing(self): + loop = self.create_event_loop() + callback = lambda: None + r, w = test_utils.socketpair() + self.addCleanup(r.close) + self.addCleanup(w.close) + loop.close() + with self.assertRaises(RuntimeError): + loop.add_reader(r, callback) + with self.assertRaises(RuntimeError): + loop.add_writer(w, callback) + class SubprocessTestsMixin: @@ -1370,20 +1369,13 @@ self.assertEqual(-signal.SIGKILL, returncode) def test_subprocess_exec(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) @@ -1396,20 +1388,13 @@ self.assertEqual(b'Python The Winner', proto.data[1]) def test_subprocess_interactive(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) @@ -1430,18 +1415,11 @@ self.check_terminated(proto.returncode) def test_subprocess_shell(self): - proto = None - transp = None - - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_shell( - functools.partial(MySubprocessProtocol, self.loop), - 'echo Python') - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'echo Python') + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) transp.get_pipe_transport(0).close() @@ -1452,33 +1430,20 @@ self.assertEqual(proto.data[2], b'') def test_subprocess_exitcode(self): - proto = None - - @asyncio.coroutine - def connect(): - nonlocal proto - transp, proto = yield from self.loop.subprocess_shell( - functools.partial(MySubprocessProtocol, self.loop), - 'exit 7', stdin=None, stdout=None, stderr=None) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.completed) self.assertEqual(7, proto.returncode) def test_subprocess_close_after_finish(self): - proto = None - transp = None - - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_shell( - functools.partial(MySubprocessProtocol, self.loop), - 'exit 7', stdin=None, stdout=None, stderr=None) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.assertIsNone(transp.get_pipe_transport(0)) self.assertIsNone(transp.get_pipe_transport(1)) self.assertIsNone(transp.get_pipe_transport(2)) @@ -1487,20 +1452,13 @@ self.assertIsNone(transp.close()) def test_subprocess_kill(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) transp.kill() @@ -1508,20 +1466,13 @@ self.check_killed(proto.returncode) def test_subprocess_terminate(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) transp.terminate() @@ -1530,20 +1481,13 @@ @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP") def test_subprocess_send_signal(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) transp.send_signal(signal.SIGHUP) @@ -1551,20 +1495,13 @@ self.assertEqual(-signal.SIGHUP, proto.returncode) def test_subprocess_stderr(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo2.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) stdin = transp.get_pipe_transport(0) @@ -1578,20 +1515,13 @@ self.assertEqual(0, proto.returncode) def test_subprocess_stderr_redirect_to_stdout(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo2.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog, stderr=subprocess.STDOUT) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog, stderr=subprocess.STDOUT) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) stdin = transp.get_pipe_transport(0) @@ -1608,20 +1538,13 @@ self.assertEqual(0, proto.returncode) def test_subprocess_close_client_stream(self): - proto = None - transp = None - prog = os.path.join(os.path.dirname(__file__), 'echo3.py') - @asyncio.coroutine - def connect(): - nonlocal proto, transp - transp, proto = yield from self.loop.subprocess_exec( - functools.partial(MySubprocessProtocol, self.loop), - sys.executable, prog) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + connect = self.loop.subprocess_exec( + functools.partial(MySubprocessProtocol, self.loop), + sys.executable, prog) + transp, proto = self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.connected) stdin = transp.get_pipe_transport(0) @@ -1647,20 +1570,13 @@ self.check_terminated(proto.returncode) def test_subprocess_wait_no_same_group(self): - proto = None - transp = None - - @asyncio.coroutine - def connect(): - nonlocal proto - # start the new process in a new session - transp, proto = yield from self.loop.subprocess_shell( - functools.partial(MySubprocessProtocol, self.loop), - 'exit 7', stdin=None, stdout=None, stderr=None, - start_new_session=True) - self.assertIsInstance(proto, MySubprocessProtocol) - - self.loop.run_until_complete(connect()) + # start the new process in a new session + connect = self.loop.subprocess_shell( + functools.partial(MySubprocessProtocol, self.loop), + 'exit 7', stdin=None, stdout=None, stderr=None, + start_new_session=True) + _, proto = yield self.loop.run_until_complete(connect) + self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.completed) self.assertEqual(7, proto.returncode) @@ -1741,6 +1657,9 @@ def test_create_datagram_endpoint(self): raise unittest.SkipTest( "IocpEventLoop does not have create_datagram_endpoint()") + + def test_remove_fds_after_closing(self): + raise unittest.SkipTest("IocpEventLoop does not have add_reader()") else: from asyncio import selectors @@ -1812,7 +1731,7 @@ return args args = () - h = asyncio.Handle(callback, args, unittest.mock.Mock()) + h = asyncio.Handle(callback, args, mock.Mock()) self.assertIs(h._callback, callback) self.assertIs(h._args, args) self.assertFalse(h._cancelled) @@ -1844,15 +1763,15 @@ def callback(): raise ValueError() - m_loop = unittest.mock.Mock() - m_loop.call_exception_handler = unittest.mock.Mock() + m_loop = mock.Mock() + m_loop.call_exception_handler = mock.Mock() h = asyncio.Handle(callback, (), m_loop) h._run() m_loop.call_exception_handler.assert_called_with({ 'message': test_utils.MockPattern('Exception in callback.*'), - 'exception': unittest.mock.ANY, + 'exception': mock.ANY, 'handle': h }) @@ -1862,7 +1781,7 @@ def test_hash(self): when = time.monotonic() h = asyncio.TimerHandle(when, lambda: False, (), - unittest.mock.Mock()) + mock.Mock()) self.assertEqual(hash(h), hash(when)) def test_timer(self): @@ -1871,7 +1790,7 @@ args = () when = time.monotonic() - h = asyncio.TimerHandle(when, callback, args, unittest.mock.Mock()) + h = asyncio.TimerHandle(when, callback, args, mock.Mock()) self.assertIs(h._callback, callback) self.assertIs(h._args, args) self.assertFalse(h._cancelled) @@ -1887,10 +1806,10 @@ self.assertRaises(AssertionError, asyncio.TimerHandle, None, callback, args, - unittest.mock.Mock()) + mock.Mock()) def test_timer_comparison(self): - loop = unittest.mock.Mock() + loop = mock.Mock() def callback(*args): return args @@ -1935,7 +1854,7 @@ class AbstractEventLoopTests(unittest.TestCase): def test_not_implemented(self): - f = unittest.mock.Mock() + f = mock.Mock() loop = asyncio.AbstractEventLoop() self.assertRaises( NotImplementedError, loop.run_forever) @@ -1995,13 +1914,13 @@ NotImplementedError, loop.remove_signal_handler, 1) self.assertRaises( NotImplementedError, loop.connect_read_pipe, f, - unittest.mock.sentinel.pipe) + mock.sentinel.pipe) self.assertRaises( NotImplementedError, loop.connect_write_pipe, f, - unittest.mock.sentinel.pipe) + mock.sentinel.pipe) self.assertRaises( NotImplementedError, loop.subprocess_shell, f, - unittest.mock.sentinel) + mock.sentinel) self.assertRaises( NotImplementedError, loop.subprocess_exec, f) @@ -2009,7 +1928,7 @@ class ProtocolsAbsTests(unittest.TestCase): def test_empty(self): - f = unittest.mock.Mock() + f = mock.Mock() p = asyncio.Protocol() self.assertIsNone(p.connection_made(f)) self.assertIsNone(p.connection_lost(f)) @@ -2055,7 +1974,7 @@ def test_get_event_loop_calls_set_event_loop(self): policy = asyncio.DefaultEventLoopPolicy() - with unittest.mock.patch.object( + with mock.patch.object( policy, "set_event_loop", wraps=policy.set_event_loop) as m_set_event_loop: @@ -2073,7 +1992,7 @@ policy.set_event_loop(None) self.assertRaises(AssertionError, policy.get_event_loop) - @unittest.mock.patch('asyncio.events.threading.current_thread') + @mock.patch('asyncio.events.threading.current_thread') def test_get_event_loop_thread(self, m_current_thread): def f(): diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -3,7 +3,7 @@ import concurrent.futures import threading import unittest -import unittest.mock +from unittest import mock import asyncio from asyncio import test_utils @@ -174,20 +174,20 @@ self.assertRaises(AssertionError, test) fut.cancel() - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_abandoned(self, m_log): fut = asyncio.Future(loop=self.loop) del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_result_unretrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_result(42) del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_result_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_result(42) @@ -195,7 +195,7 @@ del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_unretrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) @@ -203,7 +203,7 @@ test_utils.run_briefly(self.loop) self.assertTrue(m_log.error.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) @@ -211,7 +211,7 @@ del fut self.assertFalse(m_log.error.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_tb_logger_exception_result_retrieved(self, m_log): fut = asyncio.Future(loop=self.loop) fut.set_exception(RuntimeError('boom')) @@ -236,7 +236,7 @@ f2 = asyncio.wrap_future(f1) self.assertIs(f1, f2) - @unittest.mock.patch('asyncio.futures.events') + @mock.patch('asyncio.futures.events') def test_wrap_future_use_global_loop(self, m_events): def run(arg): return (arg, threading.get_ident()) diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py --- a/Lib/test/test_asyncio/test_locks.py +++ b/Lib/test/test_asyncio/test_locks.py @@ -1,7 +1,7 @@ """Tests for lock.py""" import unittest -import unittest.mock +from unittest import mock import re import asyncio @@ -27,7 +27,7 @@ self.loop.close() def test_ctor_loop(self): - loop = unittest.mock.Mock() + loop = mock.Mock() lock = asyncio.Lock(loop=loop) self.assertIs(lock._loop, loop) @@ -250,7 +250,7 @@ self.loop.close() def test_ctor_loop(self): - loop = unittest.mock.Mock() + loop = mock.Mock() ev = asyncio.Event(loop=loop) self.assertIs(ev._loop, loop) @@ -275,7 +275,7 @@ self.assertTrue(repr(ev).endswith('[set]>')) self.assertTrue(RGX_REPR.match(repr(ev))) - ev._waiters.append(unittest.mock.Mock()) + ev._waiters.append(mock.Mock()) self.assertTrue('waiters:1' in repr(ev)) self.assertTrue(RGX_REPR.match(repr(ev))) @@ -386,7 +386,7 @@ self.loop.close() def test_ctor_loop(self): - loop = unittest.mock.Mock() + loop = mock.Mock() cond = asyncio.Condition(loop=loop) self.assertIs(cond._loop, loop) @@ -644,11 +644,11 @@ self.loop.run_until_complete(cond.acquire()) self.assertTrue('locked' in repr(cond)) - cond._waiters.append(unittest.mock.Mock()) + cond._waiters.append(mock.Mock()) self.assertTrue('waiters:1' in repr(cond)) self.assertTrue(RGX_REPR.match(repr(cond))) - cond._waiters.append(unittest.mock.Mock()) + cond._waiters.append(mock.Mock()) self.assertTrue('waiters:2' in repr(cond)) self.assertTrue(RGX_REPR.match(repr(cond))) @@ -688,7 +688,7 @@ self.loop.close() def test_ctor_loop(self): - loop = unittest.mock.Mock() + loop = mock.Mock() sem = asyncio.Semaphore(loop=loop) self.assertIs(sem._loop, loop) @@ -717,11 +717,11 @@ self.assertTrue('waiters' not in repr(sem)) self.assertTrue(RGX_REPR.match(repr(sem))) - sem._waiters.append(unittest.mock.Mock()) + sem._waiters.append(mock.Mock()) self.assertTrue('waiters:1' in repr(sem)) self.assertTrue(RGX_REPR.match(repr(sem))) - sem._waiters.append(unittest.mock.Mock()) + sem._waiters.append(mock.Mock()) self.assertTrue('waiters:2' in repr(sem)) self.assertTrue(RGX_REPR.match(repr(sem))) diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -2,7 +2,7 @@ import socket import unittest -import unittest.mock +from unittest import mock import asyncio from asyncio.proactor_events import BaseProactorEventLoop @@ -16,10 +16,10 @@ def setUp(self): self.loop = test_utils.TestLoop() - self.proactor = unittest.mock.Mock() + self.proactor = mock.Mock() self.loop._proactor = self.proactor self.protocol = test_utils.make_test_protocol(asyncio.Protocol) - self.sock = unittest.mock.Mock(socket.socket) + self.sock = mock.Mock(socket.socket) def test_ctor(self): fut = asyncio.Future(loop=self.loop) @@ -56,7 +56,7 @@ self.assertRaises(AssertionError, tr._loop_reading, res) - tr.close = unittest.mock.Mock() + tr.close = mock.Mock() tr._read_fut = res tr._loop_reading(res) self.assertFalse(self.loop._proactor.recv.called) @@ -67,7 +67,7 @@ err = self.loop._proactor.recv.side_effect = ConnectionAbortedError() tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( err, @@ -78,7 +78,7 @@ tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._closing = True - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr._loop_reading() self.assertFalse(tr._fatal_error.called) @@ -86,7 +86,7 @@ self.loop._proactor.recv.side_effect = ConnectionAbortedError() tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._closing = False - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr._loop_reading() self.assertTrue(tr._fatal_error.called) @@ -95,8 +95,8 @@ tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._closing = False - tr._fatal_error = unittest.mock.Mock() - tr._force_close = unittest.mock.Mock() + tr._fatal_error = mock.Mock() + tr._force_close = mock.Mock() tr._loop_reading() self.assertFalse(tr._fatal_error.called) tr._force_close.assert_called_with(err) @@ -105,7 +105,7 @@ err = self.loop._proactor.recv.side_effect = (OSError()) tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( err, @@ -113,7 +113,7 @@ def test_write(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._loop_writing = unittest.mock.Mock() + tr._loop_writing = mock.Mock() tr.write(b'data') self.assertEqual(tr._buffer, None) tr._loop_writing.assert_called_with(data=b'data') @@ -125,8 +125,8 @@ def test_write_more(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._write_fut = unittest.mock.Mock() - tr._loop_writing = unittest.mock.Mock() + tr._write_fut = mock.Mock() + tr._loop_writing = mock.Mock() tr.write(b'data') self.assertEqual(tr._buffer, b'data') self.assertFalse(tr._loop_writing.called) @@ -139,11 +139,11 @@ self.loop._proactor.send.return_value.add_done_callback.\ assert_called_with(tr._loop_writing) - @unittest.mock.patch('asyncio.proactor_events.logger') + @mock.patch('asyncio.proactor_events.logger') def test_loop_writing_err(self, m_log): err = self.loop._proactor.send.side_effect = OSError() tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr._buffer = [b'da', b'ta'] tr._loop_writing() tr._fatal_error.assert_called_with( @@ -182,7 +182,7 @@ def test_abort(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._force_close = unittest.mock.Mock() + tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) @@ -201,7 +201,7 @@ def test_close_write_fut(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._write_fut = unittest.mock.Mock() + tr._write_fut = mock.Mock() tr.close() test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) @@ -213,10 +213,10 @@ test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_fatal_error(self, m_logging): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._force_close = unittest.mock.Mock() + tr._force_close = mock.Mock() tr._fatal_error(None) self.assertTrue(tr._force_close.called) self.assertTrue(m_logging.error.called) @@ -224,8 +224,8 @@ def test_force_close(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._buffer = [b'data'] - read_fut = tr._read_fut = unittest.mock.Mock() - write_fut = tr._write_fut = unittest.mock.Mock() + read_fut = tr._read_fut = mock.Mock() + write_fut = tr._write_fut = mock.Mock() tr._force_close(None) read_fut.cancel.assert_called_with() @@ -346,10 +346,10 @@ class BaseProactorEventLoopTests(unittest.TestCase): def setUp(self): - self.sock = unittest.mock.Mock(socket.socket) - self.proactor = unittest.mock.Mock() + self.sock = mock.Mock(socket.socket) + self.proactor = mock.Mock() - self.ssock, self.csock = unittest.mock.Mock(), unittest.mock.Mock() + self.ssock, self.csock = mock.Mock(), mock.Mock() class EventLoop(BaseProactorEventLoop): def _socketpair(s): @@ -357,11 +357,11 @@ self.loop = EventLoop(self.proactor) - @unittest.mock.patch.object(BaseProactorEventLoop, 'call_soon') - @unittest.mock.patch.object(BaseProactorEventLoop, '_socketpair') + @mock.patch.object(BaseProactorEventLoop, 'call_soon') + @mock.patch.object(BaseProactorEventLoop, '_socketpair') def test_ctor(self, socketpair, call_soon): ssock, csock = socketpair.return_value = ( - unittest.mock.Mock(), unittest.mock.Mock()) + mock.Mock(), mock.Mock()) loop = BaseProactorEventLoop(self.proactor) self.assertIs(loop._ssock, ssock) self.assertIs(loop._csock, csock) @@ -377,7 +377,7 @@ self.assertIsNone(self.loop._csock) def test_close(self): - self.loop._close_self_pipe = unittest.mock.Mock() + self.loop._close_self_pipe = mock.Mock() self.loop.close() self.assertTrue(self.loop._close_self_pipe.called) self.assertTrue(self.proactor.close.called) @@ -418,7 +418,7 @@ self.loop._loop_self_reading) def test_loop_self_reading_fut(self): - fut = unittest.mock.Mock() + fut = mock.Mock() self.loop._loop_self_reading(fut) self.assertTrue(fut.result.called) self.proactor.recv.assert_called_with(self.ssock, 4096) @@ -426,7 +426,7 @@ self.loop._loop_self_reading) def test_loop_self_reading_exception(self): - self.loop.close = unittest.mock.Mock() + self.loop.close = mock.Mock() self.proactor.recv.side_effect = OSError() self.assertRaises(OSError, self.loop._loop_self_reading) self.assertTrue(self.loop.close.called) @@ -438,10 +438,10 @@ def test_process_events(self): self.loop._process_events([]) - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_create_server(self, m_log): - pf = unittest.mock.Mock() - call_soon = self.loop.call_soon = unittest.mock.Mock() + pf = mock.Mock() + call_soon = self.loop.call_soon = mock.Mock() self.loop._start_serving(pf, self.sock) self.assertTrue(call_soon.called) @@ -452,10 +452,10 @@ self.proactor.accept.assert_called_with(self.sock) # conn - fut = unittest.mock.Mock() - fut.result.return_value = (unittest.mock.Mock(), unittest.mock.Mock()) + fut = mock.Mock() + fut.result.return_value = (mock.Mock(), mock.Mock()) - make_tr = self.loop._make_socket_transport = unittest.mock.Mock() + make_tr = self.loop._make_socket_transport = mock.Mock() loop(fut) self.assertTrue(fut.result.called) self.assertTrue(make_tr.called) @@ -467,8 +467,8 @@ self.assertTrue(m_log.error.called) def test_create_server_cancel(self): - pf = unittest.mock.Mock() - call_soon = self.loop.call_soon = unittest.mock.Mock() + pf = mock.Mock() + call_soon = self.loop.call_soon = mock.Mock() self.loop._start_serving(pf, self.sock) loop = call_soon.call_args[0][0] @@ -480,7 +480,7 @@ self.assertTrue(self.sock.close.called) def test_stop_serving(self): - sock = unittest.mock.Mock() + sock = mock.Mock() self.loop._stop_serving(sock) self.assertTrue(sock.close.called) self.proactor._stop_serving.assert_called_with(sock) diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py --- a/Lib/test/test_asyncio/test_queues.py +++ b/Lib/test/test_asyncio/test_queues.py @@ -1,7 +1,7 @@ """Tests for queues.py""" import unittest -import unittest.mock +from unittest import mock import asyncio from asyncio import test_utils @@ -72,7 +72,7 @@ self.assertTrue('_queue=[1]' in fn(q)) def test_ctor_loop(self): - loop = unittest.mock.Mock() + loop = mock.Mock() q = asyncio.Queue(loop=loop) self.assertIs(q._loop, loop) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1,13 +1,12 @@ """Tests for selector_events.py""" -import collections import errno import gc import pprint import socket import sys import unittest -import unittest.mock +from unittest import mock try: import ssl except ImportError: @@ -23,14 +22,14 @@ from asyncio.selector_events import _SelectorDatagramTransport -MOCK_ANY = unittest.mock.ANY +MOCK_ANY = mock.ANY class TestBaseSelectorEventLoop(BaseSelectorEventLoop): def _make_self_pipe(self): - self._ssock = unittest.mock.Mock() - self._csock = unittest.mock.Mock() + self._ssock = mock.Mock() + self._csock = mock.Mock() self._internal_fds += 1 @@ -41,34 +40,34 @@ class BaseSelectorEventLoopTests(unittest.TestCase): def setUp(self): - selector = unittest.mock.Mock() + selector = mock.Mock() self.loop = TestBaseSelectorEventLoop(selector) def test_make_socket_transport(self): - m = unittest.mock.Mock() - self.loop.add_reader = unittest.mock.Mock() + m = mock.Mock() + self.loop.add_reader = mock.Mock() transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) @unittest.skipIf(ssl is None, 'No ssl module') def test_make_ssl_transport(self): - m = unittest.mock.Mock() - self.loop.add_reader = unittest.mock.Mock() - self.loop.add_writer = unittest.mock.Mock() - self.loop.remove_reader = unittest.mock.Mock() - self.loop.remove_writer = unittest.mock.Mock() + m = mock.Mock() + self.loop.add_reader = mock.Mock() + self.loop.add_writer = mock.Mock() + self.loop.remove_reader = mock.Mock() + self.loop.remove_writer = mock.Mock() waiter = asyncio.Future(loop=self.loop) transport = self.loop._make_ssl_transport( m, asyncio.Protocol(), m, waiter) self.assertIsInstance(transport, _SelectorSslTransport) - @unittest.mock.patch('asyncio.selector_events.ssl', None) + @mock.patch('asyncio.selector_events.ssl', None) def test_make_ssl_transport_without_ssl_error(self): - m = unittest.mock.Mock() - self.loop.add_reader = unittest.mock.Mock() - self.loop.add_writer = unittest.mock.Mock() - self.loop.remove_reader = unittest.mock.Mock() - self.loop.remove_writer = unittest.mock.Mock() + m = mock.Mock() + self.loop.add_reader = mock.Mock() + self.loop.add_writer = mock.Mock() + self.loop.remove_reader = mock.Mock() + self.loop.remove_writer = mock.Mock() with self.assertRaises(RuntimeError): self.loop._make_ssl_transport(m, m, m, m) @@ -77,10 +76,10 @@ ssock.fileno.return_value = 7 csock = self.loop._csock csock.fileno.return_value = 1 - remove_reader = self.loop.remove_reader = unittest.mock.Mock() + remove_reader = self.loop.remove_reader = mock.Mock() self.loop._selector.close() - self.loop._selector = selector = unittest.mock.Mock() + self.loop._selector = selector = mock.Mock() self.loop.close() self.assertIsNone(self.loop._selector) self.assertIsNone(self.loop._csock) @@ -96,7 +95,7 @@ def test_close_no_selector(self): ssock = self.loop._ssock csock = self.loop._csock - remove_reader = self.loop.remove_reader = unittest.mock.Mock() + remove_reader = self.loop.remove_reader = mock.Mock() self.loop._selector.close() self.loop._selector = None @@ -126,15 +125,15 @@ self.assertRaises(OSError, self.loop._write_to_self) def test_sock_recv(self): - sock = unittest.mock.Mock() - self.loop._sock_recv = unittest.mock.Mock() + sock = mock.Mock() + self.loop._sock_recv = mock.Mock() f = self.loop.sock_recv(sock, 1024) self.assertIsInstance(f, asyncio.Future) self.loop._sock_recv.assert_called_with(f, False, sock, 1024) def test__sock_recv_canceled_fut(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) f.cancel() @@ -143,30 +142,30 @@ self.assertFalse(sock.recv.called) def test__sock_recv_unregister(self): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 f = asyncio.Future(loop=self.loop) f.cancel() - self.loop.remove_reader = unittest.mock.Mock() + self.loop.remove_reader = mock.Mock() self.loop._sock_recv(f, True, sock, 1024) self.assertEqual((10,), self.loop.remove_reader.call_args[0]) def test__sock_recv_tryagain(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.recv.side_effect = BlockingIOError - self.loop.add_reader = unittest.mock.Mock() + self.loop.add_reader = mock.Mock() self.loop._sock_recv(f, False, sock, 1024) self.assertEqual((10, self.loop._sock_recv, f, True, sock, 1024), self.loop.add_reader.call_args[0]) def test__sock_recv_exception(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 err = sock.recv.side_effect = OSError() @@ -174,8 +173,8 @@ self.assertIs(err, f.exception()) def test_sock_sendall(self): - sock = unittest.mock.Mock() - self.loop._sock_sendall = unittest.mock.Mock() + sock = mock.Mock() + self.loop._sock_sendall = mock.Mock() f = self.loop.sock_sendall(sock, b'data') self.assertIsInstance(f, asyncio.Future) @@ -184,8 +183,8 @@ self.loop._sock_sendall.call_args[0]) def test_sock_sendall_nodata(self): - sock = unittest.mock.Mock() - self.loop._sock_sendall = unittest.mock.Mock() + sock = mock.Mock() + self.loop._sock_sendall = mock.Mock() f = self.loop.sock_sendall(sock, b'') self.assertIsInstance(f, asyncio.Future) @@ -194,7 +193,7 @@ self.assertFalse(self.loop._sock_sendall.called) def test__sock_sendall_canceled_fut(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) f.cancel() @@ -203,23 +202,23 @@ self.assertFalse(sock.send.called) def test__sock_sendall_unregister(self): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 f = asyncio.Future(loop=self.loop) f.cancel() - self.loop.remove_writer = unittest.mock.Mock() + self.loop.remove_writer = mock.Mock() self.loop._sock_sendall(f, True, sock, b'data') self.assertEqual((10,), self.loop.remove_writer.call_args[0]) def test__sock_sendall_tryagain(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.send.side_effect = BlockingIOError - self.loop.add_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() self.loop._sock_sendall(f, False, sock, b'data') self.assertEqual( (10, self.loop._sock_sendall, f, True, sock, b'data'), @@ -227,11 +226,11 @@ def test__sock_sendall_interrupted(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.send.side_effect = InterruptedError - self.loop.add_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() self.loop._sock_sendall(f, False, sock, b'data') self.assertEqual( (10, self.loop._sock_sendall, f, True, sock, b'data'), @@ -239,7 +238,7 @@ def test__sock_sendall_exception(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 err = sock.send.side_effect = OSError() @@ -247,7 +246,7 @@ self.assertIs(f.exception(), err) def test__sock_sendall(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) sock.fileno.return_value = 10 @@ -258,13 +257,13 @@ self.assertIsNone(f.result()) def test__sock_sendall_partial(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) sock.fileno.return_value = 10 sock.send.return_value = 2 - self.loop.add_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() self.loop._sock_sendall(f, False, sock, b'data') self.assertFalse(f.done()) self.assertEqual( @@ -272,13 +271,13 @@ self.loop.add_writer.call_args[0]) def test__sock_sendall_none(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) sock.fileno.return_value = 10 sock.send.return_value = 0 - self.loop.add_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() self.loop._sock_sendall(f, False, sock, b'data') self.assertFalse(f.done()) self.assertEqual( @@ -286,8 +285,8 @@ self.loop.add_writer.call_args[0]) def test_sock_connect(self): - sock = unittest.mock.Mock() - self.loop._sock_connect = unittest.mock.Mock() + sock = mock.Mock() + self.loop._sock_connect = mock.Mock() f = self.loop.sock_connect(sock, ('127.0.0.1', 8080)) self.assertIsInstance(f, asyncio.Future) @@ -298,7 +297,7 @@ def test__sock_connect(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 self.loop._sock_connect(f, False, sock, ('127.0.0.1', 8080)) @@ -307,7 +306,7 @@ self.assertTrue(sock.connect.called) def test__sock_connect_canceled_fut(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) f.cancel() @@ -316,24 +315,24 @@ self.assertFalse(sock.connect.called) def test__sock_connect_unregister(self): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 f = asyncio.Future(loop=self.loop) f.cancel() - self.loop.remove_writer = unittest.mock.Mock() + self.loop.remove_writer = mock.Mock() self.loop._sock_connect(f, True, sock, ('127.0.0.1', 8080)) self.assertEqual((10,), self.loop.remove_writer.call_args[0]) def test__sock_connect_tryagain(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.getsockopt.return_value = errno.EAGAIN - self.loop.add_writer = unittest.mock.Mock() - self.loop.remove_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() + self.loop.remove_writer = mock.Mock() self.loop._sock_connect(f, True, sock, ('127.0.0.1', 8080)) self.assertEqual( @@ -343,17 +342,17 @@ def test__sock_connect_exception(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.getsockopt.return_value = errno.ENOTCONN - self.loop.remove_writer = unittest.mock.Mock() + self.loop.remove_writer = mock.Mock() self.loop._sock_connect(f, True, sock, ('127.0.0.1', 8080)) self.assertIsInstance(f.exception(), OSError) def test_sock_accept(self): - sock = unittest.mock.Mock() - self.loop._sock_accept = unittest.mock.Mock() + sock = mock.Mock() + self.loop._sock_accept = mock.Mock() f = self.loop.sock_accept(sock) self.assertIsInstance(f, asyncio.Future) @@ -363,9 +362,9 @@ def test__sock_accept(self): f = asyncio.Future(loop=self.loop) - conn = unittest.mock.Mock() + conn = mock.Mock() - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.accept.return_value = conn, ('127.0.0.1', 1000) @@ -375,7 +374,7 @@ self.assertEqual((False,), conn.setblocking.call_args[0]) def test__sock_accept_canceled_fut(self): - sock = unittest.mock.Mock() + sock = mock.Mock() f = asyncio.Future(loop=self.loop) f.cancel() @@ -384,23 +383,23 @@ self.assertFalse(sock.accept.called) def test__sock_accept_unregister(self): - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 f = asyncio.Future(loop=self.loop) f.cancel() - self.loop.remove_reader = unittest.mock.Mock() + self.loop.remove_reader = mock.Mock() self.loop._sock_accept(f, True, sock) self.assertEqual((10,), self.loop.remove_reader.call_args[0]) def test__sock_accept_tryagain(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 sock.accept.side_effect = BlockingIOError - self.loop.add_reader = unittest.mock.Mock() + self.loop.add_reader = mock.Mock() self.loop._sock_accept(f, False, sock) self.assertEqual( (10, self.loop._sock_accept, f, True, sock), @@ -408,7 +407,7 @@ def test__sock_accept_exception(self): f = asyncio.Future(loop=self.loop) - sock = unittest.mock.Mock() + sock = mock.Mock() sock.fileno.return_value = 10 err = sock.accept.side_effect = OSError() @@ -428,8 +427,8 @@ self.assertIsNone(w) def test_add_reader_existing(self): - reader = unittest.mock.Mock() - writer = unittest.mock.Mock() + reader = mock.Mock() + writer = mock.Mock() self.loop._selector.get_key.return_value = selectors.SelectorKey( 1, 1, selectors.EVENT_WRITE, (reader, writer)) cb = lambda: True @@ -445,7 +444,7 @@ self.assertEqual(writer, w) def test_add_reader_existing_writer(self): - writer = unittest.mock.Mock() + writer = mock.Mock() self.loop._selector.get_key.return_value = selectors.SelectorKey( 1, 1, selectors.EVENT_WRITE, (None, writer)) cb = lambda: True @@ -467,8 +466,8 @@ self.assertTrue(self.loop._selector.unregister.called) def test_remove_reader_read_write(self): - reader = unittest.mock.Mock() - writer = unittest.mock.Mock() + reader = mock.Mock() + writer = mock.Mock() self.loop._selector.get_key.return_value = selectors.SelectorKey( 1, 1, selectors.EVENT_READ | selectors.EVENT_WRITE, (reader, writer)) @@ -498,8 +497,8 @@ self.assertEqual(cb, w._callback) def test_add_writer_existing(self): - reader = unittest.mock.Mock() - writer = unittest.mock.Mock() + reader = mock.Mock() + writer = mock.Mock() self.loop._selector.get_key.return_value = selectors.SelectorKey( 1, 1, selectors.EVENT_READ, (reader, writer)) cb = lambda: True @@ -522,8 +521,8 @@ self.assertTrue(self.loop._selector.unregister.called) def test_remove_writer_read_write(self): - reader = unittest.mock.Mock() - writer = unittest.mock.Mock() + reader = mock.Mock() + writer = mock.Mock() self.loop._selector.get_key.return_value = selectors.SelectorKey( 1, 1, selectors.EVENT_READ | selectors.EVENT_WRITE, (reader, writer)) @@ -541,10 +540,10 @@ self.loop.remove_writer(1)) def test_process_events_read(self): - reader = unittest.mock.Mock() + reader = mock.Mock() reader._cancelled = False - self.loop._add_callback = unittest.mock.Mock() + self.loop._add_callback = mock.Mock() self.loop._process_events( [(selectors.SelectorKey( 1, 1, selectors.EVENT_READ, (reader, None)), @@ -553,10 +552,10 @@ self.loop._add_callback.assert_called_with(reader) def test_process_events_read_cancelled(self): - reader = unittest.mock.Mock() + reader = mock.Mock() reader.cancelled = True - self.loop.remove_reader = unittest.mock.Mock() + self.loop.remove_reader = mock.Mock() self.loop._process_events( [(selectors.SelectorKey( 1, 1, selectors.EVENT_READ, (reader, None)), @@ -564,10 +563,10 @@ self.loop.remove_reader.assert_called_with(1) def test_process_events_write(self): - writer = unittest.mock.Mock() + writer = mock.Mock() writer._cancelled = False - self.loop._add_callback = unittest.mock.Mock() + self.loop._add_callback = mock.Mock() self.loop._process_events( [(selectors.SelectorKey(1, 1, selectors.EVENT_WRITE, (None, writer)), @@ -575,9 +574,9 @@ self.loop._add_callback.assert_called_with(writer) def test_process_events_write_cancelled(self): - writer = unittest.mock.Mock() + writer = mock.Mock() writer.cancelled = True - self.loop.remove_writer = unittest.mock.Mock() + self.loop.remove_writer = mock.Mock() self.loop._process_events( [(selectors.SelectorKey(1, 1, selectors.EVENT_WRITE, @@ -591,7 +590,7 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.Protocol) - self.sock = unittest.mock.Mock(socket.socket) + self.sock = mock.Mock(socket.socket) self.sock.fileno.return_value = 7 def test_ctor(self): @@ -602,7 +601,7 @@ def test_abort(self): tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) - tr._force_close = unittest.mock.Mock() + tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) @@ -632,8 +631,8 @@ def test_force_close(self): tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) tr._buffer.extend(b'1') - self.loop.add_reader(7, unittest.mock.sentinel) - self.loop.add_writer(7, unittest.mock.sentinel) + self.loop.add_reader(7, mock.sentinel) + self.loop.add_writer(7, mock.sentinel) tr._force_close(None) self.assertTrue(tr._closing) @@ -646,11 +645,11 @@ self.assertFalse(self.loop.readers) self.assertEqual(1, self.loop.remove_reader_count[7]) - @unittest.mock.patch('asyncio.log.logger.error') + @mock.patch('asyncio.log.logger.error') def test_fatal_error(self, m_exc): exc = OSError() tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) - tr._force_close = unittest.mock.Mock() + tr._force_close = mock.Mock() tr._fatal_error(exc) m_exc.assert_called_with( @@ -682,7 +681,7 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.Protocol) - self.sock = unittest.mock.Mock(socket.socket) + self.sock = mock.Mock(socket.socket) self.sock_fd = self.sock.fileno.return_value = 7 def test_ctor(self): @@ -724,7 +723,7 @@ def test_read_ready_eof(self): transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport.close = unittest.mock.Mock() + transport.close = mock.Mock() self.sock.recv.return_value = b'' transport._read_ready() @@ -735,7 +734,7 @@ def test_read_ready_eof_keep_open(self): transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport.close = unittest.mock.Mock() + transport.close = mock.Mock() self.sock.recv.return_value = b'' self.protocol.eof_received.return_value = True @@ -744,45 +743,45 @@ self.protocol.eof_received.assert_called_with() self.assertFalse(transport.close.called) - @unittest.mock.patch('logging.exception') + @mock.patch('logging.exception') def test_read_ready_tryagain(self, m_exc): self.sock.recv.side_effect = BlockingIOError transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() self.assertFalse(transport._fatal_error.called) - @unittest.mock.patch('logging.exception') + @mock.patch('logging.exception') def test_read_ready_tryagain_interrupted(self, m_exc): self.sock.recv.side_effect = InterruptedError transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() self.assertFalse(transport._fatal_error.called) - @unittest.mock.patch('logging.exception') + @mock.patch('logging.exception') def test_read_ready_conn_reset(self, m_exc): err = self.sock.recv.side_effect = ConnectionResetError() transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._force_close = unittest.mock.Mock() + transport._force_close = mock.Mock() transport._read_ready() transport._force_close.assert_called_with(err) - @unittest.mock.patch('logging.exception') + @mock.patch('logging.exception') def test_read_ready_err(self, m_exc): err = self.sock.recv.side_effect = OSError() transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() transport._fatal_error.assert_called_with( @@ -891,14 +890,14 @@ self.loop.assert_writer(7, transport._write_ready) self.assertEqual(list_to_buffer([b'data']), transport._buffer) - @unittest.mock.patch('asyncio.selector_events.logger') + @mock.patch('asyncio.selector_events.logger') def test_write_exception(self, m_log): err = self.sock.send.side_effect = OSError() data = b'data' transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport.write(data) transport._fatal_error.assert_called_with( err, @@ -1002,17 +1001,17 @@ transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._buffer.extend(b'data') transport._write_ready() transport._fatal_error.assert_called_with( err, 'Fatal write error on socket transport') - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.base_events.logger') def test_write_ready_exception_and_close(self, m_log): self.sock.send.side_effect = OSError() - remove_writer = self.loop.remove_writer = unittest.mock.Mock() + remove_writer = self.loop.remove_writer = mock.Mock() transport = _SelectorSocketTransport( self.loop, self.sock, self.protocol) @@ -1053,11 +1052,11 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.Protocol) - self.sock = unittest.mock.Mock(socket.socket) + self.sock = mock.Mock(socket.socket) self.sock.fileno.return_value = 7 - self.sslsock = unittest.mock.Mock() + self.sslsock = mock.Mock() self.sslsock.fileno.return_value = 1 - self.sslcontext = unittest.mock.Mock() + self.sslcontext = mock.Mock() self.sslcontext.wrap_socket.return_value = self.sslsock def _make_one(self, create_waiter=None): @@ -1162,7 +1161,7 @@ transport.write(b'data') self.assertEqual(transport._conn_lost, 2) - @unittest.mock.patch('asyncio.selector_events.logger') + @mock.patch('asyncio.selector_events.logger') def test_write_exception(self, m_log): transport = self._make_one() transport._conn_lost = 1 @@ -1182,11 +1181,11 @@ self.assertEqual((b'data',), self.protocol.data_received.call_args[0]) def test_read_ready_write_wants_read(self): - self.loop.add_writer = unittest.mock.Mock() + self.loop.add_writer = mock.Mock() self.sslsock.recv.side_effect = BlockingIOError transport = self._make_one() transport._write_wants_read = True - transport._write_ready = unittest.mock.Mock() + transport._write_ready = mock.Mock() transport._buffer.extend(b'data') transport._read_ready() @@ -1198,7 +1197,7 @@ def test_read_ready_recv_eof(self): self.sslsock.recv.return_value = b'' transport = self._make_one() - transport.close = unittest.mock.Mock() + transport.close = mock.Mock() transport._read_ready() transport.close.assert_called_with() self.protocol.eof_received.assert_called_with() @@ -1206,7 +1205,7 @@ def test_read_ready_recv_conn_reset(self): err = self.sslsock.recv.side_effect = ConnectionResetError() transport = self._make_one() - transport._force_close = unittest.mock.Mock() + transport._force_close = mock.Mock() transport._read_ready() transport._force_close.assert_called_with(err) @@ -1226,8 +1225,8 @@ self.assertFalse(self.protocol.data_received.called) def test_read_ready_recv_write(self): - self.loop.remove_reader = unittest.mock.Mock() - self.loop.add_writer = unittest.mock.Mock() + self.loop.remove_reader = mock.Mock() + self.loop.add_writer = mock.Mock() self.sslsock.recv.side_effect = ssl.SSLWantWriteError transport = self._make_one() transport._read_ready() @@ -1241,7 +1240,7 @@ def test_read_ready_recv_exc(self): err = self.sslsock.recv.side_effect = OSError() transport = self._make_one() - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() transport._fatal_error.assert_called_with( err, @@ -1313,7 +1312,7 @@ transport = self._make_one() transport._buffer = list_to_buffer([b'data']) - self.loop.remove_writer = unittest.mock.Mock() + self.loop.remove_writer = mock.Mock() self.sslsock.send.side_effect = ssl.SSLWantReadError transport._write_ready() self.assertFalse(self.protocol.data_received.called) @@ -1325,7 +1324,7 @@ transport = self._make_one() transport._buffer = list_to_buffer([b'data']) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._write_ready() transport._fatal_error.assert_called_with( err, @@ -1333,11 +1332,11 @@ self.assertEqual(list_to_buffer(), transport._buffer) def test_write_ready_read_wants_write(self): - self.loop.add_reader = unittest.mock.Mock() + self.loop.add_reader = mock.Mock() self.sslsock.send.side_effect = BlockingIOError transport = self._make_one() transport._read_wants_write = True - transport._read_ready = unittest.mock.Mock() + transport._read_ready = mock.Mock() transport._write_ready() self.assertFalse(transport._read_wants_write) @@ -1374,11 +1373,11 @@ class SelectorSslWithoutSslTransportTests(unittest.TestCase): - @unittest.mock.patch('asyncio.selector_events.ssl', None) + @mock.patch('asyncio.selector_events.ssl', None) def test_ssl_transport_requires_ssl_module(self): - Mock = unittest.mock.Mock + Mock = mock.Mock with self.assertRaises(RuntimeError): - transport = _SelectorSslTransport(Mock(), Mock(), Mock(), Mock()) + _SelectorSslTransport(Mock(), Mock(), Mock(), Mock()) class SelectorDatagramTransportTests(unittest.TestCase): @@ -1386,7 +1385,7 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.DatagramProtocol) - self.sock = unittest.mock.Mock(spec_set=socket.socket) + self.sock = mock.Mock(spec_set=socket.socket) self.sock.fileno.return_value = 7 def test_read_ready(self): @@ -1404,7 +1403,7 @@ self.loop, self.sock, self.protocol) self.sock.recvfrom.side_effect = BlockingIOError - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() self.assertFalse(transport._fatal_error.called) @@ -1414,7 +1413,7 @@ self.loop, self.sock, self.protocol) err = self.sock.recvfrom.side_effect = RuntimeError() - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() transport._fatal_error.assert_called_with( @@ -1426,7 +1425,7 @@ self.loop, self.sock, self.protocol) err = self.sock.recvfrom.side_effect = OSError() - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._read_ready() self.assertFalse(transport._fatal_error.called) @@ -1518,14 +1517,14 @@ self.assertEqual( [(b'data', ('0.0.0.0', 12345))], list(transport._buffer)) - @unittest.mock.patch('asyncio.selector_events.logger') + @mock.patch('asyncio.selector_events.logger') def test_sendto_exception(self, m_log): data = b'data' err = self.sock.sendto.side_effect = RuntimeError() transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport.sendto(data, ()) self.assertTrue(transport._fatal_error.called) @@ -1549,7 +1548,7 @@ transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport.sendto(data, ()) self.assertEqual(transport._conn_lost, 0) @@ -1562,7 +1561,7 @@ transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport.sendto(data) self.assertFalse(transport._fatal_error.called) @@ -1643,7 +1642,7 @@ transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1656,7 +1655,7 @@ transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1667,14 +1666,14 @@ transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) - transport._fatal_error = unittest.mock.Mock() + transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() self.assertFalse(transport._fatal_error.called) self.assertTrue(self.protocol.error_received.called) - @unittest.mock.patch('asyncio.base_events.logger.error') + @mock.patch('asyncio.base_events.logger.error') def test_fatal_error_connected(self, m_exc): transport = _SelectorDatagramTransport( self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -1,10 +1,9 @@ """Tests for streams.py.""" -import functools import gc import socket import unittest -import unittest.mock +from unittest import mock try: import ssl except ImportError: @@ -29,7 +28,7 @@ self.loop.close() gc.collect() - @unittest.mock.patch('asyncio.streams.events') + @mock.patch('asyncio.streams.events') def test_ctor_global_loop(self, m_events): stream = asyncio.StreamReader() self.assertIs(stream._loop, m_events.get_event_loop.return_value) diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -830,7 +830,7 @@ v = yield from f self.assertEqual(v, 'a') - res = loop.run_until_complete(asyncio.Task(foo(), loop=loop)) + loop.run_until_complete(asyncio.Task(foo(), loop=loop)) def test_as_completed_reverse_wait(self): @@ -964,13 +964,9 @@ loop = test_utils.TestLoop(gen) self.addCleanup(loop.close) - sleepfut = None - @asyncio.coroutine def sleep(dt): - nonlocal sleepfut - sleepfut = asyncio.sleep(dt, loop=loop) - yield from sleepfut + yield from asyncio.sleep(dt, loop=loop) @asyncio.coroutine def doit(): diff --git a/Lib/test/test_asyncio/test_transports.py b/Lib/test/test_asyncio/test_transports.py --- a/Lib/test/test_asyncio/test_transports.py +++ b/Lib/test/test_asyncio/test_transports.py @@ -1,7 +1,7 @@ """Tests for transports.py.""" import unittest -import unittest.mock +from unittest import mock import asyncio from asyncio import transports @@ -23,7 +23,7 @@ def test_writelines(self): transport = asyncio.Transport() - transport.write = unittest.mock.Mock() + transport.write = mock.Mock() transport.writelines([b'line1', bytearray(b'line2'), @@ -70,7 +70,7 @@ return 512 transport = MyTransport() - transport._protocol = unittest.mock.Mock() + transport._protocol = mock.Mock() self.assertFalse(transport._protocol_paused) diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -13,7 +13,7 @@ import tempfile import threading import unittest -import unittest.mock +from unittest import mock if sys.platform == 'win32': raise unittest.SkipTest('UNIX only') @@ -25,7 +25,7 @@ from asyncio import unix_events -MOCK_ANY = unittest.mock.ANY +MOCK_ANY = mock.ANY @unittest.skipUnless(signal, 'Signals are not supported') @@ -48,15 +48,15 @@ self.loop._handle_signal(signal.NSIG + 1, ()) def test_handle_signal_cancelled_handler(self): - h = asyncio.Handle(unittest.mock.Mock(), (), - loop=unittest.mock.Mock()) + h = asyncio.Handle(mock.Mock(), (), + loop=mock.Mock()) h.cancel() self.loop._signal_handlers[signal.NSIG + 1] = h - self.loop.remove_signal_handler = unittest.mock.Mock() + self.loop.remove_signal_handler = mock.Mock() self.loop._handle_signal(signal.NSIG + 1, ()) self.loop.remove_signal_handler.assert_called_with(signal.NSIG + 1) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_add_signal_handler_setup_error(self, m_signal): m_signal.NSIG = signal.NSIG m_signal.set_wakeup_fd.side_effect = ValueError @@ -66,7 +66,7 @@ self.loop.add_signal_handler, signal.SIGINT, lambda: True) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_add_signal_handler(self, m_signal): m_signal.NSIG = signal.NSIG @@ -76,7 +76,7 @@ self.assertIsInstance(h, asyncio.Handle) self.assertEqual(h._callback, cb) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_add_signal_handler_install_error(self, m_signal): m_signal.NSIG = signal.NSIG @@ -94,8 +94,8 @@ self.loop.add_signal_handler, signal.SIGINT, lambda: True) - @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') def test_add_signal_handler_install_error2(self, m_logging, m_signal): m_signal.NSIG = signal.NSIG @@ -111,8 +111,8 @@ self.assertFalse(m_logging.info.called) self.assertEqual(1, m_signal.set_wakeup_fd.call_count) - @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') def test_add_signal_handler_install_error3(self, m_logging, m_signal): class Err(OSError): errno = errno.EINVAL @@ -126,7 +126,7 @@ self.assertFalse(m_logging.info.called) self.assertEqual(2, m_signal.set_wakeup_fd.call_count) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_remove_signal_handler(self, m_signal): m_signal.NSIG = signal.NSIG @@ -139,7 +139,7 @@ self.assertEqual( (signal.SIGHUP, m_signal.SIG_DFL), m_signal.signal.call_args[0]) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_remove_signal_handler_2(self, m_signal): m_signal.NSIG = signal.NSIG m_signal.SIGINT = signal.SIGINT @@ -156,8 +156,8 @@ (signal.SIGINT, m_signal.default_int_handler), m_signal.signal.call_args[0]) - @unittest.mock.patch('asyncio.unix_events.signal') - @unittest.mock.patch('asyncio.base_events.logger') + @mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.base_events.logger') def test_remove_signal_handler_cleanup_error(self, m_logging, m_signal): m_signal.NSIG = signal.NSIG self.loop.add_signal_handler(signal.SIGHUP, lambda: True) @@ -167,7 +167,7 @@ self.loop.remove_signal_handler(signal.SIGHUP) self.assertTrue(m_logging.info) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_remove_signal_handler_error(self, m_signal): m_signal.NSIG = signal.NSIG self.loop.add_signal_handler(signal.SIGHUP, lambda: True) @@ -177,7 +177,7 @@ self.assertRaises( OSError, self.loop.remove_signal_handler, signal.SIGHUP) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_remove_signal_handler_error2(self, m_signal): m_signal.NSIG = signal.NSIG self.loop.add_signal_handler(signal.SIGHUP, lambda: True) @@ -189,7 +189,7 @@ self.assertRaises( RuntimeError, self.loop.remove_signal_handler, signal.SIGHUP) - @unittest.mock.patch('asyncio.unix_events.signal') + @mock.patch('asyncio.unix_events.signal') def test_close(self, m_signal): m_signal.NSIG = signal.NSIG @@ -291,16 +291,16 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.Protocol) - self.pipe = unittest.mock.Mock(spec_set=io.RawIOBase) + self.pipe = mock.Mock(spec_set=io.RawIOBase) self.pipe.fileno.return_value = 5 - fcntl_patcher = unittest.mock.patch('fcntl.fcntl') + fcntl_patcher = mock.patch('fcntl.fcntl') fcntl_patcher.start() self.addCleanup(fcntl_patcher.stop) - fstat_patcher = unittest.mock.patch('os.fstat') + fstat_patcher = mock.patch('os.fstat') m_fstat = fstat_patcher.start() - st = unittest.mock.Mock() + st = mock.Mock() st.st_mode = stat.S_IFIFO m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) @@ -319,7 +319,7 @@ test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test__read_ready(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) @@ -329,7 +329,7 @@ m_read.assert_called_with(5, tr.max_size) self.protocol.data_received.assert_called_with(b'data') - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test__read_ready_eof(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) @@ -342,7 +342,7 @@ self.protocol.eof_received.assert_called_with() self.protocol.connection_lost.assert_called_with(None) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test__read_ready_blocked(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) @@ -353,14 +353,14 @@ test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.data_received.called) - @unittest.mock.patch('asyncio.log.logger.error') - @unittest.mock.patch('os.read') + @mock.patch('asyncio.log.logger.error') + @mock.patch('os.read') def test__read_ready_error(self, m_read, m_logexc): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) err = OSError() m_read.side_effect = err - tr._close = unittest.mock.Mock() + tr._close = mock.Mock() tr._read_ready() m_read.assert_called_with(5, tr.max_size) @@ -371,17 +371,17 @@ '\nprotocol:.*\ntransport:.*'), exc_info=(OSError, MOCK_ANY, MOCK_ANY)) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test_pause_reading(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) - m = unittest.mock.Mock() + m = mock.Mock() self.loop.add_reader(5, m) tr.pause_reading() self.assertFalse(self.loop.readers) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test_resume_reading(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) @@ -389,26 +389,26 @@ tr.resume_reading() self.loop.assert_reader(5, tr._read_ready) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test_close(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) - tr._close = unittest.mock.Mock() + tr._close = mock.Mock() tr.close() tr._close.assert_called_with(None) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test_close_already_closing(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) tr._closing = True - tr._close = unittest.mock.Mock() + tr._close = mock.Mock() tr.close() self.assertFalse(tr._close.called) - @unittest.mock.patch('os.read') + @mock.patch('os.read') def test__close(self, m_read): tr = unix_events._UnixReadPipeTransport( self.loop, self.pipe, self.protocol) @@ -459,16 +459,16 @@ def setUp(self): self.loop = test_utils.TestLoop() self.protocol = test_utils.make_test_protocol(asyncio.BaseProtocol) - self.pipe = unittest.mock.Mock(spec_set=io.RawIOBase) + self.pipe = mock.Mock(spec_set=io.RawIOBase) self.pipe.fileno.return_value = 5 - fcntl_patcher = unittest.mock.patch('fcntl.fcntl') + fcntl_patcher = mock.patch('fcntl.fcntl') fcntl_patcher.start() self.addCleanup(fcntl_patcher.stop) - fstat_patcher = unittest.mock.patch('os.fstat') + fstat_patcher = mock.patch('os.fstat') m_fstat = fstat_patcher.start() - st = unittest.mock.Mock() + st = mock.Mock() st.st_mode = stat.S_IFSOCK m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) @@ -493,7 +493,7 @@ self.loop, self.pipe, self.protocol) self.assertTrue(tr.can_write_eof()) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -504,7 +504,7 @@ self.assertFalse(self.loop.writers) self.assertEqual([], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write_no_data(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -514,7 +514,7 @@ self.assertFalse(self.loop.writers) self.assertEqual([], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write_partial(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -525,7 +525,7 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'ta'], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write_buffer(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -537,7 +537,7 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'previous', b'data'], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write_again(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -548,15 +548,15 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'data'], tr._buffer) - @unittest.mock.patch('asyncio.unix_events.logger') - @unittest.mock.patch('os.write') + @mock.patch('asyncio.unix_events.logger') + @mock.patch('os.write') def test_write_err(self, m_write, m_log): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) err = OSError() m_write.side_effect = err - tr._fatal_error = unittest.mock.Mock() + tr._fatal_error = mock.Mock() tr.write(b'data') m_write.assert_called_with(5, b'data') self.assertFalse(self.loop.writers) @@ -576,7 +576,7 @@ m_log.warning.assert_called_with( 'pipe closed by peer or os.write(pipe, data) raised exception.') - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_write_close(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -597,7 +597,7 @@ test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(None) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test__write_ready(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -609,7 +609,7 @@ self.assertFalse(self.loop.writers) self.assertEqual([], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test__write_ready_partial(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -622,7 +622,7 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'a'], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test__write_ready_again(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -635,7 +635,7 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'data'], tr._buffer) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test__write_ready_empty(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -648,8 +648,8 @@ self.loop.assert_writer(5, tr._write_ready) self.assertEqual([b'data'], tr._buffer) - @unittest.mock.patch('asyncio.log.logger.error') - @unittest.mock.patch('os.write') + @mock.patch('asyncio.log.logger.error') + @mock.patch('os.write') def test__write_ready_err(self, m_write, m_logexc): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -672,7 +672,7 @@ test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(err) - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test__write_ready_closing(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -689,7 +689,7 @@ self.protocol.connection_lost.assert_called_with(None) self.pipe.close.assert_called_with() - @unittest.mock.patch('os.write') + @mock.patch('os.write') def test_abort(self, m_write): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) @@ -742,7 +742,7 @@ tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) - tr.write_eof = unittest.mock.Mock() + tr.write_eof = mock.Mock() tr.close() tr.write_eof.assert_called_with() @@ -750,7 +750,7 @@ tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) - tr.write_eof = unittest.mock.Mock() + tr.write_eof = mock.Mock() tr._closing = True tr.close() self.assertFalse(tr.write_eof.called) @@ -777,7 +777,7 @@ class AbstractChildWatcherTests(unittest.TestCase): def test_not_implemented(self): - f = unittest.mock.Mock() + f = mock.Mock() watcher = asyncio.AbstractChildWatcher() self.assertRaises( NotImplementedError, watcher.add_child_handler, f, f) @@ -796,7 +796,7 @@ class BaseChildWatcherTests(unittest.TestCase): def test_not_implemented(self): - f = unittest.mock.Mock() + f = mock.Mock() watcher = unix_events.BaseChildWatcher() self.assertRaises( NotImplementedError, watcher._do_waitpid, f) @@ -813,14 +813,14 @@ class ChildWatcherTestsMixin: - ignore_warnings = unittest.mock.patch.object(log.logger, "warning") + ignore_warnings = mock.patch.object(log.logger, "warning") def setUp(self): self.loop = test_utils.TestLoop() self.running = False self.zombies = {} - with unittest.mock.patch.object( + with mock.patch.object( self.loop, "add_signal_handler") as self.m_add_signal_handler: self.watcher = self.create_watcher() self.watcher.attach_loop(self.loop) @@ -864,8 +864,8 @@ def waitpid_mocks(func): def wrapped_func(self): def patch(target, wrapper): - return unittest.mock.patch(target, wraps=wrapper, - new_callable=unittest.mock.Mock) + return mock.patch(target, wraps=wrapper, + new_callable=mock.Mock) with patch('os.WTERMSIG', self.WTERMSIG) as m_WTERMSIG, \ patch('os.WEXITSTATUS', self.WEXITSTATUS) as m_WEXITSTATUS, \ @@ -881,7 +881,7 @@ @waitpid_mocks def test_sigchld(self, m): # register a child - callback = unittest.mock.Mock() + callback = mock.Mock() with self.watcher: self.running = True @@ -941,8 +941,8 @@ @waitpid_mocks def test_sigchld_two_children(self, m): - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() # register child 1 with self.watcher: @@ -1045,8 +1045,8 @@ @waitpid_mocks def test_sigchld_two_children_terminating_together(self, m): - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() # register child 1 with self.watcher: @@ -1115,7 +1115,7 @@ @waitpid_mocks def test_sigchld_race_condition(self, m): # register a child - callback = unittest.mock.Mock() + callback = mock.Mock() with self.watcher: # child terminates before being registered @@ -1136,8 +1136,8 @@ @waitpid_mocks def test_sigchld_replace_handler(self, m): - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() # register a child with self.watcher: @@ -1189,7 +1189,7 @@ @waitpid_mocks def test_sigchld_remove_handler(self, m): - callback = unittest.mock.Mock() + callback = mock.Mock() # register a child with self.watcher: @@ -1221,7 +1221,7 @@ @waitpid_mocks def test_sigchld_unknown_status(self, m): - callback = unittest.mock.Mock() + callback = mock.Mock() # register a child with self.watcher: @@ -1258,9 +1258,9 @@ @waitpid_mocks def test_remove_child_handler(self, m): - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() - callback3 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() + callback3 = mock.Mock() # register children with self.watcher: @@ -1291,7 +1291,7 @@ @waitpid_mocks def test_sigchld_unhandled_exception(self, m): - callback = unittest.mock.Mock() + callback = mock.Mock() # register a child with self.watcher: @@ -1301,8 +1301,8 @@ # raise an exception m.waitpid.side_effect = ValueError - with unittest.mock.patch.object(log.logger, - 'error') as m_error: + with mock.patch.object(log.logger, + 'error') as m_error: self.assertEqual(self.watcher._sig_chld(), None) self.assertTrue(m_error.called) @@ -1310,7 +1310,7 @@ @waitpid_mocks def test_sigchld_child_reaped_elsewhere(self, m): # register a child - callback = unittest.mock.Mock() + callback = mock.Mock() with self.watcher: self.running = True @@ -1346,8 +1346,8 @@ @waitpid_mocks def test_sigchld_unknown_pid_during_registration(self, m): # register two children - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() with self.ignore_warnings, self.watcher: self.running = True @@ -1367,7 +1367,7 @@ @waitpid_mocks def test_set_loop(self, m): # register a child - callback = unittest.mock.Mock() + callback = mock.Mock() with self.watcher: self.running = True @@ -1376,19 +1376,16 @@ # attach a new loop old_loop = self.loop self.loop = test_utils.TestLoop() + patch = mock.patch.object - with unittest.mock.patch.object( - old_loop, - "remove_signal_handler") as m_old_remove_signal_handler, \ - unittest.mock.patch.object( - self.loop, - "add_signal_handler") as m_new_add_signal_handler: + with patch(old_loop, "remove_signal_handler") as m_old_remove, \ + patch(self.loop, "add_signal_handler") as m_new_add: self.watcher.attach_loop(self.loop) - m_old_remove_signal_handler.assert_called_once_with( + m_old_remove.assert_called_once_with( signal.SIGCHLD) - m_new_add_signal_handler.assert_called_once_with( + m_new_add.assert_called_once_with( signal.SIGCHLD, self.watcher._sig_chld) # child terminates @@ -1401,9 +1398,9 @@ @waitpid_mocks def test_set_loop_race_condition(self, m): # register 3 children - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() - callback3 = unittest.mock.Mock() + callback1 = mock.Mock() + callback2 = mock.Mock() + callback3 = mock.Mock() with self.watcher: self.running = True @@ -1415,7 +1412,7 @@ old_loop = self.loop self.loop = None - with unittest.mock.patch.object( + with mock.patch.object( old_loop, "remove_signal_handler") as m_remove_signal_handler: self.watcher.attach_loop(None) @@ -1435,7 +1432,7 @@ # attach a new loop self.loop = test_utils.TestLoop() - with unittest.mock.patch.object( + with mock.patch.object( self.loop, "add_signal_handler") as m_add_signal_handler: self.watcher.attach_loop(self.loop) @@ -1461,8 +1458,7 @@ @waitpid_mocks def test_close(self, m): # register two children - callback1 = unittest.mock.Mock() - callback2 = unittest.mock.Mock() + callback1 = mock.Mock() with self.watcher: self.running = True @@ -1479,7 +1475,7 @@ if isinstance(self.watcher, asyncio.FastChildWatcher): self.assertEqual(len(self.watcher._zombies), 1) - with unittest.mock.patch.object( + with mock.patch.object( self.loop, "remove_signal_handler") as m_remove_signal_handler: diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -8,7 +8,6 @@ import _winapi import asyncio -from asyncio import test_utils from asyncio import _overlapped from asyncio import windows_events @@ -50,7 +49,7 @@ ADDRESS = r'\\.\pipe\test_double_bind-%s' % os.getpid() server1 = windows_events.PipeServer(ADDRESS) with self.assertRaises(PermissionError): - server2 = windows_events.PipeServer(ADDRESS) + windows_events.PipeServer(ADDRESS) server1.close() def test_pipe(self): diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py --- a/Lib/test/test_asyncio/test_windows_utils.py +++ b/Lib/test/test_asyncio/test_windows_utils.py @@ -1,9 +1,11 @@ """Tests for window_utils""" +import socket import sys import test.support import unittest -import unittest.mock +from test.support import IPV6_ENABLED +from unittest import mock if sys.platform != 'win32': raise unittest.SkipTest('Windows only') @@ -16,23 +18,40 @@ class WinsocketpairTests(unittest.TestCase): - def test_winsocketpair(self): - ssock, csock = windows_utils.socketpair() - + def check_winsocketpair(self, ssock, csock): csock.send(b'xxx') self.assertEqual(b'xxx', ssock.recv(1024)) - csock.close() ssock.close() - @unittest.mock.patch('asyncio.windows_utils.socket') + def test_winsocketpair(self): + ssock, csock = windows_utils.socketpair() + self.check_winsocketpair(ssock, csock) + + @unittest.skipUnless(IPV6_ENABLED, 'IPv6 not supported or enabled') + def test_winsocketpair_ipv6(self): + ssock, csock = windows_utils.socketpair(family=socket.AF_INET6) + self.check_winsocketpair(ssock, csock) + + @mock.patch('asyncio.windows_utils.socket') def test_winsocketpair_exc(self, m_socket): + m_socket.AF_INET = socket.AF_INET + m_socket.SOCK_STREAM = socket.SOCK_STREAM m_socket.socket.return_value.getsockname.return_value = ('', 12345) m_socket.socket.return_value.accept.return_value = object(), object() m_socket.socket.return_value.connect.side_effect = OSError() self.assertRaises(OSError, windows_utils.socketpair) + def test_winsocketpair_invalid_args(self): + self.assertRaises(ValueError, + windows_utils.socketpair, family=socket.AF_UNSPEC) + self.assertRaises(ValueError, + windows_utils.socketpair, type=socket.SOCK_DGRAM) + self.assertRaises(ValueError, + windows_utils.socketpair, proto=1) + + class PipeTests(unittest.TestCase): diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py --- a/Lib/test/test_copy.py +++ b/Lib/test/test_copy.py @@ -98,6 +98,7 @@ pass tests = [None, 42, 2**100, 3.14, True, False, 1j, "hello", "hello\u1234", f.__code__, + b"world", bytes(range(256)), NewStyle, range(10), Classic, max, WithMetaclass] for x in tests: self.assertIs(copy.copy(x), x) diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -124,6 +124,14 @@ msg.set_payload([]) self.assertEqual(msg.get_payload(), []) + def test_attach_when_payload_is_string(self): + msg = Message() + msg['Content-Type'] = 'multipart/mixed' + msg.set_payload('string payload') + sub_msg = MIMEMessage(Message()) + self.assertRaisesRegex(TypeError, "[Aa]ttach.*non-multipart", + msg.attach, sub_msg) + def test_get_charsets(self): eq = self.assertEqual diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py --- a/Lib/test/test_email/test_policy.py +++ b/Lib/test/test_email/test_policy.py @@ -319,5 +319,14 @@ self.assertEqual(msg.as_string(), "Subject: testXTo: fooXX") +class TestConcretePolicies(unittest.TestCase): + + def test_header_store_parse_rejects_newlines(self): + instance = email.policy.EmailPolicy() + self.assertRaises(ValueError, + instance.header_store_parse, + 'From', 'spam\negg at foo.py') + + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py --- a/Lib/test/test_fileinput.py +++ b/Lib/test/test_fileinput.py @@ -260,6 +260,27 @@ fi.readline() self.assertTrue(custom_open_hook.invoked, "openhook not invoked") + def test_readline(self): + with open(TESTFN, 'wb') as f: + f.write(b'A\nB\r\nC\r') + # Fill TextIOWrapper buffer. + f.write(b'123456789\n' * 1000) + # Issue #20501: readline() shouldn't read whole file. + f.write(b'\x80') + self.addCleanup(safe_unlink, TESTFN) + + with FileInput(files=TESTFN, + openhook=hook_encoded('ascii'), bufsize=8) as fi: + try: + self.assertEqual(fi.readline(), 'A\n') + self.assertEqual(fi.readline(), 'B\n') + self.assertEqual(fi.readline(), 'C\n') + except UnicodeDecodeError: + self.fail('Read to end of file') + with self.assertRaises(UnicodeDecodeError): + # Read to the end of file. + list(fi) + def test_context_manager(self): try: t1 = writeTmp(1, ["A\nB\nC"]) @@ -837,6 +858,26 @@ self.assertIs(kwargs.pop('encoding'), encoding) self.assertFalse(kwargs) + def test_modes(self): + with open(TESTFN, 'wb') as f: + # UTF-7 is a convenient, seldom used encoding + f.write(b'A\nB\r\nC\rD+IKw-') + self.addCleanup(safe_unlink, TESTFN) + + def check(mode, expected_lines): + with FileInput(files=TESTFN, mode=mode, + openhook=hook_encoded('utf-7')) as fi: + lines = list(fi) + self.assertEqual(lines, expected_lines) + + check('r', ['A\n', 'B\n', 'C\n', 'D\u20ac']) + with self.assertWarns(DeprecationWarning): + check('rU', ['A\n', 'B\n', 'C\n', 'D\u20ac']) + with self.assertWarns(DeprecationWarning): + check('U', ['A\n', 'B\n', 'C\n', 'D\u20ac']) + with self.assertRaises(ValueError): + check('rb', ['A\n', 'B\r\n', 'C\r', 'D\u20ac']) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py --- a/Lib/test/test_genericpath.py +++ b/Lib/test/test_genericpath.py @@ -329,7 +329,6 @@ self.assertEqual(expandvars("$[foo]bar"), "$[foo]bar") self.assertEqual(expandvars("$bar bar"), "$bar bar") self.assertEqual(expandvars("$?bar"), "$?bar") - self.assertEqual(expandvars("${foo}bar"), "barbar") self.assertEqual(expandvars("$foo}bar"), "bar}bar") self.assertEqual(expandvars("${foo"), "${foo") self.assertEqual(expandvars("${{foo}}"), "baz1}") @@ -342,13 +341,40 @@ self.assertEqual(expandvars(b"$[foo]bar"), b"$[foo]bar") self.assertEqual(expandvars(b"$bar bar"), b"$bar bar") self.assertEqual(expandvars(b"$?bar"), b"$?bar") - self.assertEqual(expandvars(b"${foo}bar"), b"barbar") self.assertEqual(expandvars(b"$foo}bar"), b"bar}bar") self.assertEqual(expandvars(b"${foo"), b"${foo") self.assertEqual(expandvars(b"${{foo}}"), b"baz1}") self.assertEqual(expandvars(b"$foo$foo"), b"barbar") self.assertEqual(expandvars(b"$bar$bar"), b"$bar$bar") + @unittest.skipUnless(support.FS_NONASCII, 'need support.FS_NONASCII') + def test_expandvars_nonascii(self): + if self.pathmodule.__name__ == 'macpath': + self.skipTest('macpath.expandvars is a stub') + expandvars = self.pathmodule.expandvars + def check(value, expected): + self.assertEqual(expandvars(value), expected) + with support.EnvironmentVarGuard() as env: + env.clear() + nonascii = support.FS_NONASCII + env['spam'] = nonascii + env[nonascii] = 'ham' + nonascii + check(nonascii, nonascii) + check('$spam bar', '%s bar' % nonascii) + check('${spam}bar', '%sbar' % nonascii) + check('${%s}bar' % nonascii, 'ham%sbar' % nonascii) + check('$bar%s bar' % nonascii, '$bar%s bar' % nonascii) + check('$spam}bar', '%s}bar' % nonascii) + + check(os.fsencode(nonascii), os.fsencode(nonascii)) + check(b'$spam bar', os.fsencode('%s bar' % nonascii)) + check(b'${spam}bar', os.fsencode('%sbar' % nonascii)) + check(os.fsencode('${%s}bar' % nonascii), + os.fsencode('ham%sbar' % nonascii)) + check(os.fsencode('$bar%s bar' % nonascii), + os.fsencode('$bar%s bar' % nonascii)) + check(b'$spam}bar', os.fsencode('%s}bar' % nonascii)) + def test_abspath(self): self.assertIn("foo", self.pathmodule.abspath("foo")) with warnings.catch_warnings(): diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -319,8 +319,8 @@ def f(self, *, __kw:1): pass class Ham(Spam): pass - self.assertEquals(Spam.f.__annotations__, {'_Spam__kw': 1}) - self.assertEquals(Ham.f.__annotations__, {'_Spam__kw': 1}) + self.assertEqual(Spam.f.__annotations__, {'_Spam__kw': 1}) + self.assertEqual(Ham.f.__annotations__, {'_Spam__kw': 1}) # Check for SF Bug #1697248 - mixing decorators and a return annotation def null(x): return x @null diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -421,6 +421,13 @@ with gzip.GzipFile(fileobj=io.BytesIO(gzdata)) as f: self.assertEqual(f.read(), b'Test') + def test_prepend_error(self): + # See issue #20875 + with gzip.open(self.filename, "wb") as f: + f.write(data1) + with gzip.open(self.filename, "rb") as f: + f.fileobj.prepend() + class TestOpen(BaseTest): def test_binary_modes(self): uncompressed = data1 * 50 diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py --- a/Lib/test/test_idle.py +++ b/Lib/test/test_idle.py @@ -14,6 +14,7 @@ try: root = tk.Tk() root.destroy() + del root except tk.TclError: while 'gui' in use_resources: use_resources.remove('gui') diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py --- a/Lib/test/test_importlib/source/test_file_loader.py +++ b/Lib/test/test_importlib/source/test_file_loader.py @@ -190,6 +190,7 @@ if os.path.exists(pycache): shutil.rmtree(pycache) + @source_util.writes_bytecode_files def test_timestamp_overflow(self): # When a modification timestamp is larger than 2**32, it should be # truncated rather than raise an OverflowError. diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -855,6 +855,16 @@ bufio.__init__(rawio) self.assertEqual(b"abc", bufio.read()) + def test_uninitialized(self): + bufio = self.tp.__new__(self.tp) + del bufio + bufio = self.tp.__new__(self.tp) + self.assertRaisesRegex((ValueError, AttributeError), + 'uninitialized|has no attribute', + bufio.read, 0) + bufio.__init__(self.MockRawIO()) + self.assertEqual(bufio.read(0), b'') + def test_read(self): for arg in (None, 7): rawio = self.MockRawIO((b"abc", b"d", b"efg")) @@ -1106,6 +1116,16 @@ bufio.flush() self.assertEqual(b"".join(rawio._write_stack), b"abcghi") + def test_uninitialized(self): + bufio = self.tp.__new__(self.tp) + del bufio + bufio = self.tp.__new__(self.tp) + self.assertRaisesRegex((ValueError, AttributeError), + 'uninitialized|has no attribute', + bufio.write, b'') + bufio.__init__(self.MockRawIO()) + self.assertEqual(bufio.write(b''), 0) + def test_detach_flush(self): raw = self.MockRawIO() buf = self.tp(raw) @@ -1390,6 +1410,20 @@ pair = self.tp(self.MockRawIO(), self.MockRawIO()) self.assertFalse(pair.closed) + def test_uninitialized(self): + pair = self.tp.__new__(self.tp) + del pair + pair = self.tp.__new__(self.tp) + self.assertRaisesRegex((ValueError, AttributeError), + 'uninitialized|has no attribute', + pair.read, 0) + self.assertRaisesRegex((ValueError, AttributeError), + 'uninitialized|has no attribute', + pair.write, b'') + pair.__init__(self.MockRawIO(), self.MockRawIO()) + self.assertEqual(pair.read(0), b'') + self.assertEqual(pair.write(b''), 0) + def test_detach(self): pair = self.tp(self.MockRawIO(), self.MockRawIO()) self.assertRaises(self.UnsupportedOperation, pair.detach) @@ -1516,6 +1550,10 @@ BufferedReaderTest.test_constructor(self) BufferedWriterTest.test_constructor(self) + def test_uninitialized(self): + BufferedReaderTest.test_uninitialized(self) + BufferedWriterTest.test_uninitialized(self) + def test_read_and_write(self): raw = self.MockRawIO((b"asdf", b"ghjk")) rw = self.tp(raw, 8) diff --git a/Lib/test/test_modulefinder.py b/Lib/test/test_modulefinder.py --- a/Lib/test/test_modulefinder.py +++ b/Lib/test/test_modulefinder.py @@ -1,5 +1,7 @@ import os import errno +import importlib.machinery +import py_compile import shutil import unittest import tempfile @@ -208,6 +210,14 @@ from . import * """] +bytecode_test = [ + "a", + ["a"], + [], + [], + "" +] + def open_file(path): dirname = os.path.dirname(path) @@ -288,6 +298,16 @@ def test_relative_imports_4(self): self._do_test(relative_import_test_4) + def test_bytecode(self): + base_path = os.path.join(TEST_DIR, 'a') + source_path = base_path + importlib.machinery.SOURCE_SUFFIXES[0] + bytecode_path = base_path + importlib.machinery.BYTECODE_SUFFIXES[0] + with open_file(source_path) as file: + file.write('testing_modulefinder = True\n') + py_compile.compile(source_path, cfile=bytecode_path) + os.remove(source_path) + self._do_test(bytecode_test) + def test_main(): support.run_unittest(ModuleFinderTest) diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -22,13 +22,15 @@ fn = fn.replace('["', '[b"') fn = fn.replace(", '", ", b'") fn = fn.replace(', "', ', b"') + fn = os.fsencode(fn).decode('latin1') + fn = fn.encode('ascii', 'backslashreplace').decode('ascii') with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) gotResult = eval(fn) if isinstance(wantResult, str): - wantResult = wantResult.encode('ascii') + wantResult = os.fsencode(wantResult) elif isinstance(wantResult, tuple): - wantResult = tuple(r.encode('ascii') for r in wantResult) + wantResult = tuple(os.fsencode(r) for r in wantResult) gotResult = eval(fn) if wantResult != gotResult: @@ -223,7 +225,6 @@ tester('ntpath.expandvars("$[foo]bar")', "$[foo]bar") tester('ntpath.expandvars("$bar bar")', "$bar bar") tester('ntpath.expandvars("$?bar")', "$?bar") - tester('ntpath.expandvars("${foo}bar")', "barbar") tester('ntpath.expandvars("$foo}bar")', "bar}bar") tester('ntpath.expandvars("${foo")', "${foo") tester('ntpath.expandvars("${{foo}}")', "baz1}") @@ -237,6 +238,26 @@ tester('ntpath.expandvars("%foo%%bar")', "bar%bar") tester('ntpath.expandvars("\'%foo%\'%bar")', "\'%foo%\'%bar") + @unittest.skipUnless(support.FS_NONASCII, 'need support.FS_NONASCII') + def test_expandvars_nonascii(self): + def check(value, expected): + tester('ntpath.expandvars(%r)' % value, expected) + with support.EnvironmentVarGuard() as env: + env.clear() + nonascii = support.FS_NONASCII + env['spam'] = nonascii + env[nonascii] = 'ham' + nonascii + check('$spam bar', '%s bar' % nonascii) + check('$%s bar' % nonascii, '$%s bar' % nonascii) + check('${spam}bar', '%sbar' % nonascii) + check('${%s}bar' % nonascii, 'ham%sbar' % nonascii) + check('$spam}bar', '%s}bar' % nonascii) + check('$%s}bar' % nonascii, '$%s}bar' % nonascii) + check('%spam% bar', '%s bar' % nonascii) + check('%{}% bar'.format(nonascii), 'ham%s bar' % nonascii) + check('%spam%bar', '%sbar' % nonascii) + check('%{}%bar'.format(nonascii), 'ham%sbar' % nonascii) + def test_abspath(self): # ntpath.abspath() can only be used on a system with the "nt" module # (reasonably), so we protect this test with "import nt". This allows diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -368,6 +368,11 @@ def test_main(): run_unittest(PkgutilTests, PkgutilPEP302Tests, ExtendPathTests, NestedNamespacePackageTest, ImportlibMigrationTests) + # this is necessary if test is run repeated (like when finding leaks) + import zipimport + import importlib + zipimport._zip_directory_cache.clear() + importlib.invalidate_caches() if __name__ == '__main__': diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1161,7 +1161,7 @@ def test_initgroups(self): # find missing group - g = max(self.saved_groups) + 1 + g = max(self.saved_groups or [0]) + 1 name = pwd.getpwuid(posix.getuid()).pw_name posix.initgroups(name, g) self.assertIn(g, posix.getgroups()) diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -387,6 +387,16 @@ print_diffs(expected_text, result) self.fail("outputs are not equal, see diff above") + def test_text_enum_member_with_value_zero(self): + # Test issue #20654 to ensure enum member with value 0 can be + # displayed. It used to throw KeyError: 'zero'. + import enum + class BinaryInteger(enum.IntEnum): + zero = 0 + one = 1 + doc = pydoc.render_doc(BinaryInteger) + self.assertIn('', doc) + def test_issue8225(self): # Test issue8225 to ensure no doc link appears for xml.etree result, doc_loc = get_pydoc_text(xml.etree) diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py --- a/Lib/test/test_range.py +++ b/Lib/test/test_range.py @@ -380,6 +380,30 @@ it = pickle.loads(d) self.assertEqual(list(it), data[1:]) + def test_exhausted_iterator_pickling(self): + r = range(2**65, 2**65+2) + i = iter(r) + while True: + r = next(i) + if r == 2**65+1: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + + def test_large_exhausted_iterator_pickling(self): + r = range(20) + i = iter(r) + while True: + r = next(i) + if r == 19: + break + d = pickle.dumps(i) + i2 = pickle.loads(d) + self.assertEqual(list(i), []) + self.assertEqual(list(i2), []) + def test_odd_bug(self): # This used to raise a "SystemError: NULL result without error" # because the range validation step was eating the exception diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1205,6 +1205,24 @@ self.assertEqual(out.getvalue().splitlines(), ['literal 102 ', 'literal 111 ', 'literal 111 ']) + def test_keyword_parameters(self): + # Issue #20283: Accepting the string keyword parameter. + pat = re.compile(r'(ab)') + self.assertEqual( + pat.match(string='abracadabra', pos=7, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.fullmatch(string='abracadabra', pos=7, endpos=9).span(), (7, 9)) + self.assertEqual( + pat.search(string='abracadabra', pos=3, endpos=10).span(), (7, 9)) + self.assertEqual( + pat.findall(string='abracadabra', pos=3, endpos=10), ['ab']) + self.assertEqual( + pat.split(string='abracadabra', maxsplit=1), + ['', 'ab', 'racadabra']) + self.assertEqual( + pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(), + (7, 9)) + class PatternReprTests(unittest.TestCase): def check(self, pattern, expected): diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -275,6 +275,7 @@ self.skipTest('%s is unavailable' % url) self.assertEqual(parser.can_fetch("*", robots_url), False) + @unittest.skip('does not handle the gzip encoding delivered by pydotorg') def testPythonOrg(self): support.requires('network') with support.transient_internet('www.python.org'): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1492,6 +1492,15 @@ # Move a dir inside an existing dir on another filesystem. self.test_move_dir_to_dir() + def test_move_dir_sep_to_dir(self): + self._check_move_dir(self.src_dir + os.path.sep, self.dst_dir, + os.path.join(self.dst_dir, os.path.basename(self.src_dir))) + + @unittest.skipUnless(os.path.altsep, 'requires os.path.altsep') + def test_move_dir_altsep_to_dir(self): + self._check_move_dir(self.src_dir + os.path.altsep, self.dst_dir, + os.path.join(self.dst_dir, os.path.basename(self.src_dir))) + def test_existing_file_inside_dest_dir(self): # A file with the same name inside the destination dir already exists. with open(self.dst_file, "wb"): diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1234,9 +1234,15 @@ # Issue #6697. self.assertRaises(UnicodeEncodeError, socket.getaddrinfo, 'localhost', '\uD800') - # Issue 17269 + # Issue 17269: test workaround for OS X platform bug segfault if hasattr(socket, 'AI_NUMERICSERV'): - socket.getaddrinfo("localhost", None, 0, 0, 0, socket.AI_NUMERICSERV) + try: + # The arguments here are undefined and the call may succeed + # or fail. All we care here is that it doesn't segfault. + socket.getaddrinfo("localhost", None, 0, 0, 0, + socket.AI_NUMERICSERV) + except socket.gaierror: + pass def test_getnameinfo(self): # only IP addresses are allowed diff --git a/Lib/test/test_source_encoding.py b/Lib/test/test_source_encoding.py --- a/Lib/test/test_source_encoding.py +++ b/Lib/test/test_source_encoding.py @@ -5,6 +5,7 @@ import importlib import os import sys +import subprocess class SourceEncodingTest(unittest.TestCase): @@ -58,6 +59,15 @@ # two bytes in common with the UTF-8 BOM self.assertRaises(SyntaxError, eval, b'\xef\xbb\x20') + def test_20731(self): + sub = subprocess.Popen([sys.executable, + os.path.join(os.path.dirname(__file__), + 'coding20731.py')], + stderr=subprocess.PIPE) + err = sub.communicate()[1] + self.assertEqual(sub.returncode, 0) + self.assertNotIn(b'SyntaxError', err) + def test_error_message(self): compile(b'# -*- coding: iso-8859-15 -*-\n', 'dummy', 'exec') compile(b'\xef\xbb\xbf\n', 'dummy', 'exec') diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -102,7 +102,7 @@ self.assertTrue(os.path.isdir(path)) self.assertFalse(os.path.isdir(path)) finally: - shutil.rmtree(parent_dir) + support.rmtree(parent_dir) def test_temp_dir__path_none(self): """Test passing no path.""" diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1,4 +1,5 @@ import unittest, test.support +from test.script_helper import assert_python_ok, assert_python_failure import sys, io, os import struct import subprocess @@ -89,74 +90,54 @@ # Python/pythonrun.c::PyErr_PrintEx() is tricky. def test_exit(self): - + # call with two arguments self.assertRaises(TypeError, sys.exit, 42, 42) # call without argument - try: - sys.exit(0) - except SystemExit as exc: - self.assertEqual(exc.code, 0) - except: - self.fail("wrong exception") - else: - self.fail("no exception") + with self.assertRaises(SystemExit) as cm: + sys.exit() + self.assertIsNone(cm.exception.code) + + rc, out, err = assert_python_ok('-c', 'import sys; sys.exit()') + self.assertEqual(rc, 0) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + + # call with integer argument + with self.assertRaises(SystemExit) as cm: + sys.exit(42) + self.assertEqual(cm.exception.code, 42) # call with tuple argument with one entry # entry will be unpacked - try: - sys.exit(42) - except SystemExit as exc: - self.assertEqual(exc.code, 42) - except: - self.fail("wrong exception") - else: - self.fail("no exception") - - # call with integer argument - try: + with self.assertRaises(SystemExit) as cm: sys.exit((42,)) - except SystemExit as exc: - self.assertEqual(exc.code, 42) - except: - self.fail("wrong exception") - else: - self.fail("no exception") + self.assertEqual(cm.exception.code, 42) # call with string argument - try: + with self.assertRaises(SystemExit) as cm: sys.exit("exit") - except SystemExit as exc: - self.assertEqual(exc.code, "exit") - except: - self.fail("wrong exception") - else: - self.fail("no exception") + self.assertEqual(cm.exception.code, "exit") # call with tuple argument with two entries - try: + with self.assertRaises(SystemExit) as cm: sys.exit((17, 23)) - except SystemExit as exc: - self.assertEqual(exc.code, (17, 23)) - except: - self.fail("wrong exception") - else: - self.fail("no exception") + self.assertEqual(cm.exception.code, (17, 23)) # test that the exit machinery handles SystemExits properly - rc = subprocess.call([sys.executable, "-c", - "raise SystemExit(47)"]) + rc, out, err = assert_python_failure('-c', 'raise SystemExit(47)') self.assertEqual(rc, 47) + self.assertEqual(out, b'') + self.assertEqual(err, b'') - def check_exit_message(code, expected, env=None): - process = subprocess.Popen([sys.executable, "-c", code], - stderr=subprocess.PIPE, env=env) - stdout, stderr = process.communicate() - self.assertEqual(process.returncode, 1) - self.assertTrue(stderr.startswith(expected), - "%s doesn't start with %s" % (ascii(stderr), ascii(expected))) + def check_exit_message(code, expected, **env_vars): + rc, out, err = assert_python_failure('-c', code, **env_vars) + self.assertEqual(rc, 1) + self.assertEqual(out, b'') + self.assertTrue(err.startswith(expected), + "%s doesn't start with %s" % (ascii(err), ascii(expected))) - # test that stderr buffer if flushed before the exit message is written + # test that stderr buffer is flushed before the exit message is written # into stderr check_exit_message( r'import sys; sys.stderr.write("unflushed,"); sys.exit("message")', @@ -170,11 +151,9 @@ # test that the unicode message is encoded to the stderr encoding # instead of the default encoding (utf8) - env = os.environ.copy() - env['PYTHONIOENCODING'] = 'latin-1' check_exit_message( r'import sys; sys.exit("h\xe9")', - b"h\xe9", env=env) + b"h\xe9", PYTHONIOENCODING='latin-1') def test_getdefaultencoding(self): self.assertRaises(TypeError, sys.getdefaultencoding, 42) diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -240,14 +240,16 @@ self.assertIn(b'ustar/dirtype/', out) self.assertIn(b'ustar/dirtype-with-size/', out) # Make sure it is able to print unencodable characters - self.assertIn(br'ustar/umlauts-' - br'\udcc4\udcd6\udcdc\udce4\udcf6\udcfc\udcdf', out) - self.assertIn(br'misc/regtype-hpux-signed-chksum-' - br'\udcc4\udcd6\udcdc\udce4\udcf6\udcfc\udcdf', out) - self.assertIn(br'misc/regtype-old-v7-signed-chksum-' - br'\udcc4\udcd6\udcdc\udce4\udcf6\udcfc\udcdf', out) - self.assertIn(br'pax/bad-pax-\udce4\udcf6\udcfc', out) - self.assertIn(br'pax/hdrcharset-\udce4\udcf6\udcfc', out) + def conv(b): + s = b.decode(self.tar.encoding, 'surrogateescape') + return s.encode('ascii', 'backslashreplace') + self.assertIn(conv(b'ustar/umlauts-\xc4\xd6\xdc\xe4\xf6\xfc\xdf'), out) + self.assertIn(conv(b'misc/regtype-hpux-signed-chksum-' + b'\xc4\xd6\xdc\xe4\xf6\xfc\xdf'), out) + self.assertIn(conv(b'misc/regtype-old-v7-signed-chksum-' + b'\xc4\xd6\xdc\xe4\xf6\xfc\xdf'), out) + self.assertIn(conv(b'pax/bad-pax-\xe4\xf6\xfc'), out) + self.assertIn(conv(b'pax/hdrcharset-\xe4\xf6\xfc'), out) # Make sure it prints files separated by one newline without any # 'ls -l'-like accessories if verbose flag is not being used # ... diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -376,6 +376,7 @@ result = arg return arg self.interp.createcommand('testfunc', testfunc) + self.addCleanup(self.interp.tk.deletecommand, 'testfunc') def check(value, expected, eq=self.assertEqual): r = self.interp.call('testfunc', value) self.assertIsInstance(result, str) diff --git a/Lib/test/test_threadsignals.py b/Lib/test/test_threadsignals.py --- a/Lib/test/test_threadsignals.py +++ b/Lib/test/test_threadsignals.py @@ -74,6 +74,9 @@ @unittest.skipIf(USING_PTHREAD_COND, 'POSIX condition variables cannot be interrupted') + # Issue #20564: sem_timedwait() cannot be interrupted on OpenBSD + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'lock cannot be interrupted on OpenBSD') def test_lock_acquire_interruption(self): # Mimic receiving a SIGINT (KeyboardInterrupt) with SIGALRM while stuck # in a deadlock. @@ -97,6 +100,9 @@ @unittest.skipIf(USING_PTHREAD_COND, 'POSIX condition variables cannot be interrupted') + # Issue #20564: sem_timedwait() cannot be interrupted on OpenBSD + @unittest.skipIf(sys.platform.startswith('openbsd'), + 'lock cannot be interrupted on OpenBSD') def test_rlock_acquire_interruption(self): # Mimic receiving a SIGINT (KeyboardInterrupt) with SIGALRM while stuck # in a deadlock. diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -14,6 +14,8 @@ SIZEOF_INT = sysconfig.get_config_var('SIZEOF_INT') or 4 TIME_MAXYEAR = (1 << 8 * SIZEOF_INT - 1) - 1 TIME_MINYEAR = -TIME_MAXYEAR - 1 +_PyTime_ROUND_DOWN = 0 +_PyTime_ROUND_UP = 1 class TimeTestCase(unittest.TestCase): @@ -226,7 +228,7 @@ self.assertEqual(time.ctime(t), 'Sun Sep 16 01:03:52 1973') t = time.mktime((2000, 1, 1, 0, 0, 0, 0, 0, -1)) self.assertEqual(time.ctime(t), 'Sat Jan 1 00:00:00 2000') - for year in [-100, 100, 1000, 2000, 10000]: + for year in [-100, 100, 1000, 2000, 2050, 10000]: try: testval = time.mktime((year, 1, 10) + (0,)*6) except (ValueError, OverflowError): @@ -344,6 +346,13 @@ def test_mktime(self): # Issue #1726687 for t in (-2, -1, 0, 1): + if sys.platform.startswith('aix') and t == -1: + # Issue #11188, #19748: mktime() returns -1 on error. On Linux, + # the tm_wday field is used as a sentinel () to detect if -1 is + # really an error or a valid timestamp. On AIX, tm_wday is + # unchanged even on success and so cannot be used as a + # sentinel. + continue try: tt = time.localtime(t) except (OverflowError, OSError): @@ -585,58 +594,116 @@ @support.cpython_only def test_time_t(self): from _testcapi import pytime_object_to_time_t - for obj, time_t in ( - (0, 0), - (-1, -1), - (-1.0, -1), - (-1.9, -1), - (1.0, 1), - (1.9, 1), + for obj, time_t, rnd in ( + # Round towards zero + (0, 0, _PyTime_ROUND_DOWN), + (-1, -1, _PyTime_ROUND_DOWN), + (-1.0, -1, _PyTime_ROUND_DOWN), + (-1.9, -1, _PyTime_ROUND_DOWN), + (1.0, 1, _PyTime_ROUND_DOWN), + (1.9, 1, _PyTime_ROUND_DOWN), + # Round away from zero + (0, 0, _PyTime_ROUND_UP), + (-1, -1, _PyTime_ROUND_UP), + (-1.0, -1, _PyTime_ROUND_UP), + (-1.9, -2, _PyTime_ROUND_UP), + (1.0, 1, _PyTime_ROUND_UP), + (1.9, 2, _PyTime_ROUND_UP), ): - self.assertEqual(pytime_object_to_time_t(obj), time_t) + self.assertEqual(pytime_object_to_time_t(obj, rnd), time_t) + rnd = _PyTime_ROUND_DOWN for invalid in self.invalid_values: - self.assertRaises(OverflowError, pytime_object_to_time_t, invalid) + self.assertRaises(OverflowError, + pytime_object_to_time_t, invalid, rnd) @support.cpython_only def test_timeval(self): from _testcapi import pytime_object_to_timeval - for obj, timeval in ( - (0, (0, 0)), - (-1, (-1, 0)), - (-1.0, (-1, 0)), - (1e-6, (0, 1)), - (-1e-6, (-1, 999999)), - (-1.2, (-2, 800000)), - (1.1234560, (1, 123456)), - (1.1234569, (1, 123456)), - (-1.1234560, (-2, 876544)), - (-1.1234561, (-2, 876543)), + for obj, timeval, rnd in ( + # Round towards zero + (0, (0, 0), _PyTime_ROUND_DOWN), + (-1, (-1, 0), _PyTime_ROUND_DOWN), + (-1.0, (-1, 0), _PyTime_ROUND_DOWN), + (1e-6, (0, 1), _PyTime_ROUND_DOWN), + (1e-7, (0, 0), _PyTime_ROUND_DOWN), + (-1e-6, (-1, 999999), _PyTime_ROUND_DOWN), + (-1e-7, (-1, 999999), _PyTime_ROUND_DOWN), + (-1.2, (-2, 800000), _PyTime_ROUND_DOWN), + (0.9999999, (0, 999999), _PyTime_ROUND_DOWN), + (0.0000041, (0, 4), _PyTime_ROUND_DOWN), + (1.1234560, (1, 123456), _PyTime_ROUND_DOWN), + (1.1234569, (1, 123456), _PyTime_ROUND_DOWN), + (-0.0000040, (-1, 999996), _PyTime_ROUND_DOWN), + (-0.0000041, (-1, 999995), _PyTime_ROUND_DOWN), + (-1.1234560, (-2, 876544), _PyTime_ROUND_DOWN), + (-1.1234561, (-2, 876543), _PyTime_ROUND_DOWN), + # Round away from zero + (0, (0, 0), _PyTime_ROUND_UP), + (-1, (-1, 0), _PyTime_ROUND_UP), + (-1.0, (-1, 0), _PyTime_ROUND_UP), + (1e-6, (0, 1), _PyTime_ROUND_UP), + (1e-7, (0, 1), _PyTime_ROUND_UP), + (-1e-6, (-1, 999999), _PyTime_ROUND_UP), + (-1e-7, (-1, 999999), _PyTime_ROUND_UP), + (-1.2, (-2, 800000), _PyTime_ROUND_UP), + (0.9999999, (1, 0), _PyTime_ROUND_UP), + (0.0000041, (0, 5), _PyTime_ROUND_UP), + (1.1234560, (1, 123457), _PyTime_ROUND_UP), + (1.1234569, (1, 123457), _PyTime_ROUND_UP), + (-0.0000040, (-1, 999996), _PyTime_ROUND_UP), + (-0.0000041, (-1, 999995), _PyTime_ROUND_UP), + (-1.1234560, (-2, 876544), _PyTime_ROUND_UP), + (-1.1234561, (-2, 876543), _PyTime_ROUND_UP), ): - self.assertEqual(pytime_object_to_timeval(obj), timeval) + with self.subTest(obj=obj, round=rnd, timeval=timeval): + self.assertEqual(pytime_object_to_timeval(obj, rnd), timeval) + rnd = _PyTime_ROUND_DOWN for invalid in self.invalid_values: - self.assertRaises(OverflowError, pytime_object_to_timeval, invalid) + self.assertRaises(OverflowError, + pytime_object_to_timeval, invalid, rnd) @support.cpython_only def test_timespec(self): from _testcapi import pytime_object_to_timespec - for obj, timespec in ( - (0, (0, 0)), - (-1, (-1, 0)), - (-1.0, (-1, 0)), - (1e-9, (0, 1)), - (-1e-9, (-1, 999999999)), - (-1.2, (-2, 800000000)), - (1.1234567890, (1, 123456789)), - (1.1234567899, (1, 123456789)), - (-1.1234567890, (-2, 876543211)), - (-1.1234567891, (-2, 876543210)), + for obj, timespec, rnd in ( + # Round towards zero + (0, (0, 0), _PyTime_ROUND_DOWN), + (-1, (-1, 0), _PyTime_ROUND_DOWN), + (-1.0, (-1, 0), _PyTime_ROUND_DOWN), + (1e-9, (0, 1), _PyTime_ROUND_DOWN), + (1e-10, (0, 0), _PyTime_ROUND_DOWN), + (-1e-9, (-1, 999999999), _PyTime_ROUND_DOWN), + (-1e-10, (-1, 999999999), _PyTime_ROUND_DOWN), + (-1.2, (-2, 800000000), _PyTime_ROUND_DOWN), + (0.9999999999, (0, 999999999), _PyTime_ROUND_DOWN), + (1.1234567890, (1, 123456789), _PyTime_ROUND_DOWN), + (1.1234567899, (1, 123456789), _PyTime_ROUND_DOWN), + (-1.1234567890, (-2, 876543211), _PyTime_ROUND_DOWN), + (-1.1234567891, (-2, 876543210), _PyTime_ROUND_DOWN), + # Round away from zero + (0, (0, 0), _PyTime_ROUND_UP), + (-1, (-1, 0), _PyTime_ROUND_UP), + (-1.0, (-1, 0), _PyTime_ROUND_UP), + (1e-9, (0, 1), _PyTime_ROUND_UP), + (1e-10, (0, 1), _PyTime_ROUND_UP), + (-1e-9, (-1, 999999999), _PyTime_ROUND_UP), + (-1e-10, (-1, 999999999), _PyTime_ROUND_UP), + (-1.2, (-2, 800000000), _PyTime_ROUND_UP), + (0.9999999999, (1, 0), _PyTime_ROUND_UP), + (1.1234567890, (1, 123456790), _PyTime_ROUND_UP), + (1.1234567899, (1, 123456790), _PyTime_ROUND_UP), + (-1.1234567890, (-2, 876543211), _PyTime_ROUND_UP), + (-1.1234567891, (-2, 876543210), _PyTime_ROUND_UP), ): - self.assertEqual(pytime_object_to_timespec(obj), timespec) + with self.subTest(obj=obj, round=rnd, timespec=timespec): + self.assertEqual(pytime_object_to_timespec(obj, rnd), timespec) + rnd = _PyTime_ROUND_DOWN for invalid in self.invalid_values: - self.assertRaises(OverflowError, pytime_object_to_timespec, invalid) + self.assertRaises(OverflowError, + pytime_object_to_timespec, invalid, rnd) @unittest.skipUnless(time._STRUCT_TM_ITEMS == 11, "needs tm_zone support") def test_localtime_timezone(self): diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py --- a/Lib/test/test_tokenize.py +++ b/Lib/test/test_tokenize.py @@ -2,7 +2,7 @@ Tests for the tokenize module. The tests can be really simple. Given a small fragment of source -code, print out a table with tokens. The ENDMARK is omitted for +code, print out a table with tokens. The ENDMARKER is omitted for brevity. >>> dump_tokens("1 + 1") @@ -578,9 +578,15 @@ >>> tempdir = os.path.dirname(f) or os.curdir >>> testfiles = glob.glob(os.path.join(tempdir, "test*.py")) -tokenize is broken on test_pep3131.py because regular expressions are broken on -the obscure unicode identifiers in it. *sigh* +Tokenize is broken on test_pep3131.py because regular expressions are +broken on the obscure unicode identifiers in it. *sigh* +With roundtrip extended to test the 5-tuple mode of untokenize, +7 more testfiles fail. Remove them also until the failure is diagnosed. + >>> testfiles.remove(os.path.join(tempdir, "test_pep3131.py")) + >>> for f in ('buffer', 'builtin', 'fileio', 'inspect', 'os', 'platform', 'sys'): + ... testfiles.remove(os.path.join(tempdir, "test_%s.py") % f) + ... >>> if not support.is_resource_enabled("cpu"): ... testfiles = random.sample(testfiles, 10) ... @@ -638,7 +644,7 @@ from test import support from tokenize import (tokenize, _tokenize, untokenize, NUMBER, NAME, OP, STRING, ENDMARKER, ENCODING, tok_name, detect_encoding, - open as tokenize_open) + open as tokenize_open, Untokenizer) from io import BytesIO from unittest import TestCase import os, sys, glob @@ -659,21 +665,39 @@ def roundtrip(f): """ Test roundtrip for `untokenize`. `f` is an open file or a string. - The source code in f is tokenized, converted back to source code via - tokenize.untokenize(), and tokenized again from the latter. The test - fails if the second tokenization doesn't match the first. + The source code in f is tokenized to both 5- and 2-tuples. + Both sequences are converted back to source code via + tokenize.untokenize(), and the latter tokenized again to 2-tuples. + The test fails if the 3 pair tokenizations do not match. + + When untokenize bugs are fixed, untokenize with 5-tuples should + reproduce code that does not contain a backslash continuation + following spaces. A proper test should test this. + + This function would be more useful for correcting bugs if it reported + the first point of failure, like assertEqual, rather than just + returning False -- or if it were only used in unittests and not + doctest and actually used assertEqual. """ + # Get source code and original tokenizations if isinstance(f, str): - f = BytesIO(f.encode('utf-8')) - try: - token_list = list(tokenize(f.readline)) - finally: + code = f.encode('utf-8') + else: + code = f.read() f.close() - tokens1 = [tok[:2] for tok in token_list] - new_bytes = untokenize(tokens1) - readline = (line for line in new_bytes.splitlines(keepends=True)).__next__ - tokens2 = [tok[:2] for tok in tokenize(readline)] - return tokens1 == tokens2 + readline = iter(code.splitlines(keepends=True)).__next__ + tokens5 = list(tokenize(readline)) + tokens2 = [tok[:2] for tok in tokens5] + # Reproduce tokens2 from pairs + bytes_from2 = untokenize(tokens2) + readline2 = iter(bytes_from2.splitlines(keepends=True)).__next__ + tokens2_from2 = [tok[:2] for tok in tokenize(readline2)] + # Reproduce tokens2 from 5-tuples + bytes_from5 = untokenize(tokens5) + readline5 = iter(bytes_from5.splitlines(keepends=True)).__next__ + tokens2_from5 = [tok[:2] for tok in tokenize(readline5)] + # Compare 3 versions + return tokens2 == tokens2_from2 == tokens2_from5 # This is an example from the docs, set up as a doctest. def decistmt(s): @@ -1153,6 +1177,47 @@ # See http://bugs.python.org/issue16152 self.assertExactTypeEqual('@ ', token.AT) +class UntokenizeTest(TestCase): + + def test_bad_input_order(self): + # raise if previous row + u = Untokenizer() + u.prev_row = 2 + u.prev_col = 2 + with self.assertRaises(ValueError) as cm: + u.add_whitespace((1,3)) + self.assertEqual(cm.exception.args[0], + 'start (1,3) precedes previous end (2,2)') + # raise if previous column in row + self.assertRaises(ValueError, u.add_whitespace, (2,1)) + + def test_backslash_continuation(self): + # The problem is that \ leaves no token + u = Untokenizer() + u.prev_row = 1 + u.prev_col = 1 + u.tokens = [] + u.add_whitespace((2, 0)) + self.assertEqual(u.tokens, ['\\\n']) + u.prev_row = 2 + u.add_whitespace((4, 4)) + self.assertEqual(u.tokens, ['\\\n', '\\\n\\\n', ' ']) + self.assertTrue(roundtrip('a\n b\n c\n \\\n c\n')) + + def test_iter_compat(self): + u = Untokenizer() + token = (NAME, 'Hello') + tokens = [(ENCODING, 'utf-8'), token] + u.compat(token, iter([])) + self.assertEqual(u.tokens, ["Hello "]) + u = Untokenizer() + self.assertEqual(u.untokenize(iter([token])), 'Hello ') + u = Untokenizer() + self.assertEqual(u.untokenize(iter(tokens)), 'Hello ') + self.assertEqual(u.encoding, 'utf-8') + self.assertEqual(untokenize(iter(tokens)), b'Hello ') + + __test__ = {"doctests" : doctests, 'decistmt': decistmt} def test_main(): @@ -1162,6 +1227,7 @@ support.run_unittest(Test_Tokenize) support.run_unittest(TestDetectEncoding) support.run_unittest(TestTokenize) + support.run_unittest(UntokenizeTest) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -346,6 +346,8 @@ self.assertIsNot(snapshot5.traces, snapshot.traces) self.assertEqual(snapshot5.traces, snapshot.traces) + self.assertRaises(TypeError, snapshot.filter_traces, filter1) + def test_snapshot_group_by_line(self): snapshot, snapshot2 = create_snapshots() tb_0 = traceback_lineno('', 0) diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -1,6 +1,6 @@ # Python test set -- part 6, built-in types -from test.support import run_unittest, run_with_locale +from test.support import run_unittest, run_with_locale, cpython_only import collections import pickle import locale @@ -1170,9 +1170,31 @@ self.assertEqual(ns, ns_roundtrip, pname) +class SharedKeyTests(unittest.TestCase): + + @cpython_only + def test_subclasses(self): + # Verify that subclasses can share keys (per PEP 412) + class A: + pass + class B(A): + pass + + a, b = A(), B() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + a.x, a.y, a.z, a.w = range(4) + self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + a2 = A() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + b.u, b.v, b.w, b.t = range(4) + self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({})) + + def test_main(): run_unittest(TypesTests, MappingProxyTests, ClassCreationTests, - SimpleNamespaceTests) + SimpleNamespaceTests, SharedKeyTests) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1227,7 +1227,8 @@ self.assertTrue(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be True' % host) # Check hosts that should not trigger the proxy bypass - for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', 'test'): + for host in ('abc.foo.bar', 'bar.com', '127.0.0.2', '10.11.0.1', + 'notinbypass'): self.assertFalse(_proxy_bypass_macosx_sysconf(host, bypass), 'expected bypass of %s to be False' % host) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -1339,6 +1339,21 @@ args = args + (col2, row2) return self._getints(self.tk.call(*args)) or None bbox = grid_bbox + + def _gridconvvalue(self, value): + if isinstance(value, (str, _tkinter.Tcl_Obj)): + try: + svalue = str(value) + if not svalue: + return None + elif '.' in svalue: + return getdouble(svalue) + else: + return getint(svalue) + except ValueError: + pass + return value + def _grid_configure(self, command, index, cnf, kw): """Internal function.""" if isinstance(cnf, str) and not kw: @@ -1357,22 +1372,14 @@ for i in range(0, len(words), 2): key = words[i][1:] value = words[i+1] - if not value: - value = None - elif '.' in str(value): - value = getdouble(value) - else: - value = getint(value) - dict[key] = value + dict[key] = self._gridconvvalue(value) return dict res = self.tk.call( ('grid', command, self._w, index) + options) if len(options) == 1: - if not res: return None - # In Tk 7.5, -width can be a float - if '.' in res: return getdouble(res) - return getint(res) + return self._gridconvvalue(res) + def grid_columnconfigure(self, index, cnf={}, **kw): """Configure column INDEX of a grid. diff --git a/Lib/tokenize.py b/Lib/tokenize.py --- a/Lib/tokenize.py +++ b/Lib/tokenize.py @@ -25,12 +25,14 @@ 'Skip Montanaro, Raymond Hettinger, Trent Nelson, ' 'Michael Foord') import builtins +from codecs import lookup, BOM_UTF8 +import collections +from io import TextIOWrapper +from itertools import chain import re import sys from token import * -from codecs import lookup, BOM_UTF8 -import collections -from io import TextIOWrapper + cookie_re = re.compile(r'^[ \t\f]*#.*coding[:=][ \t]*([-\w.]+)', re.ASCII) blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII) @@ -229,20 +231,29 @@ def add_whitespace(self, start): row, col = start - assert row <= self.prev_row + if row < self.prev_row or row == self.prev_row and col < self.prev_col: + raise ValueError("start ({},{}) precedes previous end ({},{})" + .format(row, col, self.prev_row, self.prev_col)) + row_offset = row - self.prev_row + if row_offset: + self.tokens.append("\\\n" * row_offset) + self.prev_col = 0 col_offset = col - self.prev_col if col_offset: self.tokens.append(" " * col_offset) def untokenize(self, iterable): - for t in iterable: + it = iter(iterable) + for t in it: if len(t) == 2: - self.compat(t, iterable) + self.compat(t, it) break tok_type, token, start, end, line = t if tok_type == ENCODING: self.encoding = token continue + if tok_type == ENDMARKER: + break self.add_whitespace(start) self.tokens.append(token) self.prev_row, self.prev_col = end @@ -252,17 +263,12 @@ return "".join(self.tokens) def compat(self, token, iterable): - startline = False indents = [] toks_append = self.tokens.append - toknum, tokval = token + startline = token[0] in (NEWLINE, NL) + prevstring = False - if toknum in (NAME, NUMBER): - tokval += ' ' - if toknum in (NEWLINE, NL): - startline = True - prevstring = False - for tok in iterable: + for tok in chain([token], iterable): toknum, tokval = tok[:2] if toknum == ENCODING: self.encoding = tokval diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py --- a/Lib/tracemalloc.py +++ b/Lib/tracemalloc.py @@ -1,4 +1,4 @@ -from collections import Sequence +from collections import Sequence, Iterable from functools import total_ordering import fnmatch import linecache @@ -119,12 +119,12 @@ previous = old_group.pop(traceback, None) if previous is not None: stat = StatisticDiff(traceback, - stat.size, stat.size - previous.size, - stat.count, stat.count - previous.count) + stat.size, stat.size - previous.size, + stat.count, stat.count - previous.count) else: stat = StatisticDiff(traceback, - stat.size, stat.size, - stat.count, stat.count) + stat.size, stat.size, + stat.count, stat.count) statistics.append(stat) for traceback, stat in old_group.items(): @@ -141,6 +141,7 @@ __slots__ = ("_frame",) def __init__(self, frame): + # frame is a tuple: (filename: str, lineno: int) self._frame = frame @property @@ -177,6 +178,8 @@ def __init__(self, frames): Sequence.__init__(self) + # frames is a tuple of frame tuples: see Frame constructor for the + # format of a frame tuple self._frames = frames def __len__(self): @@ -241,6 +244,8 @@ __slots__ = ("_trace",) def __init__(self, trace): + # trace is a tuple: (size, traceback), see Traceback constructor + # for the format of the traceback tuple self._trace = trace @property @@ -268,6 +273,7 @@ class _Traces(Sequence): def __init__(self, traces): Sequence.__init__(self) + # traces is a tuple of trace tuples: see Trace constructor self._traces = traces def __len__(self): @@ -338,6 +344,8 @@ """ def __init__(self, traces, traceback_limit): + # traces is a tuple of trace tuples: see _Traces constructor for + # the exact format self.traces = _Traces(traces) self.traceback_limit = traceback_limit @@ -374,6 +382,9 @@ is a list of Filter instances. If filters is an empty list, return a new Snapshot instance with a copy of the traces. """ + if not isinstance(filters, Iterable): + raise TypeError("filters must be a list of filters, not %s" + % type(filters).__name__) if filters: include_filters = [] exclude_filters = [] diff --git a/Mac/README b/Mac/README --- a/Mac/README +++ b/Mac/README @@ -86,6 +86,13 @@ and the 10.4u SDK. Starting with Xcode 3 and OS X 10.5, more configurations are available. +In general, universal builds depend on specific features provided by the +Apple-supplied compilers and other build tools included in Apple's Xcode +development tools. You should install Xcode and the command line tools +component appropriate for the OS X release you are running on. See the +Python Developer's Guide (http://docs.python.org/devguide/setup.html) +for more information. + 2.1 Flavors of universal binaries ................................. diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -40,6 +40,7 @@ Oliver Andrich Ross Andrus Juancarlo A?ez +Chris Angelico J?r?my Anger Ankur Ankan Jon Anglin @@ -585,6 +586,7 @@ Lars Immisch Bobby Impollonia Meador Inge +Peter Ingebretson Tony Ingraldi John Interrante Bob Ippolito @@ -949,6 +951,7 @@ James Oakley Elena Oat Jon Oberheide +Milan Oberkirch Pascal Oberndoerfer Jeffrey Ollie Adam Olsen @@ -1095,6 +1098,7 @@ Jim Robinson Mark Roddy Kevin Rodgers +Sean Rodman Giampaolo Rodola Elson Rodriguez Adi Roiban @@ -1186,6 +1190,7 @@ Ha Shao Mark Shannon Richard Shapiro +Varun Sharma Vlad Shcherbina Justin Sheehy Charlie Shepherd diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,9 +10,69 @@ Core and Builtins ----------------- +- Issue #20929: Add a type cast to avoid shifting a negative number. + +- Issue #20731: Properly position in source code files even if they + are opened in text mode. Patch by Serhiy Storchaka. + +- Issue #20637: Key-sharing now also works for instance dictionaries of + subclasses. Patch by Peter Ingebretson. + Library ------- +- Issue #19157: Include the broadcast address in the usuable hosts for IPv6 + in ipaddress. + +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + +- Issue #4931: distutils should not produce unhelpful "error: None" messages + anymore. distutils.util.grok_environment_error is kept but doc-deprecated. + +- Issue #20875: Prevent possible gzip "'read' is not defined" NameError. + Patch by Claudiu Popa. + +- Issue #11558: ``email.message.Message.attach`` now returns a more + useful error message if ``attach`` is called on a message for which + ``is_multipart`` is False. + +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + +- Issue #20778: Fix modulefinder to work with bytecode-only modules. + +- Issue #20791: copy.copy() now doesn't make a copy when the input is + a bytes object. Initial patch by Peter Otten. + +- Issue #19748: On AIX, time.mktime() now raises an OverflowError for year + outsize range [1902; 2037]. + +Documentation +------------- + +- Issue #20765: Add missing documentation for PurePath.with_name() and + PurePath.with_suffix(). + +- Issue #19407: New package installation and distribution guides based on + the Python Packaging Authority tools. Existing guides have been retained + as legacy links from the distutils docs, as they still contain some + required reference material for tool developers that isn't recorded + anywhere else. + +Tests +----- + +- Issue #20743: Fix a reference leak in test_tcl. + +Tools/Demos +----------- + +- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. + Patch by Arfrever Frehtes Taifersar Arahesis. + What's New in Python 3.4.0? =========================== @@ -158,6 +218,8 @@ - Issue #19255: The builtins module is restored to initial value before cleaning other modules. The sys and builtins modules are cleaned last. +- Issue #20588: Make Python-ast.c C89 compliant. + - Issue #20437: Fixed 22 potential bugs when deleting objects references. - Issue #20500: Displaying an exception at interpreter shutdown no longer @@ -1217,8 +1279,9 @@ - Issue #19523: Closed FileHandler leak which occurred when delay was set. -- Issue #19544 and #6516: Restore support for --user and --group parameters to - sdist command accidentally rolled back as part of the distutils2 rollback. +- Issue #19544 and Issue #6516: Restore support for --user and --group + parameters to sdist command accidentally rolled back as part of the + distutils2 rollback. - Issue #13674: Prevented time.strftime from crashing on Windows when given a year before 1900 and a format of %y. @@ -1374,8 +1437,6 @@ - Issue #19324: Expose Linux-specific constants in resource module. -- Issue #17400: ipaddress should make it easy to identify rfc6598 addresses. - - Load SSL's error strings in hashlib. - Issue #18527: Upgrade internal copy of zlib to 1.2.8. @@ -1422,7 +1483,7 @@ - Issue #19085: Added basic tests for all tkinter widget options. -- Issue 19384: Fix test_py_compile for root user, patch by Claudiu Popa. +- Issue #19384: Fix test_py_compile for root user, patch by Claudiu Popa. Documentation ------------- @@ -1472,9 +1533,8 @@ - Issue #19551: PEP 453 - the OS X installer now installs pip by default. -- Update third-party libraries for OS X installers: - xz 5.0.3 -> 5.0.5 - SQLite 3.7.13 -> 3.8.1 +- Update third-party libraries for OS X installers: xz 5.0.3 -> 5.0.5, + SQLite 3.7.13 -> 3.8.1 - Issue #15663: Revert OS X installer built-in Tcl/Tk support for 3.4.0b1. Some third-party projects, such as Matplotlib and PIL/Pillow, @@ -1567,8 +1627,9 @@ - Issue #19254: Provide an optimized Python implementation of pbkdf2_hmac. -- Issues #19201, #19222, #19223: Add "x" mode (exclusive creation) in opening - file to bz2, gzip and lzma modules. Patches by Tim Heaney and Vajrasky Kok. +- Issues #19201, Issue #19222, Issue #19223: Add "x" mode (exclusive creation) + in opening file to bz2, gzip and lzma modules. Patches by Tim Heaney and + Vajrasky Kok. - Fix a reference count leak in _sre. @@ -1589,6 +1650,9 @@ - Issue #18281: Unused stat constants removed from `tarfile`. +- Issue #18999: Multiprocessing now supports 'contexts' with the same API + as the module, but bound to specified start methods. + - Issue #18468: The re.split, re.findall, and re.sub functions and the group() and groups() methods of match object now always return a string or a bytes object. @@ -1793,9 +1857,9 @@ - Issue #19018: The heapq.merge() function no longer suppresses IndexError in the underlying iterables. -- Issue #18784: The uuid module no more attempts to load libc via ctypes.CDLL, - if all necessary functions are already found in libuuid. - Patch by Evgeny Sologubov. +- Issue #18784: The uuid module no longer attempts to load libc via ctypes.CDLL + if all the necessary functions have already been found in libuuid. Patch by + Evgeny Sologubov. - The :envvar:`PYTHONFAULTHANDLER` environment variable now only enables the faulthandler module if the variable is non-empty. Same behaviour than other @@ -1850,9 +1914,6 @@ - Issue #18942: sys._debugmallocstats() output was damaged on Windows. -- Issue #18780: %-formatting now prints value instead of str for - int subclasses when using %d, %i, and %u codes. - - Issue #18571: Implementation of the PEP 446: file descriptors and file handles are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and @@ -1864,7 +1925,7 @@ - Issue #18808: Non-daemon threads are now automatically joined when a sub-interpreter is shutdown (it would previously dump a fatal error). -- Remove supporting for compiling on systems without getcwd(). +- Remove support for compiling on systems without getcwd(). - Issue #18774: Remove last bits of GNU PTH thread code and thread_pth.h. @@ -1910,7 +1971,7 @@ - Issue #18672: Fixed format specifiers for Py_ssize_t in debugging output in the _sre module. -- Issue #18830: inspect.getclasstree() no more produces duplicated entries even +- Issue #18830: inspect.getclasstree() no longer produces duplicate entries even when input list contains duplicates. - Issue #18878: sunau.open now supports the context manager protocol. Based on @@ -1957,7 +2018,7 @@ - Issue #18394: Close cgi.FieldStorage's optional file. -- Issue #17702: On error, os.environb now removes suppress the except context +- Issue #17702: On error, os.environb now suppresses the exception context when raising a new KeyError with the original key. - Issue #16809: Fixed some tkinter incompabilities with Tcl/Tk 8.6. @@ -2007,6 +2068,10 @@ - Issue #18532: Change the builtin hash algorithms' names to lower case names as promised by hashlib's documentation. +- Issue #8713: add new spwan and forkserver start methods, and new functions + get_all_start_methods, get_start_method, and set_start_method, to + multiprocessing. + - Issue #18405: Improve the entropy of crypt.mksalt(). - Issue #12015: The tempfile module now uses a suffix of 8 random characters @@ -2332,6 +2397,9 @@ - Issue #14850: Now a charmap decoder treats U+FFFE as "undefined mapping" in any mapping, not only in a string. +- Issue #16613: Add *m* argument to ``collections.Chainmap.new_child`` to + allow the new child map to be specified explicitly. + - Issue #16730: importlib.machinery.FileFinder now no longers raises an exception when trying to populate its cache and it finds out the directory is unreadable or has turned into a file. Reported and diagnosed by @@ -2390,10 +2458,10 @@ builtins. - Issue #16455: On FreeBSD and Solaris, if the locale is C, the - ASCII/surrogateescape codec is now used, instead of the locale encoding, to + ASCII/surrogateescape codec is now used (instead of the locale encoding) to decode the command line arguments. This change fixes inconsistencies with - os.fsencode() and os.fsdecode() because these operating systems announces an - ASCII locale encoding, whereas the ISO-8859-1 encoding is used in practice. + os.fsencode() and os.fsdecode(), because these operating systems announce an + ASCII locale encoding, but actually use the ISO-8859-1 encoding in practice. - Issue #16562: Optimize dict equality testing. Patch by Serhiy Storchaka. @@ -2484,6 +2552,9 @@ - Issue #16160: Subclass support now works for types.SimpleNamespace. +- Issue #16148: Implement PEP 424, adding operator.length_hint and + PyObject_LengthHint. + - Upgrade Unicode data (UCD) to version 6.2. - Issue #15379: Fix passing of non-BMP characters as integers for the charmap @@ -2996,7 +3067,7 @@ to procedurally generate, in an easy way, small test instances. - Issue #17485: Also delete the Request Content-Length header if the data - attribute is deleted. (Follow on to issue 16464). + attribute is deleted. (Follow on to issue Issue #16464). - Issue #15927: CVS now correctly parses escaped newlines and carriage when parsing with quoting turned off. @@ -3423,6 +3494,9 @@ list() calls aren't added to filter(), map(), and zip() which are directly passed enumerate(). +- Issue #16464: Reset the Content-Length header when a urllib Request is reused + with new data. + - Issue #12848: The pure Python pickle implementation now treats object lengths as unsigned 32-bit integers, like the C implementation does. Patch by Serhiy Storchaka. @@ -3573,6 +3647,9 @@ - Issue #16169: Fix ctypes.WinError()'s confusion between errno and winerror. +- Issue #16110: logging.fileConfig now accepts a pre-initialised ConfigParser + instance. + - Issue #1492704: shutil.copyfile() raises a distinct SameFileError now if source and destination are the same file. Patch by Atsuo Ishimoto. @@ -3591,6 +3668,10 @@ - Issue #9650: List commonly used format codes in time.strftime and time.strptime docsttings. Original patch by Mike Hoy. +- Issue #15452: logging configuration socket listener now has a verify option + that allows an application to apply a verification function to the + received configuration data before it is acted upon. + - Issue #16034: Fix performance regressions in the new `bz2.BZ2File` implementation. Initial patch by Serhiy Storchaka. @@ -6290,7 +6371,7 @@ deallocator calls one of the methods on the type (e.g. when subclassing IOBase). Diagnosis and patch by Davide Rizzo. -- Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on Windows. +- Issue #9611, Issue #9015: FileIO.read() clamps the length to INT_MAX on Windows. - Issue #9642: Uniformize the tests on the availability of the mbcs codec, add a new HAVE_MBCS define. @@ -6758,7 +6839,7 @@ 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', and private attributes of 'smtpd.SMTPChannel'. -- Issue #5905, #13560: time.strftime() is now using the current locale +- Issue #5905, Issue #13560: time.strftime() is now using the current locale encoding, instead of UTF-8, if the wcsftime() function is not available. - Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1640,9 +1640,9 @@ /* #define CHAR_ALIGN (sizeof(s_char) - sizeof(char)) #define SHORT_ALIGN (sizeof(s_short) - sizeof(short)) -#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define LONG_ALIGN (sizeof(s_long) - sizeof(long)) */ +#define INT_ALIGN (sizeof(s_int) - sizeof(int)) #define FLOAT_ALIGN (sizeof(s_float) - sizeof(float)) #define DOUBLE_ALIGN (sizeof(s_double) - sizeof(double)) #define LONGDOUBLE_ALIGN (sizeof(s_long_double) - sizeof(long double)) @@ -1684,8 +1684,8 @@ ffi_type ffi_type_uint16 = { 2, 2, FFI_TYPE_UINT16 }; ffi_type ffi_type_sint16 = { 2, 2, FFI_TYPE_SINT16 }; -ffi_type ffi_type_uint32 = { 4, 4, FFI_TYPE_UINT32 }; -ffi_type ffi_type_sint32 = { 4, 4, FFI_TYPE_SINT32 }; +ffi_type ffi_type_uint32 = { 4, INT_ALIGN, FFI_TYPE_UINT32 }; +ffi_type ffi_type_sint32 = { 4, INT_ALIGN, FFI_TYPE_SINT32 }; ffi_type ffi_type_uint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_UINT64 }; ffi_type ffi_type_sint64 = { 8, LONG_LONG_ALIGN, FFI_TYPE_SINT64 }; diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -2459,7 +2459,7 @@ struct tm *tm; time_t t; - if (_PyTime_ObjectToTime_t(obj, &t) == -1) + if (_PyTime_ObjectToTime_t(obj, &t, _PyTime_ROUND_DOWN) == -1) return NULL; tm = localtime(&t); @@ -4127,7 +4127,7 @@ time_t timet; long us; - if (_PyTime_ObjectToTimeval(timestamp, &timet, &us) == -1) + if (_PyTime_ObjectToTimeval(timestamp, &timet, &us, _PyTime_ROUND_DOWN) == -1) return NULL; return datetime_from_timet_and_us(cls, f, timet, (int)us, tzinfo); } diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -3928,9 +3928,6 @@ return ret; } -static mpd_uint_t data_zero[1] = {0}; -static const mpd_t zero = {MPD_STATIC|MPD_CONST_DATA, 0, 1, 1, 1, data_zero}; - static PyObject * nm_mpd_qpow(PyObject *base, PyObject *exp, PyObject *mod) { diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -2305,9 +2305,14 @@ static PyObject * _forward_call(buffered *self, _Py_Identifier *name, PyObject *args) { - PyObject *func = _PyObject_GetAttrId((PyObject *)self, name); - PyObject *ret; + PyObject *func, *ret; + if (self == NULL) { + PyErr_SetString(PyExc_ValueError, + "I/O operation on uninitialized object"); + return NULL; + } + func = _PyObject_GetAttrId((PyObject *)self, name); if (func == NULL) { PyErr_SetString(PyExc_AttributeError, name->string); return NULL; diff --git a/Modules/_math.c b/Modules/_math.c --- a/Modules/_math.c +++ b/Modules/_math.c @@ -22,7 +22,9 @@ static const double ln2 = 6.93147180559945286227E-01; static const double two_pow_m28 = 3.7252902984619141E-09; /* 2**-28 */ static const double two_pow_p28 = 268435456.0; /* 2**28 */ +#ifndef Py_NAN static const double zero = 0.0; +#endif /* acosh(x) * Method : diff --git a/Modules/_sre.c b/Modules/_sre.c --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -526,59 +526,49 @@ return sre_ucs4_search(state, pattern); } -/*[clinic input] -module _sre -class _sre.SRE_Pattern "PatternObject *" "&Pattern_Type" - -_sre.SRE_Pattern.match as pattern_match - - pattern: object - pos: Py_ssize_t = 0 - endpos: Py_ssize_t(c_default="PY_SSIZE_T_MAX") = sys.maxsize - -Matches zero or more characters at the beginning of the string. -[clinic start generated code]*/ - -PyDoc_STRVAR(pattern_match__doc__, -"match($self, /, pattern, pos=0, endpos=sys.maxsize)\n" -"--\n" -"\n" -"Matches zero or more characters at the beginning of the string."); - -#define PATTERN_MATCH_METHODDEF \ - {"match", (PyCFunction)pattern_match, METH_VARARGS|METH_KEYWORDS, pattern_match__doc__}, - static PyObject * -pattern_match_impl(PatternObject *self, PyObject *pattern, Py_ssize_t pos, Py_ssize_t endpos); +fix_string_param(PyObject *string, PyObject *string2, const char *oldname) +{ + if (string2 != NULL) { + if (string != NULL) { + PyErr_Format(PyExc_TypeError, + "Argument given by name ('%s') and position (1)", + oldname); + return NULL; + } + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "The '%s' keyword parameter name is deprecated. " + "Use 'string' instead.", oldname) < 0) + return NULL; + return string2; + } + if (string == NULL) { + PyErr_SetString(PyExc_TypeError, + "Required argument 'string' (pos 1) not found"); + return NULL; + } + return string; +} static PyObject * pattern_match(PatternObject *self, PyObject *args, PyObject *kwargs) { - PyObject *return_value = NULL; - static char *_keywords[] = {"pattern", "pos", "endpos", NULL}; - PyObject *pattern; + static char *_keywords[] = {"string", "pos", "endpos", "pattern", NULL}; + PyObject *string = NULL; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - - if (!PyArg_ParseTupleAndKeywords(args, kwargs, - "O|nn:match", _keywords, - &pattern, &pos, &endpos)) - goto exit; - return_value = pattern_match_impl(self, pattern, pos, endpos); - -exit: - return return_value; -} - -static PyObject * -pattern_match_impl(PatternObject *self, PyObject *pattern, Py_ssize_t pos, Py_ssize_t endpos) -/*[clinic end generated code: output=1528eafdb8b025ad input=26f9fd31befe46b9]*/ -{ + PyObject *pattern = NULL; SRE_STATE state; Py_ssize_t status; - PyObject *string; - - string = state_init(&state, (PatternObject *)self, pattern, pos, endpos); + + if (!PyArg_ParseTupleAndKeywords(args, kwargs, + "|Onn$O:match", _keywords, + &string, &pos, &endpos, &pattern)) + return NULL; + string = fix_string_param(string, pattern, "pattern"); + if (!string) + return NULL; + string = state_init(&state, (PatternObject *)self, string, pos, endpos); if (!string) return NULL; @@ -603,12 +593,16 @@ SRE_STATE state; Py_ssize_t status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:fullmatch", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:fullmatch", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -637,12 +631,16 @@ SRE_STATE state; Py_ssize_t status; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "pattern", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:search", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "pattern", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:search", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "pattern"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -718,12 +716,16 @@ Py_ssize_t status; Py_ssize_t i, b, e; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:findall", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:findall", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, start, end); @@ -840,11 +842,15 @@ Py_ssize_t i; void* last; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t maxsplit = 0; - static char* kwlist[] = { "source", "maxsplit", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|n:split", kwlist, - &string, &maxsplit)) + static char* kwlist[] = { "string", "maxsplit", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|On$O:split", kwlist, + &string, &maxsplit, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; string = state_init(&state, self, string, 0, PY_SSIZE_T_MAX); @@ -1292,6 +1298,10 @@ return result; } +PyDoc_STRVAR(pattern_match_doc, +"match(string[, pos[, endpos]]) -> match object or None.\n\ + Matches zero or more characters at the beginning of the string"); + PyDoc_STRVAR(pattern_fullmatch_doc, "fullmatch(string[, pos[, endpos]]) -> match object or None.\n\ Matches against all of the string"); @@ -1329,7 +1339,8 @@ PyDoc_STRVAR(pattern_doc, "Compiled regular expression objects"); static PyMethodDef pattern_methods[] = { - PATTERN_MATCH_METHODDEF + {"match", (PyCFunction) pattern_match, METH_VARARGS|METH_KEYWORDS, + pattern_match_doc}, {"fullmatch", (PyCFunction) pattern_fullmatch, METH_VARARGS|METH_KEYWORDS, pattern_fullmatch_doc}, {"search", (PyCFunction) pattern_search, METH_VARARGS|METH_KEYWORDS, @@ -2654,12 +2665,16 @@ ScannerObject* self; - PyObject* string; + PyObject *string = NULL, *string2 = NULL; Py_ssize_t start = 0; Py_ssize_t end = PY_SSIZE_T_MAX; - static char* kwlist[] = { "source", "pos", "endpos", NULL }; - if (!PyArg_ParseTupleAndKeywords(args, kw, "O|nn:scanner", kwlist, - &string, &start, &end)) + static char* kwlist[] = { "string", "pos", "endpos", "source", NULL }; + if (!PyArg_ParseTupleAndKeywords(args, kw, "|Onn$O:scanner", kwlist, + &string, &start, &end, &string2)) + return NULL; + + string = fix_string_param(string, string2, "source"); + if (!string) return NULL; /* create scanner object */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2516,14 +2516,27 @@ return PyLong_FromLong(r); } +static int +check_time_rounding(int round) +{ + if (round != _PyTime_ROUND_DOWN && round != _PyTime_ROUND_UP) { + PyErr_SetString(PyExc_ValueError, "invalid rounding"); + return -1; + } + return 0; +} + static PyObject * test_pytime_object_to_time_t(PyObject *self, PyObject *args) { PyObject *obj; time_t sec; - if (!PyArg_ParseTuple(args, "O:pytime_object_to_time_t", &obj)) + int round; + if (!PyArg_ParseTuple(args, "Oi:pytime_object_to_time_t", &obj, &round)) return NULL; - if (_PyTime_ObjectToTime_t(obj, &sec) == -1) + if (check_time_rounding(round) < 0) + return NULL; + if (_PyTime_ObjectToTime_t(obj, &sec, round) == -1) return NULL; return _PyLong_FromTime_t(sec); } @@ -2534,9 +2547,12 @@ PyObject *obj; time_t sec; long usec; - if (!PyArg_ParseTuple(args, "O:pytime_object_to_timeval", &obj)) + int round; + if (!PyArg_ParseTuple(args, "Oi:pytime_object_to_timeval", &obj, &round)) return NULL; - if (_PyTime_ObjectToTimeval(obj, &sec, &usec) == -1) + if (check_time_rounding(round) < 0) + return NULL; + if (_PyTime_ObjectToTimeval(obj, &sec, &usec, round) == -1) return NULL; return Py_BuildValue("Nl", _PyLong_FromTime_t(sec), usec); } @@ -2547,9 +2563,12 @@ PyObject *obj; time_t sec; long nsec; - if (!PyArg_ParseTuple(args, "O:pytime_object_to_timespec", &obj)) + int round; + if (!PyArg_ParseTuple(args, "Oi:pytime_object_to_timespec", &obj, &round)) return NULL; - if (_PyTime_ObjectToTimespec(obj, &sec, &nsec) == -1) + if (check_time_rounding(round) < 0) + return NULL; + if (_PyTime_ObjectToTimespec(obj, &sec, &nsec, round) == -1) return NULL; return Py_BuildValue("Nl", _PyLong_FromTime_t(sec), nsec); } diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2838,6 +2838,8 @@ return NULL; if (index < 0) index = 0; + else if (index > Py_SIZE(it->ao)) + index = Py_SIZE(it->ao); /* iterator exhausted */ it->index = index; Py_RETURN_NONE; } diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -906,7 +906,9 @@ "This is the smallest integral value >= x."); FUNC2(copysign, copysign, - "copysign(x, y)\n\nReturn x with the sign of y.") + "copysign(x, y)\n\nReturn a float with the magnitude (absolute value) " + "of x but the sign \nof y. On platforms that support signed zeros, " + "copysign(1.0, -0.0) \nreturns -1.0.\n") FUNC1(cos, cos, 0, "cos(x)\n\nReturn the cosine of x (measured in radians).") FUNC1(cosh, cosh, 1, diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -4901,9 +4901,9 @@ } utime.now = 0; if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), - &a_sec, &a_nsec) == -1 || + &a_sec, &a_nsec, _PyTime_ROUND_DOWN) == -1 || _PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), - &m_sec, &m_nsec) == -1) { + &m_sec, &m_nsec, _PyTime_ROUND_DOWN) == -1) { goto exit; } utime.atime_s = a_sec; diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -212,10 +212,18 @@ return NULL; } else { + /* On OpenBSD 5.4, timeval.tv_sec is a long. + * Example: long is 64-bit, whereas time_t is 32-bit. */ + time_t sec; + /* On OS X 64-bit, timeval.tv_usec is an int (and thus still 4 + bytes as required), but no longer defined by a long. */ + long usec; + if (_PyTime_ObjectToTimeval(tout, &sec, &usec, + _PyTime_ROUND_UP) == -1) + return NULL; #ifdef MS_WINDOWS - time_t sec; - if (_PyTime_ObjectToTimeval(tout, &sec, &tv.tv_usec) == -1) - return NULL; + /* On Windows, timeval.tv_sec is a long (32 bit), + * whereas time_t can be 64-bit. */ assert(sizeof(tv.tv_sec) == sizeof(long)); #if SIZEOF_TIME_T > SIZEOF_LONG if (sec > LONG_MAX) { @@ -226,13 +234,10 @@ #endif tv.tv_sec = (long)sec; #else - /* 64-bit OS X has struct timeval.tv_usec as an int (and thus still 4 - bytes as required), but no longer defined by a long. */ - long tv_usec; - if (_PyTime_ObjectToTimeval(tout, &tv.tv_sec, &tv_usec) == -1) - return NULL; - tv.tv_usec = tv_usec; + assert(sizeof(tv.tv_sec) >= sizeof(sec)); + tv.tv_sec = sec; #endif + tv.tv_usec = usec; if (tv.tv_sec < 0) { PyErr_SetString(PyExc_ValueError, "timeout must be non-negative"); return NULL; @@ -2037,8 +2042,8 @@ ptimeoutspec = NULL; } else if (PyNumber_Check(otimeout)) { - if (_PyTime_ObjectToTimespec(otimeout, - &timeout.tv_sec, &timeout.tv_nsec) == -1) + if (_PyTime_ObjectToTimespec(otimeout, &timeout.tv_sec, + &timeout.tv_nsec, _PyTime_ROUND_UP) == -1) return NULL; if (timeout.tv_sec < 0) { diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -799,7 +799,8 @@ &signals, &timeout)) return NULL; - if (_PyTime_ObjectToTimespec(timeout, &tv_sec, &tv_nsec) == -1) + if (_PyTime_ObjectToTimespec(timeout, &tv_sec, &tv_nsec, + _PyTime_ROUND_DOWN) == -1) return NULL; buf.tv_sec = tv_sec; buf.tv_nsec = tv_nsec; diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -193,7 +193,7 @@ if (!PyArg_ParseTuple(args, "iO:clock_settime", &clk_id, &obj)) return NULL; - if (_PyTime_ObjectToTimespec(obj, &tv_sec, &tv_nsec) == -1) + if (_PyTime_ObjectToTimespec(obj, &tv_sec, &tv_nsec, _PyTime_ROUND_DOWN) == -1) return NULL; tp.tv_sec = tv_sec; tp.tv_nsec = tv_nsec; @@ -341,7 +341,7 @@ whent = time(NULL); } else { - if (_PyTime_ObjectToTime_t(ot, &whent) == -1) + if (_PyTime_ObjectToTime_t(ot, &whent, _PyTime_ROUND_DOWN) == -1) return 0; } *pwhen = whent; @@ -823,7 +823,18 @@ time_t tt; if (!gettmarg(tup, &buf)) return NULL; +#ifdef _AIX + /* year < 1902 or year > 2037 */ + if (buf.tm_year < 2 || buf.tm_year > 137) { + /* Issue #19748: On AIX, mktime() doesn't report overflow error for + * timestamp < -2^31 or timestamp > 2**31-1. */ + PyErr_SetString(PyExc_OverflowError, + "mktime argument out of range"); + return NULL; + } +#else buf.tm_wday = -1; /* sentinel; original value ignored */ +#endif tt = mktime(&buf); /* Return value of -1 does not necessarily mean an error, but tm_wday * cannot remain set to -1 if mktime succeeded. */ diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -3025,9 +3025,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyByteArray_GET_SIZE(it->it_seq)) + index = PyByteArray_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2937,9 +2937,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyBytes_GET_SIZE(it->it_seq)) + index = PyBytes_GET_SIZE(it->it_seq); /* iterator exhausted */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -1970,7 +1970,7 @@ if (keys[i] == NULL) { for (i=i-1 ; i>=0 ; i--) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); goto keyfunc_fail; } @@ -2043,7 +2043,7 @@ if (keys != NULL) { for (i = 0; i < saved_ob_size; i++) Py_DECREF(keys[i]); - if (keys != &ms.temparray[saved_ob_size+1]) + if (saved_ob_size >= MERGESTATE_TEMP_SIZE/2) PyMem_FREE(keys); } @@ -2811,6 +2811,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; + else if (index > PyList_GET_SIZE(it->it_seq)) + index = PyList_GET_SIZE(it->it_seq); /* iterator exhausted */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -37,7 +37,9 @@ static PyObject * get_small_int(sdigit ival) { - PyObject *v = (PyObject*)(small_ints + ival + NSMALLNEGINTS); + PyObject *v; + assert(-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS); + v = (PyObject *)&small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -807,10 +807,11 @@ long index = PyLong_AsLong(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0 || index >= r->len) { - PyErr_SetString(PyExc_ValueError, "index out of range"); - return NULL; - } + /* silently clip the index value */ + if (index < 0) + index = 0; + else if (index > r->len) + index = r->len; /* exhausted iterator */ r->index = index; Py_RETURN_NONE; } @@ -985,6 +986,28 @@ static PyObject * longrangeiter_setstate(longrangeiterobject *r, PyObject *state) { + int cmp; + + /* clip the value */ + PyObject *zero = PyLong_FromLong(0); + if (zero == NULL) + return NULL; + cmp = PyObject_RichCompareBool(state, zero, Py_LT); + if (cmp > 0) { + Py_CLEAR(r->index); + r->index = zero; + Py_RETURN_NONE; + } + Py_DECREF(zero); + if (cmp < 0) + return NULL; + + cmp = PyObject_RichCompareBool(r->len, state, Py_LT); + if (cmp < 0) + return NULL; + if (cmp > 0) + state = r->len; + Py_CLEAR(r->index); r->index = state; Py_INCREF(r->index); diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -1011,8 +1011,8 @@ if (it->it_seq != NULL) { if (index < 0) index = 0; - else if (it->it_seq != NULL && index > PyTuple_GET_SIZE(it->it_seq)) - index = PyTuple_GET_SIZE(it->it_seq); + else if (index > PyTuple_GET_SIZE(it->it_seq)) + index = PyTuple_GET_SIZE(it->it_seq); /* exhausted iterator */ it->it_index = index; } Py_RETURN_NONE; diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2472,6 +2472,9 @@ type->tp_dictoffset = slotoffset; slotoffset += sizeof(PyObject *); } + else if (!type->tp_dictoffset) { + type->tp_dictoffset = base->tp_dictoffset; + } if (type->tp_dictoffset) { et->ht_cached_keys = _PyDict_NewKeysForClass(); } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9666,7 +9666,7 @@ PyObject *last_obj; unsigned int kind = 0; - fseq = PySequence_Fast(seq, ""); + fseq = PySequence_Fast(seq, "can only join an iterable"); if (fseq == NULL) { return NULL; } @@ -15196,9 +15196,13 @@ Py_ssize_t index = PyLong_AsSsize_t(state); if (index == -1 && PyErr_Occurred()) return NULL; - if (index < 0) - index = 0; - it->it_index = index; + if (it->it_seq != NULL) { + if (index < 0) + index = 0; + else if (index > PyUnicode_GET_LENGTH(it->it_seq)) + index = PyUnicode_GET_LENGTH(it->it_seq); /* iterator truncated */ + it->it_index = index; + } Py_RETURN_NONE; } diff --git a/PC/winreg.c b/PC/winreg.c --- a/PC/winreg.c +++ b/PC/winreg.c @@ -943,8 +943,10 @@ fixupMultiSZ(str, data, len); obData = PyList_New(s); - if (obData == NULL) + if (obData == NULL) { + PyMem_Free(str); return NULL; + } for (index = 0; index < s; index++) { size_t len = wcslen(str[index]); @@ -952,6 +954,7 @@ PyErr_SetString(PyExc_OverflowError, "registry string is too long for a Python string"); Py_DECREF(obData); + PyMem_Free(str); return NULL; } PyList_SetItem(obData, diff --git a/PCbuild/pywlauncher.vcxproj b/PCbuild/pywlauncher.vcxproj --- a/PCbuild/pywlauncher.vcxproj +++ b/PCbuild/pywlauncher.vcxproj @@ -240,6 +240,11 @@ + + + {f0e0541e-f17d-430b-97c4-93adf0dd284e} + + diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py --- a/Parser/asdl_c.py +++ b/Parser/asdl_c.py @@ -1203,10 +1203,14 @@ mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode) { mod_ty res; - PyObject *req_type[] = {(PyObject*)Module_type, (PyObject*)Expression_type, - (PyObject*)Interactive_type}; + PyObject *req_type[3]; char *req_name[] = {"Module", "Expression", "Interactive"}; int isinstance; + + req_type[0] = (PyObject*)Module_type; + req_type[1] = (PyObject*)Expression_type; + req_type[2] = (PyObject*)Interactive_type; + assert(0 <= mode && mode <= 2); if (!init_types()) diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -498,9 +498,13 @@ fd = fileno(tok->fp); /* Due to buffering the file offset for fd can be different from the file - * position of tok->fp. */ + * position of tok->fp. If tok->fp was opened in text mode on Windows, + * its file position counts CRLF as one char and can't be directly mapped + * to the file offset for fd. Instead we step back one byte and read to + * the end of line.*/ pos = ftell(tok->fp); - if (pos == -1 || lseek(fd, (off_t)pos, SEEK_SET) == (off_t)-1) { + if (pos == -1 || + lseek(fd, (off_t)(pos > 0 ? pos - 1 : pos), SEEK_SET) == (off_t)-1) { PyErr_SetFromErrnoWithFilename(PyExc_OSError, NULL); goto cleanup; } @@ -513,6 +517,12 @@ Py_XDECREF(tok->decoding_readline); readline = _PyObject_GetAttrId(stream, &PyId_readline); tok->decoding_readline = readline; + if (pos > 0) { + if (PyObject_CallObject(readline, NULL) == NULL) { + readline = NULL; + goto cleanup; + } + } cleanup: Py_XDECREF(stream); diff --git a/Python/Python-ast.c b/Python/Python-ast.c --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -7023,10 +7023,14 @@ mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode) { mod_ty res; - PyObject *req_type[] = {(PyObject*)Module_type, (PyObject*)Expression_type, - (PyObject*)Interactive_type}; + PyObject *req_type[3]; char *req_name[] = {"Module", "Expression", "Interactive"}; int isinstance; + + req_type[0] = (PyObject*)Module_type; + req_type[1] = (PyObject*)Expression_type; + req_type[2] = (PyObject*)Interactive_type; + assert(0 <= mode && mode <= 2); if (!init_types()) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -200,8 +200,6 @@ { char msgbuf[256]; int levels[32]; - freelistentry_t static_entries[STATIC_FREELIST_ENTRIES]; - freelist_t freelist = {static_entries, 0, 0}; const char *fname = NULL; const char *message = NULL; int min = -1; @@ -212,6 +210,12 @@ Py_ssize_t i, len; char *msg; int compat = flags & FLAG_COMPAT; + freelistentry_t static_entries[STATIC_FREELIST_ENTRIES]; + freelist_t freelist; + + freelist.entries = static_entries; + freelist.first_available = 0; + freelist.entries_malloced = 0; assert(compat || (args != (PyObject*)NULL)); flags = flags & ~FLAG_COMPAT; @@ -1439,7 +1443,11 @@ Py_ssize_t nargs, nkeywords; PyObject *current_arg; freelistentry_t static_entries[STATIC_FREELIST_ENTRIES]; - freelist_t freelist = {static_entries, 0, 0}; + freelist_t freelist; + + freelist.entries = static_entries; + freelist.first_available = 0; + freelist.entries_malloced = 0; assert(args != NULL && PyTuple_Check(args)); assert(keywords == NULL || PyDict_Check(keywords)); diff --git a/Python/pytime.c b/Python/pytime.c --- a/Python/pytime.c +++ b/Python/pytime.c @@ -152,7 +152,7 @@ static int _PyTime_ObjectToDenominator(PyObject *obj, time_t *sec, long *numerator, - double denominator) + double denominator, _PyTime_round_t round) { assert(denominator <= LONG_MAX); if (PyFloat_Check(obj)) { @@ -167,6 +167,20 @@ intpart -= 1.0; } + floatpart *= denominator; + if (round == _PyTime_ROUND_UP) { + if (intpart >= 0) { + floatpart = ceil(floatpart); + if (floatpart >= denominator) { + floatpart = 0.0; + intpart += 1.0; + } + } + else { + floatpart = floor(floatpart); + } + } + *sec = (time_t)intpart; err = intpart - (double)*sec; if (err <= -1.0 || err >= 1.0) { @@ -174,7 +188,6 @@ return -1; } - floatpart *= denominator; *numerator = (long)floatpart; return 0; } @@ -188,12 +201,18 @@ } int -_PyTime_ObjectToTime_t(PyObject *obj, time_t *sec) +_PyTime_ObjectToTime_t(PyObject *obj, time_t *sec, _PyTime_round_t round) { if (PyFloat_Check(obj)) { double d, intpart, err; d = PyFloat_AsDouble(obj); + if (round == _PyTime_ROUND_UP) { + if (d >= 0) + d = ceil(d); + else + d = floor(d); + } (void)modf(d, &intpart); *sec = (time_t)intpart; @@ -213,15 +232,17 @@ } int -_PyTime_ObjectToTimespec(PyObject *obj, time_t *sec, long *nsec) +_PyTime_ObjectToTimespec(PyObject *obj, time_t *sec, long *nsec, + _PyTime_round_t round) { - return _PyTime_ObjectToDenominator(obj, sec, nsec, 1e9); + return _PyTime_ObjectToDenominator(obj, sec, nsec, 1e9, round); } int -_PyTime_ObjectToTimeval(PyObject *obj, time_t *sec, long *usec) +_PyTime_ObjectToTimeval(PyObject *obj, time_t *sec, long *usec, + _PyTime_round_t round) { - return _PyTime_ObjectToDenominator(obj, sec, usec, 1e6); + return _PyTime_ObjectToDenominator(obj, sec, usec, 1e6, round); } void diff --git a/Tools/buildbot/test-amd64.bat b/Tools/buildbot/test-amd64.bat --- a/Tools/buildbot/test-amd64.bat +++ b/Tools/buildbot/test-amd64.bat @@ -1,3 +1,3 @@ @rem Used by the buildbot "test" step. cd PCbuild -call rt.bat -d -q -x64 -uall -rwW -n %1 %2 %3 %4 %5 %6 %7 %8 %9 +call rt.bat -d -q -x64 -uall -rwW -n --timeout=3600 %1 %2 %3 %4 %5 %6 %7 %8 %9 diff --git a/Tools/hg/hgtouch.py b/Tools/hg/hgtouch.py --- a/Tools/hg/hgtouch.py +++ b/Tools/hg/hgtouch.py @@ -125,6 +125,6 @@ cmdtable = { "touch": (touch, - [('b', 'basedir', '', 'base dir of the tree to apply touching', 'BASEDIR')], + [('b', 'basedir', '', 'base dir of the tree to apply touching')], "hg touch [-b BASEDIR]") } diff --git a/Tools/scripts/run_tests.py b/Tools/scripts/run_tests.py --- a/Tools/scripts/run_tests.py +++ b/Tools/scripts/run_tests.py @@ -32,6 +32,12 @@ ] # Allow user-specified interpreter options to override our defaults. args.extend(test.support.args_from_interpreter_flags()) + + # Workaround for issue #20355 + os.environ.pop("PYTHONWARNINGS", None) + # Workaround for issue #20361 + args.extend(['-W', 'error::BytesWarning']) + args.extend(['-m', 'test', # Run the test suite '-r', # Randomize test order '-w', # Re-run failed tests in verbose mode diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1138,7 +1138,6 @@ define_macros=sqlite_defines, include_dirs=include_dirs, library_dirs=sqlite_libdir, - runtime_library_dirs=sqlite_libdir, extra_link_args=sqlite_extra_link_args, libraries=["sqlite3",])) else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:49 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_from_3=2E4_into_default=2E?= Message-ID: <3fnSQP3j8zz7LkX@mail.python.org> http://hg.python.org/cpython/rev/46e0ee06bafa changeset: 89804:46e0ee06bafa parent: 89691:5cab0ada97b2 parent: 89803:3a3a83195159 user: Larry Hastings date: Sun Mar 16 23:01:31 2014 -0700 summary: Merge from 3.4 into default. files: .hgtags | 3 + Doc/license.rst | 6 +- Doc/whatsnew/3.4.rst | 7 +- Include/patchlevel.h | 6 +- LICENSE | 4 +- Misc/NEWS | 138 ++++++++++++++---------------- 6 files changed, 77 insertions(+), 87 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -131,3 +131,6 @@ ba32913eb13ec545a46dd0ce18035b6c416f0d78 v3.4.0b2 a97ce3ecc96af79bd2e1ac66ce48d9138e0ca749 v3.4.0b3 5e088cea8660677969113741c1313d570d977e02 v3.4.0rc1 +a300712ed38c9a242b736c44e806caea25a6dc05 v3.4.0rc2 +8a81cdab3e9d521daaef989fade94b16455fc3b8 v3.4.0rc3 +04f714765c13824c3bc2835d7b008908862e083a v3.4.0 diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,9 +84,9 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ? 2001-2014 Python Software Foundation; All Rights - Reserved" are retained in Python |release| alone or in any derivative version - prepared by Licensee. + copyright, i.e., "Copyright ? 2001-2014 Python Software Foundation; All + Rights Reserved" are retained in Python |release| alone or in any derivative + version prepared by Licensee. #. In the event Licensee prepares a derivative work that is based on or incorporates Python |release| or any part thereof, and wants to make the diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -61,10 +61,7 @@ while :ref:`~mod.attr` will display as ``attr``. This article explains the new features in Python 3.4, compared to 3.3. - -.. Python 3.4 was released on TBD. - -For full details, see the +Python 3.4 was released on March 16, 2014. For full details, see the `changelog `_. @@ -2284,7 +2281,7 @@ name, they now set it to an empty list. The previous behavior could cause the import system to do the wrong thing on submodule imports if there was also a directory with the same name as the frozen package. The correct way - to determine if a module is a package or not is to use``hasattr(module, + to determine if a module is a package or not is to use ``hasattr(module, '__path__')`` (:issue:`18065`). * Frozen modules no longer define a ``__file__`` attribute. It's semantically diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 4 #define PY_MICRO_VERSION 0 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.4.0rc1+" +#define PY_VERSION "3.4.0+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained -in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are +retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,8 +2,10 @@ Python News +++++++++++ -What's New in Python 3.4.1? -=========================== +What's New in Python 3.4.1rc1? +============================== + +Release date: TBA Core and Builtins ----------------- @@ -16,9 +18,6 @@ - Issue #20637: Key-sharing now also works for instance dictionaries of subclasses. Patch by Peter Ingebretson. -- Issue #20786: Fix signatures for dict.__delitem__ and - property.__delete__ builtins. - Library ------- @@ -43,10 +42,6 @@ as documented. The pattern and source keyword parameters are left as deprecated aliases. -- Issue #20839: Don't trigger a DeprecationWarning in the still supported - pkgutil.get_loader() API when __loader__ isn't set on a module (nor - when pkgutil.find_loader() is called directly). - - Issue #20778: Fix modulefinder to work with bytecode-only modules. - Issue #20791: copy.copy() now doesn't make a copy when the input is @@ -72,11 +67,47 @@ - Issue #20743: Fix a reference leak in test_tcl. -- Issue #20946: Correct alignment assumptions of some ctypes tests. +Tools/Demos +----------- + +- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. + Patch by Arfrever Frehtes Taifersar Arahesis. + + +What's New in Python 3.4.0? +=========================== + +Release date: 2014-03-16 + +Library +------- - Issue #20939: Fix test_geturl failure in test_urllibnet due to new redirect of http://www.python.org/ to https://www.python.org. +Documentation +------------- + +- Merge in all documentation changes since branching 3.4.0rc1. + + +What's New in Python 3.4.0 release candidate 3? +=============================================== + +Release date: 2014-03-09 + +Core and Builtins +----------------- + +- Issue #20786: Fix signatures for dict.__delitem__ and + property.__delete__ builtins. + +Library +------- + +- Issue #20839: Don't trigger a DeprecationWarning in the still supported + pkgutil.get_loader() API when __loader__ isn't set on a module (nor + when pkgutil.find_loader() is called directly). Build ----- @@ -93,15 +124,9 @@ uninstalling pip (rather than failing) if the user has updated pip to a different version from the one bundled with ensurepip. -- Issue #20465: Update OS X and Windows installer builds to use +- Issue #20465: Update OS X and Windows installer builds to use SQLite 3.8.3.1. -Tools/Demos ------------ - -- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. - Patch by Arfrever Frehtes Taifersar Arahesis. - What's New in Python 3.4.0 release candidate 2? =============================================== @@ -111,6 +136,9 @@ Core and Builtins ----------------- +- Issue #20625: Parameter names in __annotations__ were not mangled properly. + Discovered by Jonas Wielicki, patch by Yury Selivanov. + - Issue #20261: In pickle, lookup __getnewargs__ and __getnewargs_ex__ on the type of the object. @@ -121,61 +149,12 @@ don't clear anymore the state of Python threads early during the Python shutdown. -- Issue #20595: Make getargs.c C89 compliant. - -- Issue #20625: Parameter names in __annotations__ were not mangled properly. - Discovered by Jonas Wielicki, patch by Yury Selivanov. - Library ------- - Issue #20710: The pydoc summary line no longer displays the "self" parameter for bound methods. -- Issue #20654: Fixed pydoc for enums with zero value. Patch by Vajrasky Kok. - -- Issue #20635: Fixed grid_columnconfigure() and grid_rowconfigure() methods of - Tkinter widgets to work in wantobjects=True mode. - -- Issue #19612: On Windows, subprocess.Popen.communicate() now ignores - OSError(22, 'Invalid argument') when writing input data into stdin, whereas - the process already exited. - -- Issue #20320: select.select() and select.kqueue.control() now round the - timeout aways from zero, instead of rounding towards zero. - -- Issue #20616: Add a format() method to tracemalloc.Traceback. - -- Issue #19744: the ensurepip installation step now just prints a warning to - stderr rather than failing outright if SSL/TLS is unavailable. This allows - local installation of POSIX builds without SSL/TLS support. - -- Issue #6815: os.path.expandvars() now supports non-ASCII environment - variables names and values. - -- Issue #17671: Fixed a crash when use non-initialized io.BufferedRWPair. - Based on patch by Stephen Tu. - -- Issue #8478: Untokenizer.compat processes first token from iterator input. - Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees. - -- Issue #20594: Avoid name clash with the libc function posix_close. - -- Issue #19856: shutil.move() failed to move a directory to other directory - on Windows if source name ends with os.altsep. - -- Issue #20673: Implement support for UNIX Domain Sockets in asyncio. - New APIs: loop.create_unix_connection(), loop.create_unix_server(), - streams.open_unix_connection(), and streams.start_unix_server(). - -- Issue #20681: Add new error handling API in asyncio. New APIs: - loop.set_exception_handler(), loop.default_exception_handler(), and - loop.call_exception_handler(). - -- Issue #20684: Fix inspect.getfullargspec() to not to follow __wrapped__ - chains. Make its behaviour consistent with bound methods first argument. - Patch by Nick Coghlan and Yury Selivanov. - - Issue #20566: Change asyncio.as_completed() to use a Queue, to avoid O(N**2) behavior. @@ -195,14 +174,25 @@ fix _check_resolved_address() for IPv6 address; and other minor improvements, along with multiple documentation updates. -Tests ------ - -- Issue #20510: Rewrote test_exit in test_sys to match existing comments, - use modern unittest features, and use helpers from test.script_helper - instead of using subprocess directly. Patch by Gareth Rees. - -- Issue #20605: Make test_socket getaddrinfo OS X segfault test more robust. +- Issue #20684: Fix inspect.getfullargspec() to not to follow __wrapped__ + chains. Make its behaviour consistent with bound methods first argument. + Patch by Nick Coghlan and Yury Selivanov. + +- Issue #20681: Add new error handling API in asyncio. New APIs: + loop.set_exception_handler(), loop.default_exception_handler(), and + loop.call_exception_handler(). + +- Issue #20673: Implement support for UNIX Domain Sockets in asyncio. + New APIs: loop.create_unix_connection(), loop.create_unix_server(), + streams.open_unix_connection(), and streams.start_unix_server(). + +- Issue #20616: Add a format() method to tracemalloc.Traceback. + +- Issue #19744: the ensurepip installation step now just prints a warning to + stderr rather than failing outright if SSL/TLS is unavailable. This allows + local installation of POSIX builds without SSL/TLS support. + +- Issue #20594: Avoid name clash with the libc function posix_close. Build ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:50 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Version_bump_to_3=2E5=2C_s?= =?utf-8?q?tep_1=2E?= Message-ID: <3fnSQQ67M3z7Ljt@mail.python.org> http://hg.python.org/cpython/rev/2cad8c670fb7 changeset: 89805:2cad8c670fb7 user: Larry Hastings date: Sun Mar 16 23:02:45 2014 -0700 summary: Version bump to 3.5, step 1. files: Misc/RPM/python-3.4.spec | 0 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Misc/RPM/python-3.4.spec b/Misc/RPM/python-3.5.spec rename from Misc/RPM/python-3.4.spec rename to Misc/RPM/python-3.5.spec -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:33:52 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 07:33:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Version_bump_to_3=2E5=2C_s?= =?utf-8?q?tep_2=2E?= Message-ID: <3fnSQS2yK2z7Ljt@mail.python.org> http://hg.python.org/cpython/rev/f2bf12fa22c1 changeset: 89806:f2bf12fa22c1 user: Larry Hastings date: Sun Mar 16 23:05:59 2014 -0700 summary: Version bump to 3.5, step 2. files: Doc/tutorial/interpreter.rst | 14 +++++++------- Doc/tutorial/stdlib.rst | 2 +- Doc/tutorial/stdlib2.rst | 2 +- Include/patchlevel.h | 6 +++--- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 11 +++++++++++ Misc/RPM/python-3.5.spec | 4 ++-- README | 16 ++++++++-------- configure | 20 ++++++++++---------- configure.ac | 2 +- 11 files changed, 46 insertions(+), 35 deletions(-) diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.4` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.4 + python3.5 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python34`, though you can change this when you're running the +:file:`C:\\Python35`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python34 + set path=%path%;C:\python35 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -95,8 +95,8 @@ prints a welcome message stating its version number and a copyright notice before printing the first prompt:: - $ python3.4 - Python 3.4 (default, Sep 24 2012, 09:25:04) + $ python3.5 + Python 3.5 (default, Sep 24 2012, 09:25:04) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> @@ -149,7 +149,7 @@ On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #! /usr/bin/env python3.4 + #! /usr/bin/env python3.5 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python34' + 'C:\\Python35' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -277,7 +277,7 @@ Traceback (most recent call last): File "", line 1, in d['primary'] # entry was automatically removed - File "C:/python34/lib/weakref.py", line 46, in __getitem__ + File "C:/python35/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -17,13 +17,13 @@ /* Version parsed out into numeric values */ /*--start constants--*/ #define PY_MAJOR_VERSION 3 -#define PY_MINOR_VERSION 4 +#define PY_MINOR_VERSION 5 #define PY_MICRO_VERSION 0 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.4.0+" +#define PY_VERSION "3.5.0a0" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.4.0rc1" +__version__ = "3.5.0a0" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.4.0rc1" +IDLE_VERSION = "3.5.0a0" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,17 @@ Python News +++++++++++ +What's New in Python 3.5 alpha 1? +================================= + +Release date: TBA + +Core and Builtins +----------------- + +Library +------- + What's New in Python 3.4.1rc1? ============================== diff --git a/Misc/RPM/python-3.5.spec b/Misc/RPM/python-3.5.spec --- a/Misc/RPM/python-3.5.spec +++ b/Misc/RPM/python-3.5.spec @@ -39,8 +39,8 @@ %define name python #--start constants-- -%define version 3.4.0rc1 -%define libvers 3.4 +%define version 3.5.0a0 +%define libvers 3.5 #--end constants-- %define release 1pydotorg %define __prefix /usr diff --git a/README b/README --- a/README +++ b/README @@ -1,5 +1,5 @@ -This is Python version 3.4.0 release candidate 1 -================================================ +This is Python version 3.5.0 alpha 1 +==================================== Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation. All rights reserved. @@ -52,9 +52,9 @@ ---------- We try to have a comprehensive overview of the changes in the "What's New in -Python 3.4" document, found at +Python 3.5" document, found at - http://docs.python.org/3.4/whatsnew/3.4.html + http://docs.python.org/3.5/whatsnew/3.5.html For a more detailed change log, read Misc/NEWS (though this file, too, is incomplete, and also doesn't list anything merged in from the 2.7 release under @@ -67,9 +67,9 @@ Documentation ------------- -Documentation for Python 3.4 is online, updated daily: +Documentation for Python 3.5 is online, updated daily: - http://docs.python.org/3.4/ + http://docs.python.org/3.5/ It can also be downloaded in many formats for faster access. The documentation is downloadable in HTML, PDF, and reStructuredText formats; the latter version @@ -94,7 +94,7 @@ A source-to-source translation tool, "2to3", can take care of the mundane task of converting large amounts of source code. It is not a complete solution but is complemented by the deprecation warnings in 2.6. See -http://docs.python.org/3.4/library/2to3.html for more information. +http://docs.python.org/3.5/library/2to3.html for more information. Testing @@ -132,7 +132,7 @@ Install that version using "make install". Install all other versions using "make altinstall". -For example, if you want to install Python 2.6, 2.7 and 3.4 with 2.7 being the +For example, if you want to install Python 2.6, 2.7 and 3.5 with 2.7 being the primary version, you would execute "make install" in your 2.7 build directory and "make altinstall" in the others. diff --git a/configure b/configure --- a/configure +++ b/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for python 3.4. +# Generated by GNU Autoconf 2.69 for python 3.5. # # Report bugs to . # @@ -580,8 +580,8 @@ # Identity of this package. PACKAGE_NAME='python' PACKAGE_TARNAME='python' -PACKAGE_VERSION='3.4' -PACKAGE_STRING='python 3.4' +PACKAGE_VERSION='3.5' +PACKAGE_STRING='python 3.5' PACKAGE_BUGREPORT='http://bugs.python.org/' PACKAGE_URL='' @@ -1368,7 +1368,7 @@ # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures python 3.4 to adapt to many kinds of systems. +\`configure' configures python 3.5 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1433,7 +1433,7 @@ if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of python 3.4:";; + short | recursive ) echo "Configuration of python 3.5:";; esac cat <<\_ACEOF @@ -1580,7 +1580,7 @@ test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -python configure 3.4 +python configure 3.5 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -2419,7 +2419,7 @@ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by python $as_me 3.4, which was +It was created by python $as_me 3.5, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2989,7 +2989,7 @@ mv confdefs.h.new confdefs.h -VERSION=3.4 +VERSION=3.5 # Version number of Python's own shared library file. @@ -15900,7 +15900,7 @@ # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by python $as_me 3.4, which was +This file was extended by python $as_me 3.5, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -15962,7 +15962,7 @@ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -python config.status 3.4 +python config.status 3.5 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -3,7 +3,7 @@ dnl *********************************************** # Set VERSION so we only need to edit in one place (i.e., here) -m4_define(PYTHON_VERSION, 3.4) +m4_define(PYTHON_VERSION, 3.5) AC_PREREQ(2.65) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:38:30 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:38:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_doc_ver?= =?utf-8?q?sion_switcher_for_3=2E4/3=2E5=2E?= Message-ID: <3fnSWp5R3tz7Lkb@mail.python.org> http://hg.python.org/cpython/rev/4495a9b11137 changeset: 89807:4495a9b11137 branch: 2.7 parent: 89687:e5a09b09bb51 user: Georg Brandl date: Mon Mar 17 07:36:45 2014 +0100 summary: Update doc version switcher for 3.4/3.5. files: Doc/tools/sphinxext/static/version_switch.js | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/tools/sphinxext/static/version_switch.js b/Doc/tools/sphinxext/static/version_switch.js --- a/Doc/tools/sphinxext/static/version_switch.js +++ b/Doc/tools/sphinxext/static/version_switch.js @@ -2,7 +2,8 @@ 'use strict'; var all_versions = { - '3.4': 'dev (3.4)', + '3.5': 'dev (3.5)', + '3.4': '3.4', '3.3': '3.3', '3.2': '3.2', '2.7': '2.7', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:39:02 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:39:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Update_doc_ver?= =?utf-8?q?sion_switcher_for_3=2E4/3=2E5=2E?= Message-ID: <3fnSXQ4Bg1z7Lm2@mail.python.org> http://hg.python.org/cpython/rev/df2dbeef46da changeset: 89808:df2dbeef46da branch: 3.3 parent: 89690:68a257ca6be6 user: Georg Brandl date: Mon Mar 17 07:36:45 2014 +0100 summary: Update doc version switcher for 3.4/3.5. files: Doc/tools/sphinxext/static/version_switch.js | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/tools/sphinxext/static/version_switch.js b/Doc/tools/sphinxext/static/version_switch.js --- a/Doc/tools/sphinxext/static/version_switch.js +++ b/Doc/tools/sphinxext/static/version_switch.js @@ -2,7 +2,8 @@ 'use strict'; var all_versions = { - '3.4': 'dev (3.4)', + '3.5': 'dev (3.5)', + '3.4': '3.4', '3.3': '3.3', '3.2': '3.2', '2.7': '2.7', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:39:03 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:39:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_with_3=2E3?= Message-ID: <3fnSXR5kDtz7Lm7@mail.python.org> http://hg.python.org/cpython/rev/0c0f90e8df26 changeset: 89809:0c0f90e8df26 branch: 3.4 parent: 89803:3a3a83195159 parent: 89808:df2dbeef46da user: Georg Brandl date: Mon Mar 17 07:36:51 2014 +0100 summary: merge with 3.3 files: Doc/tools/sphinxext/static/version_switch.js | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/tools/sphinxext/static/version_switch.js b/Doc/tools/sphinxext/static/version_switch.js --- a/Doc/tools/sphinxext/static/version_switch.js +++ b/Doc/tools/sphinxext/static/version_switch.js @@ -2,7 +2,8 @@ 'use strict'; var all_versions = { - '3.4': 'dev (3.4)', + '3.5': 'dev (3.5)', + '3.4': '3.4', '3.3': '3.3', '3.2': '3.2', '2.7': '2.7', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 07:39:04 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 07:39:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3fnSXS75Dpz7Lm4@mail.python.org> http://hg.python.org/cpython/rev/a18227d96ee8 changeset: 89810:a18227d96ee8 parent: 89806:f2bf12fa22c1 parent: 89809:0c0f90e8df26 user: Georg Brandl date: Mon Mar 17 07:36:59 2014 +0100 summary: merge with 3.4 files: Doc/tools/sphinxext/static/version_switch.js | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/tools/sphinxext/static/version_switch.js b/Doc/tools/sphinxext/static/version_switch.js --- a/Doc/tools/sphinxext/static/version_switch.js +++ b/Doc/tools/sphinxext/static/version_switch.js @@ -2,7 +2,8 @@ 'use strict'; var all_versions = { - '3.4': 'dev (3.4)', + '3.5': 'dev (3.5)', + '3.4': '3.4', '3.3': '3.3', '3.2': '3.2', '2.7': '2.7', -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Mar 17 09:51:35 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 17 Mar 2014 09:51:35 +0100 Subject: [Python-checkins] Daily reference leaks (5cab0ada97b2): sum=0 Message-ID: results for 5cab0ada97b2 on branch "default" -------------------------------------------- test_site leaked [2, -2, 0] references, sum=0 test_site leaked [2, -2, 0] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogbscmH2', '-x'] From python-checkins at python.org Mon Mar 17 16:23:41 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 17 Mar 2014 16:23:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogY2xvc2VzICMyMDk2?= =?utf-8?q?0?= Message-ID: <3fnh9n50H2z7Ljb@mail.python.org> http://hg.python.org/cpython/rev/a2fd7e452167 changeset: 89811:a2fd7e452167 branch: 3.4 parent: 89809:0c0f90e8df26 user: R David Murray date: Mon Mar 17 11:20:29 2014 -0400 summary: closes #20960 files: Doc/library/sys.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -606,7 +606,7 @@ .. versionadded:: 3.2 - .. versionchanged: 3.4 + .. versionchanged:: 3.4 Added *algorithm*, *hash_bits* and *seed_bits* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 16:23:43 2014 From: python-checkins at python.org (r.david.murray) Date: Mon, 17 Mar 2014 16:23:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_fix_for_=2320960?= Message-ID: <3fnh9q0X26z7Lk3@mail.python.org> http://hg.python.org/cpython/rev/3a3a5467baa9 changeset: 89812:3a3a5467baa9 parent: 89810:a18227d96ee8 parent: 89811:a2fd7e452167 user: R David Murray date: Mon Mar 17 11:23:07 2014 -0400 summary: Merge fix for #20960 files: Doc/library/sys.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -606,7 +606,7 @@ .. versionadded:: 3.2 - .. versionchanged: 3.4 + .. versionchanged:: 3.4 Added *algorithm*, *hash_bits* and *seed_bits* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 18:23:22 2014 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 17 Mar 2014 18:23:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Remove_stray_s?= =?utf-8?q?emicolon?= Message-ID: <3fnkqt059Cz7Lrh@mail.python.org> http://hg.python.org/cpython/rev/2684a7f7ecbd changeset: 89813:2684a7f7ecbd branch: 3.4 parent: 89811:a2fd7e452167 user: Antoine Pitrou date: Mon Mar 17 18:22:41 2014 +0100 summary: Remove stray semicolon files: Lib/threading.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1143,7 +1143,7 @@ if not self._initialized: raise RuntimeError("Thread.__init__() not called") if self._started.is_set(): - raise RuntimeError("cannot set daemon status of active thread"); + raise RuntimeError("cannot set daemon status of active thread") self._daemonic = daemonic def isDaemon(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 18:23:23 2014 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 17 Mar 2014 18:23:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Remove_stray_semicolon?= Message-ID: <3fnkqv390Cz7LrR@mail.python.org> http://hg.python.org/cpython/rev/84ac037540d5 changeset: 89814:84ac037540d5 parent: 89812:3a3a5467baa9 parent: 89813:2684a7f7ecbd user: Antoine Pitrou date: Mon Mar 17 18:23:14 2014 +0100 summary: Remove stray semicolon files: Lib/threading.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1143,7 +1143,7 @@ if not self._initialized: raise RuntimeError("Thread.__init__() not called") if self._started.is_set(): - raise RuntimeError("cannot set daemon status of active thread"); + raise RuntimeError("cannot set daemon status of active thread") self._daemonic = daemonic def isDaemon(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 18:30:53 2014 From: python-checkins at python.org (larry.hastings) Date: Mon, 17 Mar 2014 18:30:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_The_final_update_to_PEP_429?= =?utf-8?q?=2C_the_Python_3=2E4_release_schedule=2E__=28I_think!=29?= Message-ID: <3fnl0Y5myJz7LjM@mail.python.org> http://hg.python.org/peps/rev/8a6a61241046 changeset: 5413:8a6a61241046 user: Larry Hastings date: Mon Mar 17 10:30:52 2014 -0700 summary: The final update to PEP 429, the Python 3.4 release schedule. (I think!) files: pep-0429.txt | 21 ++------------------- 1 files changed, 2 insertions(+), 19 deletions(-) diff --git a/pep-0429.txt b/pep-0429.txt --- a/pep-0429.txt +++ b/pep-0429.txt @@ -34,7 +34,7 @@ Release Schedule ================ -The releases so far: +The releases: - 3.4.0 alpha 1: August 3, 2013 - 3.4.0 alpha 2: September 9, 2013 @@ -45,27 +45,10 @@ - 3.4.0 beta 3: January 26, 2014 - 3.4.0 candidate 1: February 10, 2014 - 3.4.0 candidate 2: February 23, 2014 - -(Beta 1 is also "feature freeze"--no new features beyond this point.) - -The anticipated schedule for future releases: - - 3.4.0 candidate 3: March 9, 2014 - 3.4.0 final: March 16, 2014 -.. don't forget to update final date above as well - -These are the days we expect to release the software; -Python core developers should note that the revision -used for the release will generally be "tagged" the day -before. However the actual availability of the release -for download on python.org depends on the schedules of -the crew and the existence of any release-blocking issues. - -Since posting the original schedule, we've added "beta 3" and slipped -the subsequent releases by three weeks. This is because of last-minute -changes due to adoption of "Argument Clinic". The extra beta will -give these changes extra time to mature. +(Beta 1 was also "feature freeze"--no new features beyond this point.) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 19:01:52 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:01:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Typo?= Message-ID: <3fnlhJ68mzz7Ll4@mail.python.org> http://hg.python.org/cpython/rev/ff803599edea changeset: 89815:ff803599edea branch: 3.4 parent: 89813:2684a7f7ecbd user: Jesus Cea date: Mon Mar 17 19:00:48 2014 +0100 summary: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1456,7 +1456,7 @@ Christien Heimes in :issue:`8813`.) New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` -loads a set of dfault "certificate authority" (CA) certificates from default +loads a set of default "certificate authority" (CA) certificates from default locations, which vary according to the platform. It can be used to load both TLS web server authentication certificates (``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 19:01:55 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:01:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_MERGE=3A_Typo?= Message-ID: <3fnlhM129Mz7Lks@mail.python.org> http://hg.python.org/cpython/rev/3bb83f18851f changeset: 89816:3bb83f18851f parent: 89814:84ac037540d5 parent: 89815:ff803599edea user: Jesus Cea date: Mon Mar 17 19:01:11 2014 +0100 summary: MERGE: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1456,7 +1456,7 @@ Christien Heimes in :issue:`8813`.) New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` -loads a set of dfault "certificate authority" (CA) certificates from default +loads a set of default "certificate authority" (CA) certificates from default locations, which vary according to the platform. It can be used to load both TLS web server authentication certificates (``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 19:13:56 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:13:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Typo?= Message-ID: <3fnlyD4YQCzMs9@mail.python.org> http://hg.python.org/cpython/rev/b4bd114bf1ea changeset: 89817:b4bd114bf1ea branch: 3.4 parent: 89815:ff803599edea user: Jesus Cea date: Mon Mar 17 19:13:09 2014 +0100 summary: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1839,7 +1839,7 @@ have a 64 bit data type. Any performance differences in comparison with the older FNV algorithm are trivial. -The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to +The PEP adds additional fields to the :attr:`sys.hash_info` struct sequence to describe the hash algorithm in use by the currently executing binary. Otherwise, the PEP does not alter any existing CPython APIs. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 19:13:58 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:13:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_MERGE=3A_Typo?= Message-ID: <3fnlyG6PNWz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/ea89d4059296 changeset: 89818:ea89d4059296 parent: 89816:3bb83f18851f parent: 89817:b4bd114bf1ea user: Jesus Cea date: Mon Mar 17 19:13:27 2014 +0100 summary: MERGE: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1839,7 +1839,7 @@ have a 64 bit data type. Any performance differences in comparison with the older FNV algorithm are trivial. -The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to +The PEP adds additional fields to the :attr:`sys.hash_info` struct sequence to describe the hash algorithm in use by the currently executing binary. Otherwise, the PEP does not alter any existing CPython APIs. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 19:19:47 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 17 Mar 2014 19:19:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?cGVwczogUmVtb3ZlIDMuMy42OyAzLjMuNSBp?= =?utf-8?q?s_the_last_ordinary_bugfix_release=2E?= Message-ID: <3fnm4z17mLz7LkL@mail.python.org> http://hg.python.org/peps/rev/565af186603e changeset: 5414:565af186603e user: Georg Brandl date: Mon Mar 17 19:19:22 2014 +0100 summary: Remove 3.3.6; 3.3.5 is the last ordinary bugfix release. files: pep-0398.txt | 12 +++--------- 1 files changed, 3 insertions(+), 9 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -94,19 +94,13 @@ 3.3.5 schedule -------------- +Python 3.3.5 was be the last regular maintenance release before 3.3 entered +security-fix only mode. + - 3.3.5 candidate 1: February 22, 2014 - 3.3.5 candidate 2: March 1, 2014 - 3.3.5 final: March 8, 2014 -3.3.6 schedule --------------- - -Python 3.3.6 will be the last regular maintenance release, released at some -point after 3.4.0 final (see :pep:`429`), before 3.3 goes into security-fix -only mode. - -- 3.3.6 candidate 1: around May 2014 -- 3.3.6 final: two weeks later Features for 3.3 ================ -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 19:23:47 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:23:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Typo?= Message-ID: <3fnm9b02P2z7Lkw@mail.python.org> http://hg.python.org/cpython/rev/06ce648ac6ac changeset: 89819:06ce648ac6ac branch: 3.4 parent: 89817:b4bd114bf1ea user: Jesus Cea date: Mon Mar 17 19:22:59 2014 +0100 summary: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2350,7 +2350,7 @@ * :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake` now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not connected, instead of the previous behavior of raising an - :exc:`AttributError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` + :exc:`AttributeError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` will raise a :exc:`ValueError` if the handshake has not yet been done. * :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 19:23:48 2014 From: python-checkins at python.org (jesus.cea) Date: Mon, 17 Mar 2014 19:23:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_MERGE=3A_Typo?= Message-ID: <3fnm9c35zDz7LlV@mail.python.org> http://hg.python.org/cpython/rev/81ba9ce002d6 changeset: 89820:81ba9ce002d6 parent: 89818:ea89d4059296 parent: 89819:06ce648ac6ac user: Jesus Cea date: Mon Mar 17 19:23:14 2014 +0100 summary: MERGE: Typo files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2350,7 +2350,7 @@ * :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake` now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not connected, instead of the previous behavior of raising an - :exc:`AttributError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` + :exc:`AttributeError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` will raise a :exc:`ValueError` if the handshake has not yet been done. * :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 21:32:49 2014 From: python-checkins at python.org (eric.araujo) Date: Mon, 17 Mar 2014 21:32:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_SOABI_PEP_to_reflect_c?= =?utf-8?q?onfig_var_change_=28=2316754=29=2E?= Message-ID: <3fnq2T14dTz7Ljd@mail.python.org> http://hg.python.org/peps/rev/ccb679e5ae0e changeset: 5415:ccb679e5ae0e user: ?ric Araujo date: Mon Mar 17 16:32:35 2014 -0400 summary: Update SOABI PEP to reflect config var change (#16754). Thanks to Arfrever Frehtes Taifersar Arahesis for the patch. files: pep-3149.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-3149.txt b/pep-3149.txt --- a/pep-3149.txt +++ b/pep-3149.txt @@ -130,12 +130,12 @@ changed. The tag and extension module suffix are available through the ``sysconfig`` modules via the following variables:: - >>> sysconfig.get_config_var('SO') + >>> sysconfig.get_config_var('EXT_SUFFIX') '.cpython-32mu.so' >>> sysconfig.get_config_var('SOABI') 'cpython-32mu' -Note that ``$SOABI`` contains just the tag, while ``$SO`` includes the +Note that ``$SOABI`` contains just the tag, while ``$EXT_SUFFIX`` includes the platform extension for shared library files, and is the exact suffix added to the extension module name. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 21:38:39 2014 From: python-checkins at python.org (eric.araujo) Date: Mon, 17 Mar 2014 21:38:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Update_branch_names_after?= =?utf-8?b?IDMuNC4wIHJlbGVhc2UgKHJlZiAjMjA4OTAp?= Message-ID: <3fnq9C0zcZz7LjQ@mail.python.org> http://hg.python.org/devguide/rev/414a7e2067b2 changeset: 677:414a7e2067b2 user: ?ric Araujo date: Mon Mar 17 16:37:12 2014 -0400 summary: Update branch names after 3.4.0 release (ref #20890) files: committing.rst | 40 +++++++++++++++++++------------------- devcycle.rst | 12 ++++++---- setup.rst | 4 +- 3 files changed, 29 insertions(+), 27 deletions(-) diff --git a/committing.rst b/committing.rst --- a/committing.rst +++ b/committing.rst @@ -365,15 +365,15 @@ $ hg import --no-c http://bugs.python.org/url/to/the/patch.diff $ # review, run tests, run `make patchcheck` $ hg ci -m '#12345: fix some issue.' - $ # switch to 3.3 and port the changeset using `hg graft` - $ cd ../3.3 + $ # switch to 3.4 and port the changeset using `hg graft` + $ cd ../3.4 $ hg up $ hg graft 2.7 $ # switch to 3.x, merge, commit, and push everything $ cd ../3.x $ hg up - $ hg merge 3.3 - $ hg ci -m '#12345: merge with 3.3.' + $ hg merge 3.4 + $ hg ci -m '#12345: merge with 3.4.' $ hg push Unless noted otherwise, the rest of the page will assume you are using the @@ -414,25 +414,25 @@ Merging between different branches (within the same major version) ------------------------------------------------------------------ -Assume that Python 3.4 is the current in-development version of Python and that -you have a patch that should also be applied to Python 3.3. To properly port +Assume that Python 3.5 is the current in-development version of Python and that +you have a patch that should also be applied to Python 3.4. To properly port the patch to both versions of Python, you should first apply the patch to -Python 3.3:: +Python 3.4:: - cd 3.3 + cd 3.4 hg import --no-commit patch.diff # Compile; run the test suite hg ci -m '#12345: fix some issue.' -Then you can switch to the ``3.4`` clone, merge, run the tests and commit:: +Then you can switch to the ``3.5`` clone, merge, run the tests and commit:: - cd ../3.4 - hg merge 3.3 + cd ../3.5 + hg merge 3.4 # Fix any conflicts (e.g. ``hg revert -r default Misc/NEWS``); compile; run the test suite - hg ci -m '#12345: merge with 3.3.' + hg ci -m '#12345: merge with 3.4.' If you are not using the share extension, you will need to use -``hg pull ../3.3`` before being able to merge. +``hg pull ../3.4`` before being able to merge. .. note:: Even when porting an already committed patch, you should *still* check the @@ -444,13 +444,13 @@ Porting changesets between the two major Python versions (2.x and 3.x) ---------------------------------------------------------------------- -Assume you just committed something on ``2.7``, and want to port it to ``3.3``. +Assume you just committed something on ``2.7``, and want to port it to ``3.4``. You can use ``hg graft`` as follow:: - cd ../3.3 + cd ../3.4 hg graft 2.7 -This will port the latest changeset committed in the 2.7 clone to the 3.3 clone. +This will port the latest changeset committed in the 2.7 clone to the 3.4 clone. ``hg graft`` always commits automatically, except in case of conflicts, when you have to resolve them and run ``hg graft --continue`` afterwards. Instead of the branch name you can also specify a changeset id, and you can @@ -458,18 +458,18 @@ On older version of Mercurial where ``hg graft`` is not available, you can use:: - cd ../3.3 + cd ../3.4 hg export 2.7 | hg import - The result will be the same, but in case of conflict this will create ``.rej`` files rather than using Mercurial merge capabilities. -A third option is to apply manually the patch on ``3.3``. This is convenient +A third option is to apply manually the patch on ``3.4``. This is convenient when there are too many differences with ``2.7`` or when there is already a -specific patch for ``3.3``. +specific patch for ``3.4``. .. warning:: - Never use ``hg merge`` to port changes between 2.x and 3.x (or vice versa). + Never use ``hg merge`` to port changes between 2.x and 3.x (or vice versa). Long-term development of features diff --git a/devcycle.rst b/devcycle.rst --- a/devcycle.rst +++ b/devcycle.rst @@ -31,7 +31,7 @@ '''''''' There is a branch for each *feature version*, whether released or not (e.g. -2.7, 3.3). Development is handled separately for Python 2 and Python 3: +2.7, 3.5). Development is handled separately for Python 2 and Python 3: no merging happens between 2.x and 3.x branches. In each of the 2.x and 3.x realms, the branch for a feature version is always a @@ -40,8 +40,8 @@ Therefore, each change should be made **first** in the oldest branch to which it applies and forward-ported as appropriate: if a bug must be fixed in both Python -3.3 and 3.4, first fix it in ``3.3`` and then merge ``3.3`` into ``default`` -(which holds the future 3.4). +3.4 and 3.5, first fix it in ``3.4`` and then merge ``3.4`` into ``default`` +(which holds the future 3.5). .. _indevbranch: @@ -109,9 +109,11 @@ There are 5 open branches right now in the Mercurial repository: -- the ``default`` branch holds the future 3.4 version and descends from ``3.3`` +- the ``default`` branch holds the future 3.5 version and descends from ``3.4`` (future RM: Larry Hastings) -- the ``3.3`` branch holds bug fixes for future 3.3.x maintenance releases +- the ``3.4`` branch holds bug fixes for future 3.4.x maintenance releases + and descends from ``3.3`` (RM: Larry Hastings) +- the ``3.3`` branch holds security fixes for future 3.3.x maintenance releases and descends from ``3.2`` (RM: Georg Brandl) - the ``3.2`` branch holds security fixes for future 3.2.x security releases (RM: Georg Brandl) diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -49,9 +49,9 @@ If you want a working copy of an already-released version of Python, i.e., a version in :ref:`maintenance mode `, you can update your -working copy. For instance, to update your working copy to Python 3.3, do:: +working copy. For instance, to update your working copy to Python 3.4, do:: - hg update 3.3 + hg update 3.4 You will need to re-compile CPython when you do such an update. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon Mar 17 21:50:58 2014 From: python-checkins at python.org (eric.araujo) Date: Mon, 17 Mar 2014 21:50:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_typo_in_ex?= =?utf-8?q?ample_=28=2320963=29?= Message-ID: <3fnqRQ5wyMz7Ll1@mail.python.org> http://hg.python.org/cpython/rev/e725de5a2760 changeset: 89821:e725de5a2760 branch: 3.4 parent: 89819:06ce648ac6ac user: ?ric Araujo date: Mon Mar 17 16:48:13 2014 -0400 summary: Fix typo in example (#20963) files: Doc/library/unittest.mock-examples.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -934,8 +934,8 @@ the magic methods you specifically want: >>> mock = Mock() - >>> mock.__setitem__ = Mock(side_effect=getitem) - >>> mock.__getitem__ = Mock(side_effect=setitem) + >>> mock.__getitem__ = Mock(side_effect=getitem) + >>> mock.__setitem__ = Mock(side_effect=setitem) A *third* option is to use `MagicMock` but passing in `dict` as the `spec` (or `spec_set`) argument so that the `MagicMock` created only has -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 21:51:00 2014 From: python-checkins at python.org (eric.araujo) Date: Mon, 17 Mar 2014 21:51:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40?= Message-ID: <3fnqRS72bYz7LlP@mail.python.org> http://hg.python.org/cpython/rev/8acd5ebfe8e8 changeset: 89822:8acd5ebfe8e8 parent: 89820:81ba9ce002d6 parent: 89821:e725de5a2760 user: ?ric Araujo date: Mon Mar 17 16:50:35 2014 -0400 summary: Merge 3.4 files: Doc/library/unittest.mock-examples.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -934,8 +934,8 @@ the magic methods you specifically want: >>> mock = Mock() - >>> mock.__setitem__ = Mock(side_effect=getitem) - >>> mock.__getitem__ = Mock(side_effect=setitem) + >>> mock.__getitem__ = Mock(side_effect=getitem) + >>> mock.__setitem__ = Mock(side_effect=setitem) A *third* option is to use `MagicMock` but passing in `dict` as the `spec` (or `spec_set`) argument so that the `MagicMock` created only has -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 21:58:01 2014 From: python-checkins at python.org (zach.ware) Date: Mon, 17 Mar 2014 21:58:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Bump_Windows_build_to_3=2E?= =?utf-8?q?5?= Message-ID: <3fnqbY6bcBz7LjT@mail.python.org> http://hg.python.org/cpython/rev/e9f136c394e2 changeset: 89823:e9f136c394e2 user: Zachary Ware date: Mon Mar 17 15:57:38 2014 -0500 summary: Bump Windows build to 3.5 files: PC/VS9.0/kill_python.c | 14 +- PC/VS9.0/pyproject.vsprops | 2 +- PC/example_nt/example.vcproj | 4 +- PC/pyconfig.h | 4 +- PC/python3.def | 1398 +++++++++++----------- PC/python3.mak | 12 +- PC/python34gen.py | 8 +- PC/python34stub.def | 2 +- PCbuild/build_ssl.bat | 4 +- PCbuild/kill_python.c | 14 +- PCbuild/pyproject.props | 4 +- PCbuild/readme.txt | 2 +- 12 files changed, 734 insertions(+), 734 deletions(-) diff --git a/PC/VS9.0/kill_python.c b/PC/VS9.0/kill_python.c --- a/PC/VS9.0/kill_python.c +++ b/PC/VS9.0/kill_python.c @@ -62,7 +62,7 @@ continue; len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN; - wcsncpy_s(path, MAX_PATH+1, me.szExePath, len); + wcsncpy_s(path, MAX_PATH+1, me.szExePath, len); break; @@ -80,8 +80,8 @@ * looking for python processes. When we find one, verify it lives * in the same directory we live in. If it does, kill it. If we're * unable to kill it, treat this as a fatal error and return 1. - * - * The rationale behind this is that we're called at the start of the + * + * The rationale behind this is that we're called at the start of the * build process on the basis that we'll take care of killing any * running instances, such that the build won't encounter permission * denied errors during linking. If we can't kill one of the processes, @@ -104,11 +104,11 @@ do { /* - * XXX TODO: if we really wanted to be fancy, we could check the + * XXX TODO: if we really wanted to be fancy, we could check the * modules for all processes (not just the python[_d].exe ones) - * and see if any of our DLLs are loaded (i.e. python34[_d].dll), + * and see if any of our DLLs are loaded (i.e. python35[_d].dll), * as that would also inhibit our ability to rebuild the solution. - * Not worth loosing sleep over though; for now, a simple check + * Not worth loosing sleep over though; for now, a simple check * for just the python executable should be sufficient. */ @@ -119,7 +119,7 @@ /* It's a python process, so figure out which directory it's in... */ hsm = CreateToolhelp32Snapshot(TH32CS_SNAPMODULE, pe.th32ProcessID); if (hsm == INVALID_HANDLE_VALUE) - /* + /* * If our module snapshot fails (which will happen if we don't own * the process), just ignore it and continue. (It seems different * versions of Windows return different values for GetLastError() diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -38,7 +38,7 @@ /> $(SolutionDir) $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ false - + <_ProjectFileVersion>10.0.30319.1 <_PropertySheetDisplayName>amd64 @@ -13,7 +13,7 @@ $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ - python34$(PyDebugExt) + python35$(PyDebugExt) $(OutDir)python$(PyDebugExt).exe $(OutDir)kill_python$(PyDebugExt).exe ..\.. diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -37,7 +37,7 @@ Used to build Python with extra debugging capabilities, equivalent to using ./configure --with-pydebug on UNIX. All binaries built using this configuration have "_d" added to their name: - python34_d.dll, python_d.exe, parser_d.pyd, and so on. Both the + python35_d.dll, python_d.exe, parser_d.pyd, and so on. Both the build and rt (run test) batch files in this directory accept a -d option for debug builds. If you are building Python to help with development of CPython, you will most likely use this configuration. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:00:37 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 17 Mar 2014 22:00:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_correct_the_fi?= =?utf-8?q?x_for_=2320637=3B_allow_slot_descriptor_inheritance_to_take_pla?= =?utf-8?q?ce?= Message-ID: <3fnqfY4BRZz7Lkn@mail.python.org> http://hg.python.org/cpython/rev/afae24cb81d9 changeset: 89824:afae24cb81d9 branch: 3.4 parent: 89821:e725de5a2760 user: Benjamin Peterson date: Mon Mar 17 15:57:17 2014 -0500 summary: correct the fix for #20637; allow slot descriptor inheritance to take place before creating cached keys files: Lib/test/test_descr.py | 8 ++++++++ Objects/typeobject.c | 17 ++++++++--------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4414,6 +4414,14 @@ self.assertRaises(TypeError, case, 1, 2, 3) self.assertRaises(TypeError, case, 1, 2, foo=3) + def test_subclassing_does_not_duplicate_dict_descriptors(self): + class Base: + pass + class Sub(Base): + pass + self.assertIn("__dict__", Base.__dict__) + self.assertNotIn("__dict__", Sub.__dict__) + class DictProxyTests(unittest.TestCase): def setUp(self): diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2472,12 +2472,6 @@ type->tp_dictoffset = slotoffset; slotoffset += sizeof(PyObject *); } - else if (!type->tp_dictoffset) { - type->tp_dictoffset = base->tp_dictoffset; - } - if (type->tp_dictoffset) { - et->ht_cached_keys = _PyDict_NewKeysForClass(); - } if (add_weak) { assert(!base->tp_itemsize); type->tp_weaklistoffset = slotoffset; @@ -2527,6 +2521,10 @@ /* Put the proper slots in place */ fixup_slot_dispatchers(type); + if (type->tp_dictoffset) { + et->ht_cached_keys = _PyDict_NewKeysForClass(); + } + Py_DECREF(dict); return (PyObject *)type; @@ -2643,9 +2641,6 @@ type->tp_doc = tp_doc; } } - if (type->tp_dictoffset) { - res->ht_cached_keys = _PyDict_NewKeysForClass(); - } if (type->tp_dealloc == NULL) { /* It's a heap type, so needs the heap types' dealloc. subtype_dealloc will call the base type's tp_dealloc, if @@ -2656,6 +2651,10 @@ if (PyType_Ready(type) < 0) goto fail; + if (type->tp_dictoffset) { + res->ht_cached_keys = _PyDict_NewKeysForClass(); + } + /* Set type.__module__ */ s = strrchr(spec->name, '.'); if (s != NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:00:38 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 17 Mar 2014 22:00:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fnqfZ6xjpz7LlG@mail.python.org> http://hg.python.org/cpython/rev/007265795b1b changeset: 89825:007265795b1b parent: 89823:e9f136c394e2 parent: 89824:afae24cb81d9 user: Benjamin Peterson date: Mon Mar 17 16:00:27 2014 -0500 summary: merge 3.4 files: Lib/test/test_descr.py | 8 ++++++++ Objects/typeobject.c | 17 ++++++++--------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4414,6 +4414,14 @@ self.assertRaises(TypeError, case, 1, 2, 3) self.assertRaises(TypeError, case, 1, 2, foo=3) + def test_subclassing_does_not_duplicate_dict_descriptors(self): + class Base: + pass + class Sub(Base): + pass + self.assertIn("__dict__", Base.__dict__) + self.assertNotIn("__dict__", Sub.__dict__) + class DictProxyTests(unittest.TestCase): def setUp(self): diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2472,12 +2472,6 @@ type->tp_dictoffset = slotoffset; slotoffset += sizeof(PyObject *); } - else if (!type->tp_dictoffset) { - type->tp_dictoffset = base->tp_dictoffset; - } - if (type->tp_dictoffset) { - et->ht_cached_keys = _PyDict_NewKeysForClass(); - } if (add_weak) { assert(!base->tp_itemsize); type->tp_weaklistoffset = slotoffset; @@ -2527,6 +2521,10 @@ /* Put the proper slots in place */ fixup_slot_dispatchers(type); + if (type->tp_dictoffset) { + et->ht_cached_keys = _PyDict_NewKeysForClass(); + } + Py_DECREF(dict); return (PyObject *)type; @@ -2643,9 +2641,6 @@ type->tp_doc = tp_doc; } } - if (type->tp_dictoffset) { - res->ht_cached_keys = _PyDict_NewKeysForClass(); - } if (type->tp_dealloc == NULL) { /* It's a heap type, so needs the heap types' dealloc. subtype_dealloc will call the base type's tp_dealloc, if @@ -2656,6 +2651,10 @@ if (PyType_Ready(type) < 0) goto fail; + if (type->tp_dictoffset) { + res->ht_cached_keys = _PyDict_NewKeysForClass(); + } + /* Set type.__module__ */ s = strrchr(spec->name, '.'); if (s != NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:20:54 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 17 Mar 2014 22:20:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_move_SharedKey?= =?utf-8?q?Tests_to_test=5Fdescr?= Message-ID: <3fnr5y02P8z7LjQ@mail.python.org> http://hg.python.org/cpython/rev/1724846a8676 changeset: 89826:1724846a8676 branch: 3.4 parent: 89824:afae24cb81d9 user: Benjamin Peterson date: Mon Mar 17 16:20:12 2014 -0500 summary: move SharedKeyTests to test_descr files: Lib/test/test_descr.py | 24 +++++++++++++++++++++++- Lib/test/test_types.py | 26 ++------------------------ 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4974,11 +4974,33 @@ self._assert_is_copy(obj, objcopy2) +class SharedKeyTests(unittest.TestCase): + + @support.cpython_only + def test_subclasses(self): + # Verify that subclasses can share keys (per PEP 412) + class A: + pass + class B(A): + pass + + a, b = A(), B() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + a.x, a.y, a.z, a.w = range(4) + self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + a2 = A() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + b.u, b.v, b.w, b.t = range(4) + self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({})) + + def test_main(): # Run all local test cases, with PTypesLongInitTest first. support.run_unittest(PTypesLongInitTest, OperatorsTest, ClassPropertiesAndMethods, DictProxyTests, - MiscTests, PicklingTests) + MiscTests, PicklingTests, SharedKeyTests) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -1,6 +1,6 @@ # Python test set -- part 6, built-in types -from test.support import run_unittest, run_with_locale, cpython_only +from test.support import run_unittest, run_with_locale import collections import pickle import locale @@ -1170,31 +1170,9 @@ self.assertEqual(ns, ns_roundtrip, pname) -class SharedKeyTests(unittest.TestCase): - - @cpython_only - def test_subclasses(self): - # Verify that subclasses can share keys (per PEP 412) - class A: - pass - class B(A): - pass - - a, b = A(), B() - self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) - self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) - a.x, a.y, a.z, a.w = range(4) - self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) - a2 = A() - self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) - self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) - b.u, b.v, b.w, b.t = range(4) - self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({})) - - def test_main(): run_unittest(TypesTests, MappingProxyTests, ClassCreationTests, - SimpleNamespaceTests, SharedKeyTests) + SimpleNamespaceTests) if __name__ == '__main__': test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:20:55 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 17 Mar 2014 22:20:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fnr5z3L9Mz7Ll0@mail.python.org> http://hg.python.org/cpython/rev/c425f81f7c4b changeset: 89827:c425f81f7c4b parent: 89825:007265795b1b parent: 89826:1724846a8676 user: Benjamin Peterson date: Mon Mar 17 16:20:50 2014 -0500 summary: merge 3.4 files: Lib/test/test_descr.py | 24 +++++++++++++++++++++++- Lib/test/test_types.py | 26 ++------------------------ 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4974,11 +4974,33 @@ self._assert_is_copy(obj, objcopy2) +class SharedKeyTests(unittest.TestCase): + + @support.cpython_only + def test_subclasses(self): + # Verify that subclasses can share keys (per PEP 412) + class A: + pass + class B(A): + pass + + a, b = A(), B() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + a.x, a.y, a.z, a.w = range(4) + self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) + a2 = A() + self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) + self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) + b.u, b.v, b.w, b.t = range(4) + self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({})) + + def test_main(): # Run all local test cases, with PTypesLongInitTest first. support.run_unittest(PTypesLongInitTest, OperatorsTest, ClassPropertiesAndMethods, DictProxyTests, - MiscTests, PicklingTests) + MiscTests, PicklingTests, SharedKeyTests) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -1,6 +1,6 @@ # Python test set -- part 6, built-in types -from test.support import run_unittest, run_with_locale, cpython_only +from test.support import run_unittest, run_with_locale import collections import pickle import locale @@ -1170,31 +1170,9 @@ self.assertEqual(ns, ns_roundtrip, pname) -class SharedKeyTests(unittest.TestCase): - - @cpython_only - def test_subclasses(self): - # Verify that subclasses can share keys (per PEP 412) - class A: - pass - class B(A): - pass - - a, b = A(), B() - self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) - self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) - a.x, a.y, a.z, a.w = range(4) - self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) - a2 = A() - self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) - self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({})) - b.u, b.v, b.w, b.t = range(4) - self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({})) - - def test_main(): run_unittest(TypesTests, MappingProxyTests, ClassCreationTests, - SimpleNamespaceTests, SharedKeyTests) + SimpleNamespaceTests) if __name__ == '__main__': test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:21:03 2014 From: python-checkins at python.org (terry.reedy) Date: Mon, 17 Mar 2014 22:21:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_0361=3A_nothing_tentative?= =?utf-8?q?_about_releases_any_more_=3B-=29=2E?= Message-ID: <3fnr6752rpz7LjW@mail.python.org> http://hg.python.org/peps/rev/6c041ffb70e8 changeset: 5416:6c041ffb70e8 user: Terry Jan Reedy date: Mon Mar 17 17:20:46 2014 -0400 summary: PEP 0361: nothing tentative about releases any more ;-). files: pep-0361.txt | 48 +++++++++++++++++---------------------- 1 files changed, 21 insertions(+), 27 deletions(-) diff --git a/pep-0361.txt b/pep-0361.txt --- a/pep-0361.txt +++ b/pep-0361.txt @@ -57,33 +57,27 @@ Release Schedule - Note that this schedule is completely tentative. The number of alphas, - betas and release candidates will be determined as the release process - unfolds. The minimal schedule is: - - Feb 29 2008: Python 2.6a1 and 3.0a3 are released - Apr 02 2008: Python 2.6a2 and 3.0a4 are released - May 08 2008: Python 2.6a3 and 3.0a5 are released - Jun 18 2008: Python 2.6b1 and 3.0b1 are released - Jul 17 2008: Python 2.6b2 and 3.0b2 are released - Aug 20 2008: Python 2.6b3 and 3.0b3 are released - Sep 12 2008: Python 2.6rc1 is released - Sep 17 2008: Python 2.6rc2 and 3.0rc1 released - Oct 01 2008: Python 2.6 final released - Nov 06 2008: Python 3.0rc2 released - Nov 21 2008: Python 3.0rc3 released - Dec 03 2008: Python 3.0 final released - Dec 04 2008: Python 2.6.1 final released - Apr 14 2009: Python 2.6.2 final released - Oct 02 2009: Python 2.6.3 final released - Oct 25 2009: Python 2.6.4 final released - Mar 19 2010: Python 2.6.5 final released - Aug 24 2010: Python 2.6.6 final released - Jun 03 2011: Python 2.6.7 final released (security-only) - Apr 10 2012: Python 2.6.8 final released (security-only) - Oct 29 2013: Python 2.6.9 final released (security-only) - - See the public `Google calendar`_ + Feb 29 2008: Python 2.6a1 and 3.0a3 are released + Apr 02 2008: Python 2.6a2 and 3.0a4 are released + May 08 2008: Python 2.6a3 and 3.0a5 are released + Jun 18 2008: Python 2.6b1 and 3.0b1 are released + Jul 17 2008: Python 2.6b2 and 3.0b2 are released + Aug 20 2008: Python 2.6b3 and 3.0b3 are released + Sep 12 2008: Python 2.6rc1 is released + Sep 17 2008: Python 2.6rc2 and 3.0rc1 released + Oct 01 2008: Python 2.6 final released + Nov 06 2008: Python 3.0rc2 released + Nov 21 2008: Python 3.0rc3 released + Dec 03 2008: Python 3.0 final released + Dec 04 2008: Python 2.6.1 final released + Apr 14 2009: Python 2.6.2 final released + Oct 02 2009: Python 2.6.3 final released + Oct 25 2009: Python 2.6.4 final released + Mar 19 2010: Python 2.6.5 final released + Aug 24 2010: Python 2.6.6 final released + Jun 03 2011: Python 2.6.7 final released (security-only) + Apr 10 2012: Python 2.6.8 final released (security-only) + Oct 29 2013: Python 2.6.9 final released (security-only) Completed features for 3.0 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 17 22:34:00 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 22:34:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_3=2E3_items_of_Misc/N?= =?utf-8?q?EWS_to_Misc/HISTORY?= Message-ID: <3fnrP4363dz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/6ba468d4fa96 changeset: 89828:6ba468d4fa96 user: Victor Stinner date: Mon Mar 17 22:33:49 2014 +0100 summary: Move 3.3 items of Misc/NEWS to Misc/HISTORY In fact, items were already present by Misc/NEWS format was improved. files: Misc/HISTORY | 1223 +++++----- Misc/NEWS | 4133 -------------------------------------- 2 files changed, 628 insertions(+), 4728 deletions(-) diff --git a/Misc/HISTORY b/Misc/HISTORY --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -18,13 +18,13 @@ - Issue #16046: Fix loading sourceless legacy .pyo files. -- Issue #16060: Fix refcounting bug when __trunc__ returns an object - whose __int__ gives a non-integer. Patch by Serhiy Storchaka. +- Issue #16060: Fix refcounting bug when `__trunc__()` returns an object whose + `__int__()` gives a non-integer. Patch by Serhiy Storchaka. Extension Modules ----------------- -- Issue #16012: Fix a regression in pyexpat. The parser's UseForeignDTD() +- Issue #16012: Fix a regression in pyexpat. The parser's `UseForeignDTD()` method doesn't require an argument again. @@ -36,26 +36,26 @@ Core and Builtins ----------------- -- Issue #15900: Fix reference leak in PyUnicode_TranslateCharmap(). +- Issue #15900: Fix reference leak in `PyUnicode_TranslateCharmap()`. - Issue #15926: Fix crash after multiple reinitializations of the interpreter. - Issue #15895: Fix FILE pointer leak in one error branch of - PyRun_SimpleFileExFlags() when filename points to a pyc/pyo file, closeit - is false an and set_main_loader() fails. + `PyRun_SimpleFileExFlags()` when filename points to a pyc/pyo file, closeit is + false an and set_main_loader() fails. - Fixes for a few crash and memory leak regressions found by Coverity. Library ------- -- Issue #15882: Change _decimal to accept any coefficient tuple when - constructing infinities. This is done for backwards compatibility - with decimal.py: Infinity coefficients are undefined in _decimal - (in accordance with the specification). - -- Issue #15925: Fix a regression in email.util where the parsedate() and - parsedate_tz() functions did not return None anymore when the argument could +- Issue #15882: Change `_decimal` to accept any coefficient tuple when + constructing infinities. This is done for backwards compatibility with + decimal.py: Infinity coefficients are undefined in _decimal (in accordance + with the specification). + +- Issue #15925: Fix a regression in `email.util` where the `parsedate()` and + `parsedate_tz()` functions did not return None anymore when the argument could not be parsed. Extension Modules @@ -67,7 +67,7 @@ - Issue #15977: Fix memory leak in Modules/_ssl.c when the function _set_npn_protocols() is called multiple times, thanks to Daniel Sommermann. -- Issue #15969: faulthandler module: rename dump_tracebacks_later() to +- Issue #15969: `faulthandler` module: rename dump_tracebacks_later() to dump_traceback_later() and cancel_dump_tracebacks_later() to cancel_dump_traceback_later(). @@ -83,35 +83,37 @@ ----------------- - Issue #13992: The trashcan mechanism is now thread-safe. This eliminates - sporadic crashes in multi-thread programs when several long deallocator - chains ran concurrently and involved subclasses of built-in container - types. - -- Issue #15784: Modify OSError.__str__() to better distinguish between - errno error numbers and Windows error numbers. + sporadic crashes in multi-thread programs when several long deallocator chains + ran concurrently and involved subclasses of built-in container types. + +- Issue #15784: Modify `OSError`.__str__() to better distinguish between errno + error numbers and Windows error numbers. - Issue #15781: Fix two small race conditions in import's module locking. Library ------- -- Issue #15847: Fix a regression in argparse, which did not accept tuples - as argument lists anymore. - -- Issue #15828: Restore support for C extensions in imp.load_module(). - -- Issue #15340: Fix importing the random module when /dev/urandom cannot - be opened. This was a regression caused by the hash randomization patch. - -- Issue #10650: Deprecate the watchexp parameter of the Decimal.quantize() +- Issue #17158: Add 'symbols' to help() welcome message; clarify + 'modules spam' messages. + +- Issue #15847: Fix a regression in argparse, which did not accept tuples as + argument lists anymore. + +- Issue #15828: Restore support for C extensions in `imp.load_module()`. + +- Issue #15340: Fix importing the random module when ``/dev/urandom`` cannot be + opened. This was a regression caused by the hash randomization patch. + +- Issue #10650: Deprecate the watchexp parameter of the `Decimal.quantize()` method. -- Issue #15785: Modify window.get_wch() API of the curses module: return - a character for most keys, and an integer for special keys, instead of - always returning an integer. So it is now possible to distinguish special - keys like keypad keys. - -- Issue #14223: Fix window.addch() of the curses module for special characters +- Issue #15785: Modify `window.get_wch()` API of the curses module: return a + character for most keys, and an integer for special keys, instead of always + returning an integer. So it is now possible to distinguish special keys like + keypad keys. + +- Issue #14223: Fix `window.addch()` of the curses module for special characters like curses.ACS_HLINE: the Python function addch(int) and addch(bytes) is now calling the C function waddch()/mvwaddch() (as it was done in Python 3.2), instead of wadd_wch()/mvwadd_wch(). The Python function addch(str) is still @@ -127,10 +129,10 @@ Documentation ------------- -- Issue #15814: The memoryview enhancements in 3.3.0 accidentally permitted - the hashing of multi-dimensional memorviews and memoryviews with multi-byte - item formats. The intended restrictions have now been documented - they - will be correctly enforced in 3.3.1 +- Issue #15814: The memoryview enhancements in 3.3.0 accidentally permitted the + hashing of multi-dimensional memorviews and memoryviews with multi-byte item + formats. The intended restrictions have now been documented - they will be + correctly enforced in 3.3.1. What's New in Python 3.3.0 Release Candidate 1? @@ -144,131 +146,123 @@ - Issue #15573: memoryview comparisons are now performed by value with full support for any valid struct module format definition. -- Issue #15316: When an item in the fromlist for __import__ doesn't exist, +- Issue #15316: When an item in the fromlist for `__import__()` doesn't exist, don't raise an error, but if an exception is raised as part of an import do let that propagate. -- Issue #15778: ensure that str(ImportError(msg)) returns a str - even when msg isn't a str. - -- Issue #2051: Source file permission bits are once again correctly - copied to the cached bytecode file. (The migration to importlib - reintroduced this problem because these was no regression test. A test - has been added as part of this patch) - -- Issue #15761: Fix crash when PYTHONEXECUTABLE is set on Mac OS X. - -- Issue #15726: Fix incorrect bounds checking in PyState_FindModule. - Patch by Robin Schreiber. - -- Issue #15604: Update uses of PyObject_IsTrue() to check for and handle +- Issue #15778: Ensure that ``str(ImportError(msg))`` returns a str even when + msg isn't a str. + +- Issue #2051: Source file permission bits are once again correctly copied to + the cached bytecode file. (The migration to importlib reintroduced this + problem because these was no regression test. A test has been added as part of + this patch) + +- Issue #15761: Fix crash when ``PYTHONEXECUTABLE`` is set on Mac OS X. + +- Issue #15726: Fix incorrect bounds checking in PyState_FindModule. Patch by + Robin Schreiber. + +- Issue #15604: Update uses of `PyObject_IsTrue()` to check for and handle errors correctly. Patch by Serhiy Storchaka. -- Issue #14846: importlib.FileFinder now handles the case where the - directory being searched is removed after a previous import attempt - -Library -------- - -- Issue #13370: Ensure that ctypes works on Mac OS X when Python is - compiled using the clang compiler - -- Issue #13072: The array module's 'u' format code is now deprecated and - will be removed in Python 4.0. +- Issue #14846: `importlib.FileFinder` now handles the case where the directory + being searched is removed after a previous import attempt. + +Library +------- + +- Issue #13370: Ensure that ctypes works on Mac OS X when Python is compiled + using the clang compiler. + +- Issue #13072: The array module's 'u' format code is now deprecated and will be + removed in Python 4.0. - Issue #15544: Fix Decimal.__float__ to work with payload-carrying NaNs. -- Issue #15249: BytesGenerator now correctly mangles From lines (when +- Issue #15776: Allow pyvenv to work in existing directory with --clean. + +- Issue #15249: email's BytesGenerator now correctly mangles From lines (when requested) even if the body contains undecodable bytes. - Issue #15777: Fix a refleak in _posixsubprocess. -- Issue ##665194: Update email.utils.localtime to use datetime.astimezone and +- Issue ##665194: Update `email.utils.localtime` to use datetime.astimezone and correctly handle historic changes in UTC offsets. - Issue #15199: Fix JavaScript's default MIME type to application/javascript. Patch by Bohuslav Kabrda. -- Issue #12643: code.InteractiveConsole now respects sys.excepthook when - displaying exceptions (Patch by Aaron Iles) - -- Issue #13579: string.Formatter now understands the 'a' conversion specifier. - -- Issue #15793: Stack corruption in ssl.RAND_egd(). - Patch by Serhiy Storchaka. - -- Issue #15595: Fix subprocess.Popen(universal_newlines=True) - for certain locales (utf-16 and utf-32 family). Patch by Chris Jerdonek. +- Issue #12643: `code.InteractiveConsole` now respects `sys.excepthook` when + displaying exceptions. Patch by Aaron Iles. + +- Issue #13579: `string.Formatter` now understands the 'a' conversion specifier. + +- Issue #15595: Fix ``subprocess.Popen(universal_newlines=True)`` for certain + locales (utf-16 and utf-32 family). Patch by Chris Jerdonek. - Issue #15477: In cmath and math modules, add workaround for platforms whose system-supplied log1p function doesn't respect signs of zeros. -- Issue #15715: importlib.__import__() will silence an ImportError when the use - of fromlist leads to a failed import. - -- Issue #14669: Fix pickling of connections and sockets on MacOSX - by sending/receiving an acknowledgment after file descriptor transfer. - TestPicklingConnection has been reenabled for MacOSX. +- Issue #15715: `importlib.__import__()` will silence an ImportError when the + use of fromlist leads to a failed import. + +- Issue #14669: Fix pickling of connections and sockets on Mac OS X by + sending/receiving an acknowledgment after file descriptor transfer. + TestPicklingConnection has been reenabled for Mac OS X. - Issue #11062: Fix adding a message from file to Babyl mailbox. -- Issue #15646: Prevent equivalent of a fork bomb when using - multiprocessing on Windows without the "if __name__ == '__main__'" - idiom. - -- Issue #15678: Fix IDLE menus when started from OS X command line - (3.3.0b2 regression). - -C API ------ - -Extension Modules ------------------ - -Tools/Demos ------------ +- Issue #15646: Prevent equivalent of a fork bomb when using `multiprocessing` + on Windows without the ``if __name__ == '__main__'`` idiom. + +IDLE +---- + +- Issue #15678: Fix IDLE menus when started from OS X command line (3.3.0b2 + regression). Documentation ------------- -- Issue #14674: Add a discussion of the json module's standard compliance. +- Touched up the Python 2 to 3 porting guide. + +- Issue #14674: Add a discussion of the `json` module's standard compliance. Patch by Chris Rebert. - Create a 'Concurrent Execution' section in the docs, and split up the 'Optional Operating System Services' section to use a more user-centric - classification scheme (splitting them across the new CE section, IPC and - text processing). Operating system limitatons can be reflected with - the Sphinx ``:platform:`` tag, it doesn't make sense as part of the Table of - Contents. - -- Issue #4966: Bring the sequence docs up to date for the Py3k transition - and the many language enhancements since they were original written + classification scheme (splitting them across the new CE section, IPC and text + processing). Operating system limitations can be reflected with the Sphinx + ``:platform:`` tag, it doesn't make sense as part of the Table of Contents. + +- Issue #4966: Bring the sequence docs up to date for the Py3k transition and + the many language enhancements since they were original written. - The "path importer" misnomer has been replaced with Eric Snow's - more-awkward-but-at-least-not-wrong suggestion of "path based finder" in - the import system reference docs - -- Issue #15640: Document importlib.abc.Finder as deprecated. - -- Issue #15630: Add an example for "continue" stmt in the tutorial. Patch by + more-awkward-but-at-least-not-wrong suggestion of "path based finder" in the + import system reference docs. + +- Issue #15640: Document `importlib.abc.Finder` as deprecated. + +- Issue #15630: Add an example for "continue" stmt in the tutorial. Patch by Daniel Ellis. Tests ----- - Issue #15747: ZFS always returns EOPNOTSUPP when attempting to set the - UF_IMMUTABLE flag (via either chflags or lchflags); refactor affected - tests in test_posix.py to account for this. - -- Issue #15285: Refactor the approach for testing connect timeouts using - two external hosts that have been configured specifically for this type - of test. - -- Issue #15743: Remove the deprecated method usage in urllib tests. Patch by + UF_IMMUTABLE flag (via either chflags or lchflags); refactor affected tests in + test_posix.py to account for this. + +- Issue #15285: Refactor the approach for testing connect timeouts using two + external hosts that have been configured specifically for this type of test. + +- Issue #15743: Remove the deprecated method usage in `urllib` tests. Patch by Jeff Knupp. -- Issue #15615: Add some tests for the json module's handling of invalid - input data. Patch by Kushal Das. +- Issue #15615: Add some tests for the `json` module's handling of invalid input + data. Patch by Kushal Das. Build ----- @@ -277,11 +271,11 @@ - Pick up 32-bit launcher from PGO directory on 64-bit PGO build. -- Drop PC\python_nt.h as it's not used. Add input dependency on custom +- Drop ``PC\python_nt.h`` as it's not used. Add input dependency on custom build step. -- Issue #15511: Drop explicit dependency on pythonxy.lib from _decimal - amd64 configuration. +- Issue #15511: Drop explicit dependency on pythonxy.lib from _decimal amd64 + configuration. - Add missing PGI/PGO configurations for pywlauncher. @@ -296,15 +290,15 @@ Core and Builtins ----------------- -- Issue #15568: Fix the return value of "yield from" when StopIteration is +- Issue #15568: Fix the return value of ``yield from`` when StopIteration is raised by a custom iterator. -- Issue #13119: sys.stdout and sys.stderr are now using "\r\n" newline on +- Issue #13119: `sys.stdout` and `sys.stderr` are now using "\r\n" newline on Windows, as Python 2. - Issue #15534: Fix the fast-search function for non-ASCII Unicode strings. -- Issue #15508: Fix the docstring for __import__ to have the proper default +- Issue #15508: Fix the docstring for `__import__()` to have the proper default value of 0 for 'level' and to not mention negative levels since they are not supported. @@ -317,17 +311,17 @@ byte code files) equal between 32-bit and 64-bit systems. - Issue #1692335: Move initial exception args assignment to - "BaseException.__new__" to help pickling of naive subclasses. - -- Issue #12834: Fix PyBuffer_ToContiguous() for non-contiguous arrays. - -- Issue #15456: Fix code __sizeof__ after #12399 change. Patch by Serhiy + `BaseException.__new__()` to help pickling of naive subclasses. + +- Issue #12834: Fix `PyBuffer_ToContiguous()` for non-contiguous arrays. + +- Issue #15456: Fix code `__sizeof__()` after #12399 change. Patch by Serhiy Storchaka. - Issue #15404: Refleak in PyMethodObject repr. -- Issue #15394: An issue in PyModule_Create that caused references to be leaked - on some error paths has been fixed. Patch by Julia Lawall. +- Issue #15394: An issue in `PyModule_Create()` that caused references to be + leaked on some error paths has been fixed. Patch by Julia Lawall. - Issue #15368: An issue that caused bytecode generation to be non-deterministic has been fixed. @@ -335,7 +329,7 @@ - Issue #15202: Consistently use the name "follow_symlinks" for new parameters in os and shutil functions. -- Issue #15314: __main__.__loader__ is now set correctly during interpreter +- Issue #15314: ``__main__.__loader__`` is now set correctly during interpreter startup. - Issue #15111: When a module imported using 'from import' has an ImportError @@ -350,57 +344,62 @@ - Issue #15110: Fix the tracebacks generated by "import xxx" to not show the importlib stack frames. +- Issue #16369: Global PyTypeObjects not initialized with PyType_Ready(...). + - Issue #15020: The program name used to search for Python's path is now "python3" under Unix, not "python". -- Issue #15033: Fix the exit status bug when modules invoked using -m swith, +- Issue #15897: zipimport.c doesn't check return value of fseek(). + Patch by Felipe Cruz. + +- Issue #15033: Fix the exit status bug when modules invoked using -m switch, return the proper failure return value (1). Patch contributed by Jeff Knupp. -- Issue #15229: An OSError subclass whose __init__ doesn't call back +- Issue #15229: An `OSError` subclass whose __init__ doesn't call back OSError.__init__ could produce incomplete instances, leading to crashes when calling str() on them. -- Issue 15307: Virtual environments now use symlinks with framework builds on +- Issue #15307: Virtual environments now use symlinks with framework builds on Mac OS X, like other POSIX builds. Library ------- -- Issue #15424: Add a __sizeof__ implementation for array objects. Patch by +- Issue #14590: configparser now correctly strips inline comments when delimiter + occurs earlier without preceding space. + +- Issue #15424: Add a `__sizeof__()` implementation for array objects. Patch by Ludwig H?hne. - Issue #15576: Allow extension modules to act as a package's __init__ module. -- Issue #15502: Have importlib.invalidate_caches() work on sys.meta_path instead - of sys.path_importer_cache. +- Issue #15502: Have `importlib.invalidate_caches()` work on `sys.meta_path` + instead of `sys.path_importer_cache`. - Issue #15163: Pydoc shouldn't list __loader__ as module data. - Issue #15471: Do not use mutable objects as defaults for - importlib.__import__(). + `importlib.__import__()`. - Issue #15559: To avoid a problematic failure mode when passed to the bytes - constructor, objects in the ipaddress module no longer implement __index__ - (they still implement __int__ as appropriate) + constructor, objects in the ipaddress module no longer implement `__index__()` + (they still implement `__int__()` as appropriate). - Issue #15546: Fix handling of pathological input data in the peek() and read1() methods of the BZ2File, GzipFile and LZMAFile classes. -- Issue #13052: Fix IDLE crashing when replace string in Search/Replace dialog - ended with '\'. Patch by Roger Serwy. - -- Issue #12655: Instead of requiring a custom type, os.sched_getaffinity and - os.sched_setaffinity now use regular sets of integers to represent the CPUs a - process is restricted to. - -- Issue #15538: Fix compilation of the getnameinfo() / getaddrinfo() emulation - code. Patch by Philipp Hagemeister. +- Issue #12655: Instead of requiring a custom type, `os.sched_getaffinity()` and + `os.sched_setaffinity()` now use regular sets of integers to represent the + CPUs a process is restricted to. + +- Issue #15538: Fix compilation of the `socket.getnameinfo()` / + `socket.getaddrinfo()` emulation code. Patch by Philipp Hagemeister. - Issue #15519: Properly expose WindowsRegistryFinder in importlib (and use the - correct term for it). Original patch by Eric Snow. + correct term for it). Original patch by Eric Snow. - Issue #15502: Bring the importlib ABCs into line with the current state of the - import protocols given PEP 420. Original patch by Eric Snow. + import protocols given PEP 420. Original patch by Eric Snow. - Issue #15499: Launching a webbrowser in Unix used to sleep for a few seconds. Original patch by Anton Barkovsky. @@ -408,37 +407,36 @@ - Issue #15463: The faulthandler module truncates strings to 500 characters, instead of 100, to be able to display long file paths. -- Issue #6056: Make multiprocessing use setblocking(True) on the sockets it +- Issue #6056: Make `multiprocessing` use setblocking(True) on the sockets it uses. Original patch by J Derek Wilson. - Issue #15364: Fix sysconfig.get_config_var('srcdir') to be an absolute path. -- Issue #15041: Update "see also" list in tkinter documentation. - -- Issue #15413: os.times() had disappeared under Windows. - -- Issue #15402: An issue in the struct module that caused sys.getsizeof to +- Issue #15413: `os.times()` had disappeared under Windows. + +- Issue #15402: An issue in the struct module that caused `sys.getsizeof()` to return incorrect results for struct.Struct instances has been fixed. Initial patch by Serhiy Storchaka. -- Issue #15232: When mangle_from is True, email.Generator now correctly mangles - lines that start with 'From ' that occur in a MIME preamble or epilogue. +- Issue #15232: When mangle_from is True, `email.Generator` now correctly + mangles lines that start with 'From ' that occur in a MIME preamble or + epilogue. - Issue #15094: Incorrectly placed #endif in _tkinter.c. Patch by Serhiy Storchaka. -- Issue #13922: argparse no longer incorrectly strips '--'s that appear after +- Issue #13922: `argparse` no longer incorrectly strips '--'s that appear after the first one. -- Issue #12353: argparse now correctly handles null argument values. +- Issue #12353: `argparse` now correctly handles null argument values. - Issue #10017, issue #14998: Fix TypeError using pprint on dictionaries with user-defined types as keys or other unorderable keys. -- Issue #15397: inspect.getmodulename() is now based directly on importlib via a - new importlib.machinery.all_suffixes() API. - -- Issue #14635: telnetlib will use poll() rather than select() when possible to +- Issue #15397: `inspect.getmodulename()` is now based directly on importlib via + a new `importlib.machinery.all_suffixes()` API. + +- Issue #14635: `telnetlib` will use poll() rather than select() when possible to avoid failing due to the select() file descriptor limit. - Issue #15180: Clarify posixpath.join() error message when mixing str & bytes. @@ -455,7 +453,7 @@ - Issue #15233: Python now guarantees that callables registered with the atexit module will be called in a deterministic order. -- Issue #15238: shutil.copystat now copies Linux "extended attributes". +- Issue #15238: `shutil.copystat()` now copies Linux "extended attributes". - Issue #15230: runpy.run_path now correctly sets __package__ as described in the documentation. @@ -465,42 +463,42 @@ - Issue #15294: Fix a regression in pkgutil.extend_path()'s handling of nested namespace packages. -- Issue #15056: imp.cache_from_source() and source_from_cache() raise - NotImplementedError when sys.implementation.cache_tag is set to None. - -- Issue #15256: Grammatical mistake in exception raised by imp.find_module(). - -- Issue #5931: wsgiref environ variable SERVER_SOFTWARE will specify an +- Issue #15056: `imp.cache_from_source()` and `imp.source_from_cache()` raise + NotImplementedError when `sys.implementation.cache_tag` is set to None. + +- Issue #15256: Grammatical mistake in exception raised by `imp.find_module()`. + +- Issue #5931: `wsgiref` environ variable SERVER_SOFTWARE will specify an implementation specific term like CPython, Jython instead of generic "Python". - Issue #13248: Remove obsolete argument "max_buffer_size" of BufferedWriter and BufferedRWPair, from the io module. -- Issue #13248: Remove obsolete argument "version" of argparse.ArgumentParser. +- Issue #13248: Remove obsolete argument "version" of `argparse.ArgumentParser`. - Issue #14814: Implement more consistent ordering and sorting behaviour for ipaddress objects. -- Issue #14814: ipaddress network objects correctly return NotImplemented when +- Issue #14814: `ipaddress` network objects correctly return NotImplemented when compared to arbitrary objects instead of raising TypeError. - Issue #14990: Correctly fail with SyntaxError on invalid encoding declaration. -- Issue #14814: ipaddress now provides more informative error messages when +- Issue #14814: `ipaddress` now provides more informative error messages when constructing instances directly (changes permitted during beta due to provisional API status). -- Issue #15247: FileIO now raises an error when given a file descriptor pointing - to a directory. +- Issue #15247: `io.FileIO` now raises an error when given a file descriptor + pointing to a directory. - Issue #15261: Stop os.stat(fd) crashing on Windows when fd not open. -- Issue #15166: Implement imp.get_tag() using sys.implementation.cache_tag. - -- Issue #15210: Catch KeyError when importlib.__init__ can't find +- Issue #15166: Implement `imp.get_tag()` using `sys.implementation.cache_tag`. + +- Issue #15210: Catch KeyError when `importlib.__init__()` can't find _frozen_importlib in sys.modules, not ImportError. -- Issue #15030: importlib.abc.PyPycLoader now supports the new source size +- Issue #15030: `importlib.abc.PyPycLoader` now supports the new source size header field in .pyc files. - Issue #5346: Preserve permissions of mbox, MMDF and Babyl mailbox files on @@ -513,7 +511,7 @@ renamed over the old file when flush() is called on an mbox, MMDF or Babyl mailbox. -- Issue 10924: Fixed crypt.mksalt() to use a RNG that is suitable for +- Issue #10924: Fixed `crypt.mksalt()` to use a RNG that is suitable for cryptographic purpose. - Issue #15184: Ensure consistent results of OS X configuration tailoring for @@ -524,10 +522,10 @@ C API ----- -- Issue #15610: PyImport_ImportModuleEx() now uses a 'level' of 0 instead of -1. - -- Issues #15169, #14599: Strip out the C implementation of - imp.source_from_cache() used by PyImport_ExecCodeModuleWithPathnames() and +- Issue #15610: `PyImport_ImportModuleEx()` now uses a 'level' of 0 instead of -1. + +- Issue #15169, issue #14599: Strip out the C implementation of + `imp.source_from_cache()` used by PyImport_ExecCodeModuleWithPathnames() and used the Python code instead. Leads to PyImport_ExecCodeModuleObject() to not try to infer the source path from the bytecode path as PyImport_ExecCodeModuleWithPathnames() does. @@ -535,14 +533,17 @@ Extension Modules ----------------- -- Issue #15676: Now "mmap" check for empty files before doing the - offset check. Patch by Steven Willis. - -- Issue #6493: An issue in ctypes on Windows that caused structure bitfields - of type ctypes.c_uint32 and width 32 to incorrectly be set has been fixed. +- Issue #6493: An issue in ctypes on Windows that caused structure bitfields of + type `ctypes.c_uint32` and width 32 to incorrectly be set has been fixed. - Issue #15194: Update libffi to the 3.0.11 release. +IDLE +---- + +- Issue #13052: Fix IDLE crashing when replace string in Search/Replace dialog + ended with ``\``. Patch by Roger Serwy. + Tools/Demos ----------- @@ -562,8 +563,10 @@ Documentation ------------- -- Issue #15444: Use proper spelling for non-ASCII contributor names. Patch - by Serhiy Storchaka. +- Issue #15041: Update "see also" list in tkinter documentation. + +- Issue #15444: Use proper spelling for non-ASCII contributor names. Patch by + Serhiy Storchaka. - Issue #15295: Reorganize and rewrite the documentation on the import system. @@ -578,25 +581,29 @@ "changed" since they will no longer work with modules directly imported by import itself. -- Issue #13557: Clarify effect of giving two different namespaces to exec or - execfile(). - -- Issue #15250: Document that filecmp.dircmp compares files shallowly. Patch +- Issue #13557: Clarify effect of giving two different namespaces to `exec()` or + `execfile()`. + +- Issue #15250: Document that `filecmp.dircmp()` compares files shallowly. Patch contributed by Chris Jerdonek. -Tests ------ - -- Issue #15467: Move helpers for __sizeof__ tests into test_support. Patch by - Serhiy Storchaka. +- Issue #15442: Expose the default list of directories ignored by + `filecmp.dircmp()` as a module attribute, and expand the list to more modern + values. + +Tests +----- + +- Issue #15467: Move helpers for `__sizeof__()` tests into test_support. Patch + by Serhiy Storchaka. - Issue #15320: Make iterating the list of tests thread-safe when running tests in multiprocess mode. Patch by Chris Jerdonek. -- Issue #15168: Move importlib.test to test.test_importlib. +- Issue #15168: Move `importlib.test` to `test.test_importlib`. - Issue #15091: Reactivate a test on UNIX which was failing thanks to a - forgotten importlib.invalidate_caches() call. + forgotten `importlib.invalidate_caches()` call. - Issue #15230: Adopted a more systematic approach in the runpy tests. @@ -629,6 +636,8 @@ - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. +- Issue #16256: OS X installer now sets correct permissions for doc directory. + - Issue #15431: Add _freeze_importlib project to regenerate importlib.h on Windows. Patch by Kristj?n Valur J?nsson. @@ -664,14 +673,9 @@ - Issue #11626: Add _SizeT functions to stable ABI. -- Issue #15146: Add PyType_FromSpecWithBases. Patch by Robin Schreiber. - - Issue #15142: Fix reference leak when deallocating instances of types created using PyType_FromSpec(). -- Issue #15042: Add PyState_AddModule and PyState_RemoveModule. Add version - guard for Py_LIMITED_API additions. Patch by Robin Schreiber. - - Issue #10053: Don't close FDs when FileIO.__init__ fails. Loosely based on the work by Hirokazu Yamamoto. @@ -699,9 +703,6 @@ Library ------- -- Issue #9803: Don't close IDLE on saving if breakpoint is open. - Patch by Roger Serwy. - - Issue #12288: Consider '0' and '0.0' as valid initialvalue for tkinter SimpleDialog. @@ -720,14 +721,8 @@ - Issue #15514: Correct __sizeof__ support for cpu_set. Patch by Serhiy Storchaka. -- Issue #15187: Bugfix: remove temporary directories test_shutil was leaving - behind. - - Issue #15177: Added dir_fd parameter to os.fwalk(). -- Issue #15176: Clarified behavior, documentation, and implementation - of os.listdir(). - - Issue #15061: Re-implemented hmac.compare_digest() in C to prevent further timing analysis and to support all buffer protocol aware objects as well as ASCII only str instances safely. @@ -827,10 +822,6 @@ - Issue #15006: Allow equality comparison between naive and aware time or datetime objects. -- Issue #14982: Document that pkgutil's iteration functions require the - non-standard iter_modules() method to be defined by an importer (something - the importlib importers do not define). - - Issue #15036: Mailbox no longer throws an error if a flush is done between operations when removing or changing multiple items in mbox, MMDF, or Babyl mailboxes. @@ -898,32 +889,50 @@ - Issue #14969: Better handling of exception chaining in contextlib.ExitStack +- Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative + algorithm (Patch by Alon Horev) + +- Issue #14785: Add sys._debugmallocstats() to help debug low-level memory + allocation issues + +- Issue #14443: Ensure that .py files are byte-compiled with the correct Python + executable within bdist_rpm even on older versions of RPM + +C-API +----- + +- Issue #15146: Add PyType_FromSpecWithBases. Patch by Robin Schreiber. + +- Issue #15042: Add PyState_AddModule and PyState_RemoveModule. Add version + guard for Py_LIMITED_API additions. Patch by Robin Schreiber. + +- Issue #13783: Inadvertent additions to the public C API in the PEP 380 + implementation have either been removed or marked as private interfaces. + +Extension Modules +----------------- + +- Issue #15000: Support the "unique" x32 architecture in _posixsubprocess.c. + +IDLE +---- + +- Issue #9803: Don't close IDLE on saving if breakpoint is open. + Patch by Roger Serwy. + - Issue #14962: Update text coloring in IDLE shell window after changing options. Patch by Roger Serwy. -- Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative - algorithm (Patch by Alon Horev) - -- Issue #14785: Add sys._debugmallocstats() to help debug low-level memory - allocation issues - -- Issue #14443: Ensure that .py files are byte-compiled with the correct Python - executable within bdist_rpm even on older versions of RPM - -C-API ------ - -- Issue #13783: Inadvertent additions to the public C API in the PEP 380 - implementation have either been removed or marked as private interfaces. - -Extension Modules ------------------ - -- Issue #15000: Support the "unique" x32 architecture in _posixsubprocess.c. - Documentation ------------- +- Issue #15176: Clarified behavior, documentation, and implementation + of os.listdir(). + +- Issue #14982: Document that pkgutil's iteration functions require the + non-standard iter_modules() method to be defined by an importer (something + the importlib importers do not define). + - Issue #15081: Document PyState_FindModule. Patch by Robin Schreiber. @@ -932,6 +941,9 @@ Tests ----- +- Issue #15187: Bugfix: remove temporary directories test_shutil was leaving + behind. + - Issue #14769: test_capi now has SkipitemTest, which cleverly checks for "parity" between PyArg_ParseTuple() and the Python/getargs.c static function skipitem() for all possible "format units". @@ -1020,34 +1032,18 @@ - Issue #14700: Fix two broken and undefined-behaviour-inducing overflow checks in old-style string formatting. -- Issue #14705: The PyArg_Parse() family of functions now support the 'p' format - unit, which accepts a "boolean predicate" argument. It converts any Python - value into an integer--0 if it is "false", and 1 otherwise. - Library ------- - Issue #14690: Use monotonic clock instead of system clock in the sched, subprocess and trace modules. -- Issue #14958: Change IDLE systax highlighting to recognize all string and - byte literals supported in Python 3.3. - -- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. - - Issue #14443: Tell rpmbuild to use the correct version of Python in bdist_rpm. Initial patch by Ross Lagerwall. -- Issue #14929: Stop Idle 3.x from closing on Unicode decode errors when - grepping. Patch by Roger Serwy. - - Issue #12515: email now registers a defect if it gets to EOF while parsing a MIME part without seeing the closing MIME boundary. -- Issue #12510: Attempting to get invalid tooltip no longer closes Idle. - Other tooltipss have been corrected or improved and the number of tests - has been tripled. Original patch by Roger Serwy. - - Issue #1672568: email now always decodes base64 payloads, adding padding and ignoring non-base64-alphabet characters if needed, and registering defects for any such problems. @@ -1081,9 +1077,6 @@ - Issue #14548: Make multiprocessing finalizers check pid before running to cope with possibility of gc running just after fork. -- Issue #14863: Update the documentation of os.fdopen() to reflect the - fact that it's only a thin wrapper around open() anymore. - - Issue #14036: Add an additional check to validate that port in urlparse does not go in illegal range and returns None. @@ -1210,6 +1203,21 @@ - Issue #14127 and #10148: shutil.copystat now preserves exact mtime and atime on filesystems providing nanosecond resolution. +IDLE +---- + +- Issue #14958: Change IDLE systax highlighting to recognize all string and + byte literals supported in Python 3.3. + +- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. + +- Issue #14929: Stop IDLE 3.x from closing on Unicode decode errors when + grepping. Patch by Roger Serwy. + +- Issue #12510: Attempting to get invalid tooltip no longer closes IDLE. + Other tooltipss have been corrected or improved and the number of tests + has been tripled. Original patch by Roger Serwy. + Tools/Demos ----------- @@ -1228,9 +1236,19 @@ - Issue #13210: Windows build now uses VS2010, ported from VS2008. +C-API +----- + +- Issue #14705: The PyArg_Parse() family of functions now support the 'p' format + unit, which accepts a "boolean predicate" argument. It converts any Python + value into an integer--0 if it is "false", and 1 otherwise. + Documentation ------------- +- Issue #14863: Update the documentation of os.fdopen() to reflect the + fact that it's only a thin wrapper around open() anymore. + - Issue #14588: The language reference now accurately documents the Python 3 class definition process. Patch by Nick Coghlan. @@ -1279,9 +1297,6 @@ - Issue #14339: Speed improvements to bin, oct and hex functions. Patch by Serhiy Storchaka. -- Issue #14098: New functions PyErr_GetExcInfo and PyErr_SetExcInfo. - Patch by Stefan Behnel. - - Issue #14385: It is now possible to use a custom type for the __builtins__ namespace, instead of a dict. It can be used for sandboxing for example. Raise also a NameError instead of ImportError if __build_class__ name if not @@ -1431,12 +1446,6 @@ - Don't Py_DECREF NULL variable in io.IncrementalNewlineDecoder. -- Issue #8515: Set __file__ when run file in IDLE. - Initial patch by Bruce Frederiksen. - -- Issue #14496: Fix wrong name in idlelib/tabbedpages.py. - Patch by Popa Claudiu. - - Issue #3033: Add displayof parameter to tkinter font. Patch by Guilherme Polo. - Issue #14482: Raise a ValueError, not a NameError, when trying to create @@ -1472,6 +1481,15 @@ - Issue #14355: Regrtest now supports the standard unittest test loading, and will use it if a test file contains no `test_main` method. +IDLE +---- + +- Issue #8515: Set __file__ when run file in IDLE. + Initial patch by Bruce Frederiksen. + +- Issue #14496: Fix wrong name in idlelib/tabbedpages.py. + Patch by Popa Claudiu. + Tools / Demos ------------- @@ -1481,6 +1499,12 @@ - Issue #13165: stringbench is now available in the Tools/stringbench folder. It used to live in its own SVN project. +C-API +----- + +- Issue #14098: New functions PyErr_GetExcInfo and PyErr_SetExcInfo. + Patch by Stefan Behnel. + What's New in Python 3.3.0 Alpha 2? =================================== @@ -1532,16 +1556,9 @@ - Issue #5136: deprecate old, unused functions from tkinter. -- Issue #14409: IDLE now properly executes commands in the Shell window - when it cannot read the normal config files on startup and - has to use the built-in default key bindings. - There was previously a bug in one of the defaults. - - Issue #14416: syslog now defines the LOG_ODELAY and LOG_AUTHPRIV constants if they are defined in . -- IDLE can be launched as python -m idlelib - - Issue #14295: Add unittest.mock - Issue #7652: Add --with-system-libmpdec option to configure for linking @@ -1567,9 +1584,6 @@ up the decimal module. Performance gains of the new C implementation are between 10x and 100x, depending on the application. -- Issue #3573: IDLE hangs when passing invalid command line args - (directory(ies) instead of file(s)) (Patch by Guilherme Polo) - - Issue #14269: SMTPD now conforms to the RFC and requires a HELO command before MAIL, RCPT, or DATA. @@ -1601,8 +1615,6 @@ denial of service due to hash collisions. Patch by David Malcolm with some modifications by the expat project. -- Issue #14200: Idle shell crash on printing non-BMP unicode character. - - Issue #12818: format address no longer needlessly \ escapes ()s in names when the name ends up being quoted. @@ -1618,8 +1630,6 @@ - Issue #989712: Support using Tk without a mainloop. -- Issue #5219: Prevent event handler cascade in IDLE. - - Issue #3835: Refuse to use unthreaded Tcl in threaded Python. - Issue #2843: Add new Tk API to Tkinter. @@ -1848,10 +1858,6 @@ on POSIX systems supporting anonymous memory mappings. Patch by Charles-Fran?ois Natali. -- Issue #13452: PyUnicode_EncodeDecimal() doesn't support error handlers - different than "strict" anymore. The caller was unable to compute the - size of the output buffer: it depends on the error handler. - - PEP 3155 / issue #13448: Qualified name for classes and functions. - Issue #13436: Fix a bogus error message when an AST object was passed @@ -1942,10 +1948,6 @@ - PEP 3151 / issue #12555: reworking the OS and IO exception hierarchy. -- Issue #13560: Add PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() - and PyUnicode_EncodeLocale() functions to the C API to decode/encode from/to - the current locale encoding. - - Add internal API for static strings (_Py_identifier et al.). - Issue #13063: the Windows error ERROR_NO_DATA (numbered 232 and described @@ -2028,7 +2030,7 @@ deallocator calls one of the methods on the type (e.g. when subclassing IOBase). Diagnosis and patch by Davide Rizzo. -- Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on Windows. +- Issue #9611, Issue #9015: FileIO.read() clamps the length to INT_MAX on Windows. - Issue #9642: Uniformize the tests on the availability of the mbcs codec, add a new HAVE_MBCS define. @@ -2191,17 +2193,11 @@ PyUnicode_AsUTF8String() and PyUnicode_AsEncodedString(unicode, "utf-8", NULL). -- Issue #10831: PyUnicode_FromFormat() supports %li, %lli and %zi formats. - - Issue #10829: Refactor PyUnicode_FromFormat(), use the same function to parse the format string in the 3 steps, fix crashs on invalid format strings. - Issue #13007: whichdb should recognize gdbm 1.9 magic numbers. -- Issue #11246: Fix PyUnicode_FromFormat("%V") to decode the byte string from - UTF-8 (with replace error handler) instead of ISO-8859-1 (in strict mode). - Patch written by Ray Allen. - - Issue #11286: Raise a ValueError from calling PyMemoryView_FromBuffer with a buffer struct having a NULL data pointer. @@ -2211,9 +2207,6 @@ - Issue #11828: startswith and endswith now accept None as slice index. Patch by Torsten Becker. -- Issue #10830: Fix PyUnicode_FromFormatV("%c") for non-BMP characters on - narrow build. - - Issue #11168: Remove filename debug variable from PyEval_EvalFrameEx(). It encoded the Unicode filename to UTF-8, but the encoding fails on undecodable filename (on surrogate characters) which raises an unexpected @@ -2255,15 +2248,9 @@ are dead or dying. Moreover, the implementation is now O(1) rather than O(n). -- Issue #13125: Silence spurious test_lib2to3 output when in non-verbose mode. - Patch by Mikhail Novikov. - - Issue #11841: Fix comparison bug with 'rc' versions in packaging.version. Patch by Filip Gruszczy?ski. -- Issue #13447: Add a test file to host regression tests for bugs in the - scripts found in the Tools directory. - - Issue #6884: Fix long-standing bugs with MANIFEST.in parsing in distutils on Windows. Also fixed in packaging. @@ -2319,9 +2306,6 @@ authenticating (since the result may change, according to RFC 4643). Patch by Hynek Schlawack. -- Issue #13989: Document that GzipFile does not support text mode, and give a - more helpful error message when opened with an invalid mode string. - - Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building Distutils-based packages with C extension modules may fail because Apple has removed gcc-4.2, the version used to build python.org @@ -2338,10 +2322,6 @@ - Issue #13960: HTMLParser is now able to handle broken comments when strict=False. -- Issue #13921: Undocument and clean up sqlite3.OptimizedUnicode, - which is obsolete in Python 3.x. It's now aliased to str for - backwards compatibility. - - When '' is a path (e.g. in sys.path), make sure __file__ uses the current working directory instead of '' in importlib. @@ -2363,11 +2343,6 @@ - Issue #10811: Fix recursive usage of cursors. Instead of crashing, raise a ProgrammingError now. -- Issue #10881: Fix test_site failure with OS X framework builds. - -- Issue #964437: Make IDLE help window non-modal. - Patch by Guilherme Polo and Roger Serwy. - - Issue #13734: Add os.fwalk(), a directory walking function yielding file descriptors. @@ -2377,16 +2352,8 @@ - Issue #11805: package_data in setup.cfg should allow more than one value. -- Issue #13933: IDLE auto-complete did not work with some imported - module, like hashlib. (Patch by Roger Serwy) - -- Issue #13901: Prevent test_distutils failures on OS X with --enable-shared. - - Issue #13676: Handle strings with embedded zeros correctly in sqlite3. -- Issue #13506: Add '' to path for IDLE Shell when started and restarted with Restart Shell. - Original patches by Marco Scataglini and Roger Serwy. - - Issue #8828: Add new function os.replace(), for cross-platform renaming with overwriting. @@ -2407,12 +2374,6 @@ OSError if localtime() failed. time.clock() now raises a RuntimeError if the processor time used is not available or its value cannot be represented -- Issue #13862: Fix spurious failure in test_zlib due to runtime/compile time - minor versions not matching. - -- Issue #12804: Fix test_socket and test_urllib2net failures when running tests - on a system without internet access. - - Issue #13772: In os.symlink() under Windows, do not try to guess the link target's type (file or directory). The detection was buggy and made the call non-atomic (therefore prone to race conditions). @@ -2439,9 +2400,6 @@ - Issue #13642: Unquote before b64encoding user:password during Basic Authentication. Patch contributed by Joonas Kuorilehto. -- Issue #13726: Fix the ambiguous -S flag in regrtest. It is -o/--slow for slow - tests. - - Issue #12364: Fix a hang in concurrent.futures.ProcessPoolExecutor. The hang would occur when retrieving the result of a scheduled future after the executor had been shut down. @@ -2524,10 +2482,6 @@ - Issue #13591: A bug in importlib has been fixed that caused import_module to load a module twice. -- Issue #4625: If IDLE cannot write to its recent file or breakpoint files, - display a message popup and continue rather than crash. Original patch by - Roger Serwy. - - Issue #13449 sched.scheduler.run() method has a new "blocking" parameter which when set to False makes run() execute the scheduled events due to expire soonest (if any) and then return. Patch by Giampaolo Rodol?. @@ -2544,12 +2498,9 @@ 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', and private attributes of 'smtpd.SMTPChannel'. -- Issue #5905, #13560: time.strftime() is now using the current locale +- Issue #5905, Issue #13560: time.strftime() is now using the current locale encoding, instead of UTF-8, if the wcsftime() function is not available. -- Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". - Patch by Tal Einat. - - Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch by Jon Kuhn. @@ -2661,9 +2612,6 @@ - Issue #10817: Fix urlretrieve function to raise ContentTooShortError even when reporthook is None. Patch by Jyrki Pulliainen. -- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart. - (Patch by Roger Serwy) - - Fix the xmlrpc.client user agent to return something similar to urllib.request user agent: "Python-xmlrpc/3.3". @@ -2766,10 +2714,6 @@ - Issue #13034: When decoding some SSL certificates, the subjectAltName extension could be unreported. -- Issue #9871: Prevent IDLE 3 crash when given byte stings - with invalid hex escape sequences, like b'\x0'. - (Original patch by Claudiu Popa.) - - Issue #12306: Expose the runtime version of the zlib C library as a constant, ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff. @@ -2798,8 +2742,6 @@ - Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses. -- Issue #12636: IDLE reads the coding cookie when executing a Python script. - - Issue #12494: On error, call(), check_call(), check_output() and getstatusoutput() functions of the subprocess module now kill the process, read its status (to avoid zombis) and close pipes. @@ -2869,9 +2811,6 @@ - Issue #10087: Fix the html output format of the calendar module. -- Issue #12540: Prevent zombie IDLE processes on Windows due to changes - in os.kill(). - - Issue #13121: add support for inplace math operators to collections.Counter. - Add support for unary plus and unary minus to collections.Counter. @@ -2903,7 +2842,7 @@ Condition, etc.) used to be factory functions returning instances of hidden classes (_Event, _Condition, etc.), because (if Guido recalls correctly) this code pre-dates the ability to subclass extension types. It is now possible - to inherit from these classes without having to import the private + to inherit from these classes, without having to import the private underscored names like multiprocessing did. - Issue #9723: Add shlex.quote functions, to escape filenames and command @@ -2917,14 +2856,8 @@ - Issue #12607: In subprocess, fix issue where if stdin, stdout or stderr is given as a low fd, it gets overwritten. -- Issue #12590: IDLE editor window now always displays the first line - when opening a long file. With Tk 8.5, the first line was hidden. - - Issue #12576: Fix urlopen behavior on sites which do not send (or obfuscates) - Connection:close header. - -- Issue #12102: Document that buffered files must be flushed before being used - with mmap. Patch by Steffen Daode Nurpmeso. + ``Connection: close`` header. - Issue #12560: Build libpython.so on OpenBSD. Patch by Stefan Sperling. @@ -3179,7 +3112,7 @@ - Issue #12175: FileIO.readall() now raises a ValueError instead of an IOError if the file is closed. -- Issue #11109: New service_action method for BaseServer, used by ForkingMixIn +- Issue #11109: New service_action method for BaseServer, used by ForkingMixin class for cleanup. Initial Patch by Justin Warkentin. - Issue #12045: Avoid duplicate execution of command in @@ -3224,9 +3157,6 @@ passing a ``context`` argument pointing to an ssl.SSLContext instance. Patch by Kasun Herath. -- Issue #11088: don't crash when using F5 to run a script in IDLE on MacOSX - with Tk 8.5. - - Issue #9516: Issue #9516: avoid errors in sysconfig when MACOSX_DEPLOYMENT_TARGET is set in shell. @@ -3246,10 +3176,6 @@ - Issue #9971: Write an optimized implementation of BufferedReader.readinto(). Patch by John O'Connor. -- Issue #1028: Tk returns invalid Unicode null in %A: UnicodeDecodeError. - With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused - IDLE to exit. Converted to valid Unicode null in PythonCmd(). - - Issue #11799: urllib.request Authentication Handlers will raise a ValueError when presented with an unsupported Authentication Scheme. Patch contributed by Yuval Greenfield. @@ -3486,12 +3412,12 @@ - Issue #7639: Fix short file name generation in bdist_msi -- Issue #11659: Fix ResourceWarning in test_subprocess introduced by #11459. - Patch by Ben Hayden. - - Issue #11635: Don't use polling in worker threads and processes launched by concurrent.futures. +- Issue #5845: Automatically read readline configuration to enable completion + in interactive mode. + - Issue #6811: Allow importlib to change a code object's co_filename attribute to match the path to where the source code currently is, not where the code object originally came from. @@ -3526,7 +3452,7 @@ - Issue #11127: Raise a TypeError when trying to pickle a socket object. -- Issue #11563: Connection:close header is sent by requests using URLOpener +- Issue #11563: ``Connection: close`` header is sent by requests using URLOpener class which helps in closing of sockets after connection is over. Patch contributions by Jeff McNeil and Nadeem Vawda. @@ -3541,8 +3467,6 @@ - Issue #10979: unittest stdout buffering now works with class and module setup and teardown. -- Issue #11577: fix ResourceWarning triggered by improved binhex test coverage - - Issue #11243: fix the parameter querying methods of Message to work if the headers contain un-encoded non-ASCII data. @@ -3575,9 +3499,6 @@ - Issue #11554: Fixed support for Japanese codecs; previously the body output encoding was not done if euc-jp or shift-jis was specified as the charset. -- Issue #11509: Significantly increase test coverage of fileinput. - Patch by Denver Coneybeare at PyCon 2011 Sprints. - - Issue #11407: `TestCase.run` returns the result object used or created. Contributed by Janathan Hartley. @@ -3700,11 +3621,6 @@ - Issue #9348: Raise an early error if argparse nargs and metavar don't match. -- Issue #8982: Improve the documentation for the argparse Namespace object. - -- Issue #9343: Document that argparse parent parsers must be configured before - their children. - - Issue #9026: Fix order of argparse sub-commands in help messages. - Issue #9347: Fix formatting for tuples in argparse type= error messages. @@ -3757,10 +3673,61 @@ - Issue #11495: OSF support is eliminated. It was deprecated in Python 3.2. - IDLE ---- +- Issue #14409: IDLE now properly executes commands in the Shell window + when it cannot read the normal config files on startup and + has to use the built-in default key bindings. + There was previously a bug in one of the defaults. + +- IDLE can be launched as python -m idlelib + +- Issue #3573: IDLE hangs when passing invalid command line args + (directory(ies) instead of file(s)) (Patch by Guilherme Polo) + +- Issue #14200: IDLE shell crash on printing non-BMP unicode character. + +- Issue #5219: Prevent event handler cascade in IDLE. + +- Issue #964437: Make IDLE help window non-modal. + Patch by Guilherme Polo and Roger Serwy. + +- Issue #13933: IDLE auto-complete did not work with some imported + module, like hashlib. (Patch by Roger Serwy) + +- Issue #13506: Add '' to path for IDLE Shell when started and restarted with Restart Shell. + Original patches by Marco Scataglini and Roger Serwy. + +- Issue #4625: If IDLE cannot write to its recent file or breakpoint files, + display a message popup and continue rather than crash. Original patch by + Roger Serwy. + +- Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". + Patch by Tal Einat. + +- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart. + (Patch by Roger Serwy) + +- Issue #9871: Prevent IDLE 3 crash when given byte stings + with invalid hex escape sequences, like b'\x0'. + (Original patch by Claudiu Popa.) + +- Issue #12636: IDLE reads the coding cookie when executing a Python script. + +- Issue #12540: Prevent zombie IDLE processes on Windows due to changes + in os.kill(). + +- Issue #12590: IDLE editor window now always displays the first line + when opening a long file. With Tk 8.5, the first line was hidden. + +- Issue #11088: don't crash when using F5 to run a script in IDLE on MacOSX + with Tk 8.5. + +- Issue #1028: Tk returns invalid Unicode null in %A: UnicodeDecodeError. + With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused + IDLE to exit. Converted to valid Unicode null in PythonCmd(). + - Issue #11718: IDLE's open module dialog couldn't find the __init__.py file in a package. @@ -3790,6 +3757,10 @@ Extension Modules ----------------- +- Issue #16847: Fixed improper use of _PyUnicode_CheckConsistency() in + non-pydebug builds. Several extension modules now compile cleanly when + assert()s are enabled in standard builds (-DDEBUG flag). + - Issue #13840: The error message produced by ctypes.create_string_buffer when given a Unicode string has been fixed. @@ -3852,6 +3823,33 @@ Tests ----- +- Issue #13125: Silence spurious test_lib2to3 output when in non-verbose mode. + Patch by Mikhail Novikov. + +- Issue #13447: Add a test file to host regression tests for bugs in the + scripts found in the Tools directory. + +- Issue #10881: Fix test_site failure with OS X framework builds. + +- Issue #13901: Prevent test_distutils failures on OS X with --enable-shared. + +- Issue #13862: Fix spurious failure in test_zlib due to runtime/compile time + minor versions not matching. + +- Issue #12804: Fix test_socket and test_urllib2net failures when running tests + on a system without internet access. + +- Issue #13726: Fix the ambiguous -S flag in regrtest. It is -o/--slow for slow + tests. + +- Issue #11659: Fix ResourceWarning in test_subprocess introduced by #11459. + Patch by Ben Hayden. + +- Issue #11577: fix ResourceWarning triggered by improved binhex test coverage + +- Issue #11509: Significantly increase test coverage of fileinput. + Patch by Denver Coneybeare at PyCon 2011 Sprints. + - Issue #11689: Fix a variable scoping error in an sqlite3 test - Issue #13786: Remove unimplemented 'trace' long option from regrtest.py. @@ -4030,7 +4028,7 @@ - Issue #11505: improves test coverage of string.py. Patch by Alicia Arlen -- Issue #11490: test_subprocess:test_leaking_fds_on_error no longer gives a +- Issue #11490: test_subprocess.test_leaking_fds_on_error no longer gives a false positive if the last directory in the path is inaccessible. - Issue #11223: Fix test_threadsignals to fail, not hang, when the @@ -4054,6 +4052,23 @@ C-API ----- +- Issue #13452: PyUnicode_EncodeDecimal() doesn't support error handlers + different than "strict" anymore. The caller was unable to compute the + size of the output buffer: it depends on the error handler. + +- Issue #13560: Add PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() + and PyUnicode_EncodeLocale() functions to the C API to decode/encode from/to + the current locale encoding. + +- Issue #10831: PyUnicode_FromFormat() supports %li, %lli and %zi formats. + +- Issue #11246: Fix PyUnicode_FromFormat("%V") to decode the byte string from + UTF-8 (with replace error handler) instead of ISO-8859-1 (in strict mode). + Patch written by Ray Allen. + +- Issue #10830: Fix PyUnicode_FromFormatV("%c") for non-BMP characters on + narrow build. + - Add PyObject_GenericGetDict and PyObject_GeneriSetDict. They are generic implementations for the getter and setter of a ``__dict__`` descriptor of C types. @@ -4079,6 +4094,24 @@ Documentation ------------- +- Issue #13989: Document that GzipFile does not support text mode, and give a + more helpful error message when opened with an invalid mode string. + +- Issue #13921: Undocument and clean up sqlite3.OptimizedUnicode, + which is obsolete in Python 3.x. It's now aliased to str for + backwards compatibility. + +- Issue #12102: Document that buffered files must be flushed before being used + with mmap. Patch by Steffen Daode Nurpmeso. + +- Issue #8982: Improve the documentation for the argparse Namespace object. + +- Issue #9343: Document that argparse parent parsers must be configured before + their children. + +- Issue #13498: Clarify docs of os.makedirs()'s exist_ok argument. Done with + great native-speaker help from R. David Murray. + - Issues #13491 and #13995: Fix many errors in sqlite3 documentation. Initial patch for #13491 by Johannes Vogel. @@ -9652,7 +9685,7 @@ - Issue #1210: Fixed imaplib and its documentation. -- Issue #4233: Changed semantic of ``_fileio.FileIO``'s ``close()`` +- Issue #4233: Changed semantic of ``_fileio.FileIO``'s ``close()`` method on file objects with closefd=False. The file descriptor is still kept open but the file object behaves like a closed file. The ``FileIO`` object also got a new readonly attribute ``closefd``. @@ -9796,13 +9829,13 @@ cyclic garbage collection. - Issue #3668: Fix a memory leak with the "s*" argument parser in - PyArg_ParseTuple and friends, which occurred when the argument for "s*" + PyArg_ParseTuple and friends, which occurred when the argument for "s*" was correctly parsed but parsing of subsequent arguments failed. - Issue #3611: An exception __context__ could be cleared in a complex pattern involving a __del__ method re-raising an exception. -- Issue #2534: speed up isinstance() and issubclass() by 50-70%, so as to +- Issue #2534: speed up isinstance() and issubclass() by 50-70%, so as to match Python 2.5 speed despite the __instancecheck__ / __subclasscheck__ mechanism. In the process, fix a bug where isinstance() and issubclass(), when given a tuple of classes as second argument, were looking up @@ -9880,7 +9913,7 @@ - The deprecation warnings for the camelCase threading API names were removed. -- Issue #3110: multiprocessing fails to compiel on solaris 10 due to missing +- Issue #3110: multiprocessing fails to compiel on solaris 10 due to missing SEM_VALUE_MAX. Extension Modules @@ -21593,7 +21626,7 @@ - Improved BeOS support. -- Support dynamic loading of shared libraries on NetBSD platforms that +- Support dynamic loading of shared libraries on NetBSD platforms that use ELF (i.e., MIPS and Alpha systems). Configuration/build changes @@ -21810,7 +21843,7 @@ higher-level classes in code.py. - turtle.py is a new module for simple turtle graphics. I'm still -working on it; let me know if you use this to teach Python to children +working on it; let me know if you use this to teach Python to children or other novices without prior programming experience. Obsoleted library modules @@ -21942,7 +21975,7 @@ Changes to tools ---------------- -- New, improved version of Barry Warsaw's Misc/python-mode.el (editing +- New, improved version of Barry Warsaw's Misc/python-mode.el (editing support for Emacs). - tabnanny.py: added a -q ('quiet') option to tabnanny, which causes @@ -22217,7 +22250,7 @@ ----------------- - Install zlib.dll in the DLLs directory instead of in the win32 -system directory, to avoid conflicts with other applications that have +system directory, to avoid conflicts with other applications that have their own zlib.dll. Test Suite @@ -22297,7 +22330,7 @@ so that a symlink to a symlink can work. - Added a hack so that when you type 'quit' or 'exit' at the -interpreter, you get a friendly explanation of how to press Ctrl-D (or +interpreter, you get a friendly explanation of how to press Ctrl-D (or Ctrl-Z) to exit. - New and improved Misc/python-mode.el (Python mode for Emacs). @@ -22470,13 +22503,13 @@ range. Also, randint(a, b) is now redefined as randrange(a, b+1), adding extra range and type checking to its arguments! -- Add some semi-thread-safety to random.gauss() (it used to be able to +- Add some semi-thread-safety to random.gauss() (it used to be able to crash when invoked from separate threads; now the worst it can do is give a duplicate result occasionally). - Some restructuring and generalization done to cmd.py. -- Major upgrade to ConfigParser.py; converted to using 're', added new +- Major upgrade to ConfigParser.py; converted to using 're', added new exceptions, support underscore in section header and option name. No longer add 'name' option to every section; instead, add '__name__'. @@ -22673,7 +22706,7 @@ ----------------- - The registry key used is now "1.5" instead of "1.5.x" -- so future -versions of 1.5 and Mark Hammond's win32all installer don't need to be +versions of 1.5 and Mark Hammond's win32all installer don't need to be resynchronized. Windows Tools @@ -22740,7 +22773,7 @@ dynamically add one or many entries to the table of built-in modules. - New macro Py_InitModule3(name, methods, doc) which calls -Py_InitModule4() with appropriate arguments. (The -4 variant requires +Py_InitModule4() with appropriate arguments. (The -4 variant requires you to pass an obscure version number constant which is always the same.) - New APIs PySys_WriteStdout() and PySys_WriteStderr() to write to @@ -22812,7 +22845,7 @@ Syntax change ------------- -- The raise statement can now be used without arguments, to re-raise +- The raise statement can now be used without arguments, to re-raise a previously set exception. This should be used after catching an exception with an except clause only, either in the except clause or later in the same function. @@ -22871,7 +22904,7 @@ Demo/tkinter/guido/paint.py -- Dave Mitchell Demo/sockets/unixserver.py -- Piet van Oostrum - + - Much better freeze support. The freeze script can now freeze hierarchical module names (with a corresponding change to import.c), @@ -23010,7 +23043,7 @@ - New command supported by the ftplib module: rmd(); also fixed some minor bugs. -- The profile module now uses a different timer function by default -- +- The profile module now uses a different timer function by default -- time.clock() is generally better than os.times(). This makes it work better on Windows NT, too. @@ -23049,14 +23082,14 @@ - In the multifile module, support the optional second parameter to seek() when possible. -- Several fixes to the gopherlib module by Lars Marius Garshol. Also, +- Several fixes to the gopherlib module by Lars Marius Garshol. Also, urlparse now correctly handles Gopher URLs with query strings. - Fixed a tiny bug in format_exception() in the traceback module. Also rewrite tb_lineno() to be compatible with JPython (and not disturb the current exception!); by Jim Hugunin. -- The httplib module is more robust when servers send a short response +- The httplib module is more robust when servers send a short response -- courtesy Tim O'Malley. Tkinter and friends @@ -23071,7 +23104,7 @@ no longer use the default root. - The interfaces for the bind*() and unbind() widget methods have been -redesigned; the bind*() methods now return the name of the Tcl command +redesigned; the bind*() methods now return the name of the Tcl command created for the callback, and this can be passed as a optional argument to unbind() in order to delete the command (normally, such commands are automatically unbound when the widget is destroyed, but @@ -23107,7 +23140,7 @@ dictionary to allow recursive container types to detect recursion in their repr(), str() and print implementations. -- New function PyObject_Not(x) calculates (not x) according to Python's +- New function PyObject_Not(x) calculates (not x) according to Python's standard rules (basically, it negates the outcome PyObject_IsTrue(x). - New function _PyModule_Clear(), which clears a module's dictionary @@ -23268,7 +23301,7 @@ instances with copy.py. The cPickle.c changes and some pickle.py changes are courtesy Jim Fulton. -- Locale support in he "re" (Perl regular expressions) module. Use +- Locale support in he "re" (Perl regular expressions) module. Use the flag re.L (or re.LOCALE) to enable locale-specific matching rules for \w and \b. The in-line syntax for this flag is (?L). @@ -23334,7 +23367,7 @@ - Some improvements to the _tkinter build line suggested by Case Roole. -- A full suite of platform specific files for NetBSD 1.x, submitted by +- A full suite of platform specific files for NetBSD 1.x, submitted by Anders Andersen. - New Solaris specific header STROPTS.py. @@ -23404,7 +23437,7 @@ if there are lots of duplicates, and otherwise at least as good. - Added "uue" as an alias for "uuencode" to mimetools.py. (Hm, the -uudecode bug where it complaints about trailing garbage is still there +uudecode bug where it complaints about trailing garbage is still there :-( ). - pickle.py requires integers in text mode to be in decimal notation @@ -24180,7 +24213,7 @@ The Python/C API for frames is changed (you shouldn't be using this anyway). -- Significant speedup by inlining some common opcodes for common operand +- Significant speedup by inlining some common opcodes for common operand types (e.g. i+i, i-i, and list[i]). Fredrik Lundh. - Small speedup by reordering the method tables of some common @@ -24206,34 +24239,34 @@ printing the documentation now kills fewer trees -- the margins have been reduced. -- I have started documenting the Python/C API. Unfortunately this project -hasn't been completed yet. It will be complete before the final release of -Python 1.5, though. At the moment, it's better to read the LaTeX source +- I have started documenting the Python/C API. Unfortunately this project +hasn't been completed yet. It will be complete before the final release of +Python 1.5, though. At the moment, it's better to read the LaTeX source than to attempt to run it through LaTeX and print the resulting dvi file. -- The posix module (and hence os.py) now has doc strings! Thanks to Neil -Schemenauer. I received a few other contributions of doc strings. In most +- The posix module (and hence os.py) now has doc strings! Thanks to Neil +Schemenauer. I received a few other contributions of doc strings. In most other places, doc strings are still wishful thinking... Language changes ---------------- -- Private variables with leading double underscore are now a permanent -feature of the language. (These were experimental in release 1.4. I have -favorable experience using them; I can't label them "experimental" +- Private variables with leading double underscore are now a permanent +feature of the language. (These were experimental in release 1.4. I have +favorable experience using them; I can't label them "experimental" forever.) -- There's new string literal syntax for "raw strings". Prefixing a string -literal with the letter r (or R) disables all escape processing in the -string; for example, r'\n' is a two-character string consisting of a -backslash followed by the letter n. This combines with all forms of string -quotes; it is actually useful for triple quoted doc strings which might -contain references to \n or \t. An embedded quote prefixed with a -backslash does not terminate the string, but the backslash is still -included in the string; for example, r'\'' is a two-character string -consisting of a backslash and a quote. (Raw strings are also -affectionately known as Robin strings, after their inventor, Robin +- There's new string literal syntax for "raw strings". Prefixing a string +literal with the letter r (or R) disables all escape processing in the +string; for example, r'\n' is a two-character string consisting of a +backslash followed by the letter n. This combines with all forms of string +quotes; it is actually useful for triple quoted doc strings which might +contain references to \n or \t. An embedded quote prefixed with a +backslash does not terminate the string, but the backslash is still +included in the string; for example, r'\'' is a two-character string +consisting of a backslash and a quote. (Raw strings are also +affectionately known as Robin strings, after their inventor, Robin Friedrich.) - There's a simple assert statement, and a new exception @@ -24262,10 +24295,10 @@ - The obsolete exception ConflictError (presumably used by the long obsolete access statement) has been deleted. -- There's a new function sys.exc_info() which returns the tuple +- There's a new function sys.exc_info() which returns the tuple (sys.exc_type, sys.exc_value, sys.exc_traceback) in a thread-safe way. -- There's a new variable sys.executable, pointing to the executable file +- There's a new variable sys.executable, pointing to the executable file for the Python interpreter. - The sort() methods for lists no longer uses the C library qsort(); I @@ -24291,11 +24324,11 @@ returning from a function that caught an exception. - There's a new "buffer" interface. Certain objects (e.g. strings and -arrays) now support the "buffer" protocol. Buffer objects are acceptable -whenever formerly a string was required for a write operation; mutable +arrays) now support the "buffer" protocol. Buffer objects are acceptable +whenever formerly a string was required for a write operation; mutable buffer objects can be the target of a read operation using the call -f.readinto(buffer). A cool feature is that regular expression matching now -also work on array objects. Contribution by Jack Jansen. (Needs +f.readinto(buffer). A cool feature is that regular expression matching now +also work on array objects. Contribution by Jack Jansen. (Needs documentation.) - String interning: dictionary lookups are faster when the lookup @@ -24587,7 +24620,7 @@ of message sequence specifiers without invoking a subprocess. Also added a createmessage() method by Lars Wirzenius. -- The StringIO.StringIO class now supports readline(nbytes). (Lars +- The StringIO.StringIO class now supports readline(nbytes). (Lars Wirzenius.) (Of course, you should be using cStringIO for performance.) - UserDict.py supports the new dictionary methods as well. @@ -24635,8 +24668,8 @@ - Various small fixes to the nntplib.py module that I can't bother to document in detail. -- Sjoerd Mullender's mimify.py module now supports base64 encoding and -includes functions to handle the funny encoding you sometimes see in mail +- Sjoerd Mullender's mimify.py module now supports base64 encoding and +includes functions to handle the funny encoding you sometimes see in mail headers. It is now documented. - mailbox.py: Added BabylMailbox. Improved the way the mailbox is @@ -24987,23 +25020,23 @@ NT (the old VC++ 4.2 Makefile is also still supported, but will eventually be withdrawn due to its bulkiness). -- See the note on the new module search path in the "Miscellaneous" section +- See the note on the new module search path in the "Miscellaneous" section above. - Support for Win32s (the 32-bit Windows API under Windows 3.1) is basically withdrawn. If it still works for you, you're lucky. -- There's a new extension module, msvcrt.c, which provides various -low-level operations defined in the Microsoft Visual C++ Runtime Library. -These include locking(), setmode(), get_osfhandle(), set_osfhandle(), and +- There's a new extension module, msvcrt.c, which provides various +low-level operations defined in the Microsoft Visual C++ Runtime Library. +These include locking(), setmode(), get_osfhandle(), set_osfhandle(), and console I/O functions like kbhit(), getch() and putch(). - The -u option not only sets the standard I/O streams to unbuffered status, but also sets them in binary mode. (This can also be done using msvcrt.setmode(), by the way.) -- The, sys.prefix and sys.exec_prefix variables point to the directory -where Python is installed, or to the top of the source tree, if it was run +- The, sys.prefix and sys.exec_prefix variables point to the directory +where Python is installed, or to the top of the source tree, if it was run from there. - The various os.path modules (posixpath, ntpath, macpath) now support @@ -25011,7 +25044,7 @@ os.path.join(a, b, c) is the same as os.path.join(a, os.path.join(b, c)). -- The ntpath module (normally used as os.path) supports ~ to $HOME +- The ntpath module (normally used as os.path) supports ~ to $HOME expansion in expanduser(). - The freeze tool now works on Windows. @@ -25309,47 +25342,47 @@ - New module whichdb recognizes dbm, gdbm and bsddb/dbhash files. -- The Doc/Makefile targets have been reorganized somewhat to remove the +- The Doc/Makefile targets have been reorganized somewhat to remove the insistence on always generating PostScript. - The texinfo to html filter (Doc/texi2html.py) has been improved somewhat. -- "errors.h" has been renamed to "pyerrors.h" to resolve a long-standing +- "errors.h" has been renamed to "pyerrors.h" to resolve a long-standing name conflict on the Mac. -- Linking a module compiled with a different setting for Py_TRACE_REFS now +- Linking a module compiled with a different setting for Py_TRACE_REFS now generates a linker error rather than a core dump. -- The cgi module has a new convenience function print_exception(), which -formats a python exception using HTML. It also fixes a bug in the -compatibility code and adds a dubious feature which makes it possible to +- The cgi module has a new convenience function print_exception(), which +formats a python exception using HTML. It also fixes a bug in the +compatibility code and adds a dubious feature which makes it possible to have two query strings, one in the URL and one in the POST data. -- A subtle change in the unpickling of class instances makes it possible -to unpickle in restricted execution mode, where the __dict__ attribute is +- A subtle change in the unpickling of class instances makes it possible +to unpickle in restricted execution mode, where the __dict__ attribute is not available (but setattr() is). -- Documentation for os.path.splitext() (== posixpath.splitext()) has been +- Documentation for os.path.splitext() (== posixpath.splitext()) has been cleared up. It splits at the *last* dot. - posixfile locking is now also correctly supported on AIX. -- The tempfile module once again honors an initial setting of tmpdir. It +- The tempfile module once again honors an initial setting of tmpdir. It now works on Windows, too. -- The traceback module has some new functions to extract, format and print +- The traceback module has some new functions to extract, format and print the active stack. -- Some translation functions in the urllib module have been made a little +- Some translation functions in the urllib module have been made a little less sluggish. -- The addtag_* methods for Canvas widgets in Tkinter as well as in the -separate Canvas class have been fixed so they actually do something +- The addtag_* methods for Canvas widgets in Tkinter as well as in the +separate Canvas class have been fixed so they actually do something meaningful. - A tiny _test() function has been added to Tkinter.py. -- A generic Makefile for dynamically loaded modules is provided in the Misc +- A generic Makefile for dynamically loaded modules is provided in the Misc subdirectory (Misc/gMakefile). - A new version of python-mode.el for Emacs is provided. See @@ -25357,25 +25390,25 @@ separate file pyimenu.el is no longer needed, imenu support is folded into python-mode.el. -- The configure script can finally correctly find the readline library in a -non-standard location. The LDFLAGS variable is passed on the Makefiles +- The configure script can finally correctly find the readline library in a +non-standard location. The LDFLAGS variable is passed on the Makefiles from the configure script. -- Shared libraries are now installed as programs (i.e. with executable +- Shared libraries are now installed as programs (i.e. with executable permission). This is required on HP-UX and won't hurt on other systems. -- The objc.c module is no longer part of the distribution. Objective-C +- The objc.c module is no longer part of the distribution. Objective-C support may become available as contributed software on the ftp site. - The sybase module is no longer part of the distribution. A much improved sybase module is available as contributed software from the ftp site. -- _tkinter is now compatible with Tcl 7.5 / Tk 4.1 patch1 on Windows and -Mac (don't use unpatched Tcl/Tk!). The default line in the Setup.in file +- _tkinter is now compatible with Tcl 7.5 / Tk 4.1 patch1 on Windows and +Mac (don't use unpatched Tcl/Tk!). The default line in the Setup.in file now links with Tcl 7.5 / Tk 4.1 rather than 7.4/4.0. -- In Setup, you can now write "*shared*" instead of "*noconfig*", and you +- In Setup, you can now write "*shared*" instead of "*noconfig*", and you can use *.so and *.sl as shared libraries. - Some more fidgeting for AIX shared libraries. @@ -25384,81 +25417,81 @@ (Note -- a complete replacement by Niels Mo"ller, called gpmodule, is available from the contrib directory on the ftp site.) -- A warning is written to sys.stderr when a __del__ method raises an +- A warning is written to sys.stderr when a __del__ method raises an exception (formerly, such exceptions were completely ignored). -- The configure script now defines HAVE_OLD_CPP if the C preprocessor is +- The configure script now defines HAVE_OLD_CPP if the C preprocessor is incapable of ANSI style token concatenation and stringification. -- All source files (except a few platform specific modules) are once again +- All source files (except a few platform specific modules) are once again compatible with K&R C compilers as well as ANSI compilers. In particular, -ANSI-isms have been removed or made conditional in complexobject.c, +ANSI-isms have been removed or made conditional in complexobject.c, getargs.c and operator.c. -- The abstract object API has three new functions, PyObject_DelItem, +- The abstract object API has three new functions, PyObject_DelItem, PySequence_DelItem, and PySequence_DelSlice. -- The operator module has new functions delitem and delslice, and the -functions "or" and "and" are renamed to "or_" and "and_" (since "or" and +- The operator module has new functions delitem and delslice, and the +functions "or" and "and" are renamed to "or_" and "and_" (since "or" and "and" are reserved words). ("__or__" and "__and__" are unchanged.) -- The environment module is no longer supported; putenv() is now a function +- The environment module is no longer supported; putenv() is now a function in posixmodule (also under NT). - Error in filter(, "") has been fixed. - Unrecognized keyword arguments raise TypeError, not KeyError. -- Better portability, fewer bugs and memory leaks, fewer compiler warnings, +- Better portability, fewer bugs and memory leaks, fewer compiler warnings, some more documentation. -- Bug in float power boundary case (0.0 to the negative integer power) +- Bug in float power boundary case (0.0 to the negative integer power) fixed. -- The test of negative number to the float power has been moved from the -built-in pow() functin to floatobject.c (so complex numbers can yield the +- The test of negative number to the float power has been moved from the +built-in pow() functin to floatobject.c (so complex numbers can yield the correct result). -- The bug introduced in beta2 where shared libraries loaded (using +- The bug introduced in beta2 where shared libraries loaded (using dlopen()) from the current directory would fail, has been fixed. -- Modules imported as shared libraries now also have a __file__ attribute, -giving the filename from which they were loaded. The only modules without +- Modules imported as shared libraries now also have a __file__ attribute, +giving the filename from which they were loaded. The only modules without a __file__ attribute now are built-in modules. -- On the Mac, dynamically loaded modules can end in either ".slb" or -"..slb" where is either "CFM68K" or "ppc". The ".slb" +- On the Mac, dynamically loaded modules can end in either ".slb" or +"..slb" where is either "CFM68K" or "ppc". The ".slb" extension should only be used for "fat" binaries. -- C API addition: marshal.c now supports +- C API addition: marshal.c now supports PyMarshal_WriteObjectToString(object). - C API addition: getargs.c now supports PyArg_ParseTupleAndKeywords(args, kwdict, format, kwnames, ...) to parse keyword arguments. -- The PC versioning scheme (sys.winver) has changed once again. the -version number is now "...", where the -first three s are the Python version (e.g. "1.4.0" for Python 1.4, -"1.4.1" for Python 1.4.1 -- the beta level is not included) and +- The PC versioning scheme (sys.winver) has changed once again. the +version number is now "...", where the +first three s are the Python version (e.g. "1.4.0" for Python 1.4, +"1.4.1" for Python 1.4.1 -- the beta level is not included) and is the four-digit PYTHON_API_VERSION (currently 1005). - h2py.py accepts whitespace before the # in CPP directives -- On Solaris 2.5, it should now be possible to use either Posix threads or -Solaris threads (XXX: how do you select which is used???). (Note: the -Python pthreads interface doesn't fully support semaphores yet -- anyone +- On Solaris 2.5, it should now be possible to use either Posix threads or +Solaris threads (XXX: how do you select which is used???). (Note: the +Python pthreads interface doesn't fully support semaphores yet -- anyone care to fix this?) -- Thread support should now work on AIX, using either DCE threads or +- Thread support should now work on AIX, using either DCE threads or pthreads. - New file Demo/sockets/unicast.py -- Working Mac port, with CFM68K support, with Tk 4.1 support (though not +- Working Mac port, with CFM68K support, with Tk 4.1 support (though not both) (XXX) -- New project setup for PC port, now compatible with PythonWin, with +- New project setup for PC port, now compatible with PythonWin, with _tkinter and NumPy support (XXX) - New module site.py (XXX) @@ -25475,7 +25508,7 @@ - string.atoi c.s. now raise an exception for an empty input string. -- At last, it is no longer necessary to define HAVE_CONFIG_H in order to +- At last, it is no longer necessary to define HAVE_CONFIG_H in order to have config.h included at various places. - Unrecognized keyword arguments now raise TypeError rather than KeyError. @@ -25483,25 +25516,25 @@ - The makesetup script recognizes files with extension .so or .sl as (shared) libraries. -- 'access' is no longer a reserved word, and all code related to its -implementation is gone (or at least #ifdef'ed out). This should make +- 'access' is no longer a reserved word, and all code related to its +implementation is gone (or at least #ifdef'ed out). This should make Python a little speedier too! -- Performance enhancements suggested by Sjoerd Mullender. This includes -the introduction of two new optional function pointers in type object, -getattro and setattro, which are like getattr and setattr but take a +- Performance enhancements suggested by Sjoerd Mullender. This includes +the introduction of two new optional function pointers in type object, +getattro and setattro, which are like getattr and setattr but take a string object instead of a C string pointer. -- New operations in string module: lstrip(s) and rstrip(s) strip whitespace -only on the left or only on the right, A new optional third argument to -split() specifies the maximum number of separators honored (so -splitfields(s, sep, n) returns a list of at most n+1 elements). (Since +- New operations in string module: lstrip(s) and rstrip(s) strip whitespace +only on the left or only on the right, A new optional third argument to +split() specifies the maximum number of separators honored (so +splitfields(s, sep, n) returns a list of at most n+1 elements). (Since 1.3, splitfields(s, None) is totally equivalent to split(s).) -string.capwords() has an optional second argument specifying the +string.capwords() has an optional second argument specifying the separator (which is passed to split()). -- regsub.split() has the same addition as string.split(). regsub.splitx(s, -sep, maxsep) implements the functionality that was regsub.split(s, 1) in +- regsub.split() has the same addition as string.split(). regsub.splitx(s, +sep, maxsep) implements the functionality that was regsub.split(s, 1) in 1.4beta2 (return a list containing the delimiters as well as the words). - Final touch for AIX loading, rewritten Misc/AIX-NOTES. @@ -25545,11 +25578,11 @@ meaningful value (a few things were botched in beta 1). Lib/dos_8x3 is now a standard part of the distribution (alas). -- More improvements to the installation procedure. Typing "make install" -now inserts the version number in the pathnames of almost everything -installed, and creates the machine dependent modules (FCNTL.py etc.) if not -supplied by the distribution. (XXX There's still a problem with the latter -because the "regen" script requires that Python is installed. Some manual +- More improvements to the installation procedure. Typing "make install" +now inserts the version number in the pathnames of almost everything +installed, and creates the machine dependent modules (FCNTL.py etc.) if not +supplied by the distribution. (XXX There's still a problem with the latter +because the "regen" script requires that Python is installed. Some manual intervention may still be required.) (This has been fixed in 1.4beta3.) - New modules: errno, operator (XXX). @@ -25612,8 +25645,8 @@ - Added sys.platform and sys.exec_platform for Bill Janssen. -- Installation has been completely overhauled. "make install" now installs -everything, not just the python binary. Installation uses the install-sh +- Installation has been completely overhauled. "make install" now installs +everything, not just the python binary. Installation uses the install-sh script (borrowed from X11) to install each file. - New functions in the posix module: mkfifo, plock, remove (== unlink), @@ -25623,59 +25656,59 @@ - Shared library support for FreeBSD. -- The --with-readline option can now be used without a DIRECTORY argument, -for systems where libreadline.* is in one of the standard places. It is +- The --with-readline option can now be used without a DIRECTORY argument, +for systems where libreadline.* is in one of the standard places. It is also possible for it to be a shared library. -- The extension tkinter has been renamed to _tkinter, to avoid confusion -with Tkinter.py oncase insensitive file systems. It now supports Tk 4.1 as +- The extension tkinter has been renamed to _tkinter, to avoid confusion +with Tkinter.py oncase insensitive file systems. It now supports Tk 4.1 as well as 4.0. -- Author's change of address from CWI in Amsterdam, The Netherlands, to +- Author's change of address from CWI in Amsterdam, The Netherlands, to CNRI in Reston, VA, USA. -- The math.hypot() function is now always available (if it isn't found in +- The math.hypot() function is now always available (if it isn't found in the C math library, Python provides its own implementation). -- The latex documentation is now compatible with latex2e, thanks to David +- The latex documentation is now compatible with latex2e, thanks to David Ascher. - The expression x**y is now equivalent to pow(x, y). - The indexing expression x[a, b, c] is now equivalent to x[(a, b, c)]. -- Complex numbers are now supported. Imaginary constants are written with -a 'j' or 'J' prefix, general complex numbers can be formed by adding a real -part to an imaginary part, like 3+4j. Complex numbers are always stored in -floating point form, so this is equivalent to 3.0+4.0j. It is also -possible to create complex numbers with the new built-in function -complex(re, [im]). For the footprint-conscious, complex number support can +- Complex numbers are now supported. Imaginary constants are written with +a 'j' or 'J' prefix, general complex numbers can be formed by adding a real +part to an imaginary part, like 3+4j. Complex numbers are always stored in +floating point form, so this is equivalent to 3.0+4.0j. It is also +possible to create complex numbers with the new built-in function +complex(re, [im]). For the footprint-conscious, complex number support can be disabled by defining the symbol WITHOUT_COMPLEX. - New built-in function list() is the long-awaited counterpart of tuple(). -- There's a new "cmath" module which provides the same functions as the -"math" library but with complex arguments and results. (There are very -good reasons why math.sqrt(-1) still raises an exception -- you have to use +- There's a new "cmath" module which provides the same functions as the +"math" library but with complex arguments and results. (There are very +good reasons why math.sqrt(-1) still raises an exception -- you have to use cmath.sqrt(-1) to get 1j for an answer.) -- The Python.h header file (which is really the same as allobjects.h except -it disables support for old style names) now includes several more files, +- The Python.h header file (which is really the same as allobjects.h except +it disables support for old style names) now includes several more files, so you have to have fewer #include statements in the average extension. -- The NDEBUG symbol is no longer used. Code that used to be dependent on -the presence of NDEBUG is now present on the absence of DEBUG. TRACE_REFS -and REF_DEBUG have been renamed to Py_TRACE_REFS and Py_REF_DEBUG, -respectively. At long last, the source actually compiles and links without +- The NDEBUG symbol is no longer used. Code that used to be dependent on +the presence of NDEBUG is now present on the absence of DEBUG. TRACE_REFS +and REF_DEBUG have been renamed to Py_TRACE_REFS and Py_REF_DEBUG, +respectively. At long last, the source actually compiles and links without errors when this symbol is defined. -- Several symbols that didn't follow the new naming scheme have been -renamed (usually by adding to rename2.h) to use a Py or _Py prefix. There -are no external symbols left without a Py or _Py prefix, not even those -defined by sources that were incorporated from elsewhere (regexpr.c, +- Several symbols that didn't follow the new naming scheme have been +renamed (usually by adding to rename2.h) to use a Py or _Py prefix. There +are no external symbols left without a Py or _Py prefix, not even those +defined by sources that were incorporated from elsewhere (regexpr.c, md5c.c). (Macros are a different story...) -- There are now typedefs for the structures defined in config.c and +- There are now typedefs for the structures defined in config.c and frozen.c. - New PYTHON_API_VERSION value and .pyc file magic number. @@ -25689,125 +25722,125 @@ - The binhex and binascii modules now actually work. - The cgi module has been almost totally rewritten and documented. -It now supports file upload and a new data type to handle forms more +It now supports file upload and a new data type to handle forms more flexibly. - The formatter module (for use with htmllib) has been overhauled (again). - The ftplib module now supports passive mode and has doc strings. -- In (ideally) all places where binary files are read or written, the file -is now correctly opened in binary mode ('rb' or 'wb') so the code will work +- In (ideally) all places where binary files are read or written, the file +is now correctly opened in binary mode ('rb' or 'wb') so the code will work on Mac or PC. -- Dummy versions of os.path.expandvars() and expanduser() are now provided +- Dummy versions of os.path.expandvars() and expanduser() are now provided on non-Unix platforms. -- Module urllib now has two new functions url2pathname and pathname2url -which turn local filenames into "file:..." URLs using the same rules as -Netscape (why be different). it also supports urlretrieve() with a -pathname parameter, and honors the proxy environment variables (http_proxy +- Module urllib now has two new functions url2pathname and pathname2url +which turn local filenames into "file:..." URLs using the same rules as +Netscape (why be different). it also supports urlretrieve() with a +pathname parameter, and honors the proxy environment variables (http_proxy etc.). The URL parsing has been improved somewhat, too. -- Micro improvements to urlparse. Added urlparse.urldefrag() which +- Micro improvements to urlparse. Added urlparse.urldefrag() which removes a trailing ``#fragment'' if any. - The mailbox module now supports MH style message delimiters as well. -- The mhlib module contains some new functionality: setcontext() to set the -current folder and parsesequence() to parse a sequence as commonly passed +- The mhlib module contains some new functionality: setcontext() to set the +current folder and parsesequence() to parse a sequence as commonly passed to MH commands (e.g. 1-10 or last:5). -- New module mimify for conversion to and from MIME format of email +- New module mimify for conversion to and from MIME format of email messages. -- Module ni now automatically installs itself when first imported -- this -is against the normal rule that modules should define classes and functions -but not invoke them, but appears more useful in the case that two +- Module ni now automatically installs itself when first imported -- this +is against the normal rule that modules should define classes and functions +but not invoke them, but appears more useful in the case that two different, independent modules want to use ni's features. - Some small performance enhancements in module pickle. -- Small interface change to the profile.run*() family of functions -- more +- Small interface change to the profile.run*() family of functions -- more sensible handling of return values. -- The officially registered Mac creator for Python files is 'Pyth'. This +- The officially registered Mac creator for Python files is 'Pyth'. This replaces 'PYTH' which was used before but never registered. - Added regsub.capwords(). (XXX) -- Added string.capwords(), string.capitalize() and string.translate(). +- Added string.capwords(), string.capitalize() and string.translate(). (XXX) -- Fixed an interface bug in the rexec module: it was impossible to pass a -hooks instance to the RExec class. rexec now also supports the dynamic -loading of modules from shared libraries. Some other interfaces have been +- Fixed an interface bug in the rexec module: it was impossible to pass a +hooks instance to the RExec class. rexec now also supports the dynamic +loading of modules from shared libraries. Some other interfaces have been added too. -- Module rfc822 now caches the headers in a dictionary for more efficient +- Module rfc822 now caches the headers in a dictionary for more efficient lookup. -- The sgmllib module now understands a limited number of SGML "shorthands" +- The sgmllib module now understands a limited number of SGML "shorthands" like .... (It's not clear that this was a good idea...) -- The tempfile module actually tries a number of different places to find a -usable temporary directory. (This was prompted by certain Linux -installations that appear to be missing a /usr/tmp directory.) [A bug in -the implementation that would ignore a pre-existing tmpdir global has been +- The tempfile module actually tries a number of different places to find a +usable temporary directory. (This was prompted by certain Linux +installations that appear to be missing a /usr/tmp directory.) [A bug in +the implementation that would ignore a pre-existing tmpdir global has been fixed in beta3.] - Much improved and enhanved FileDialog module for Tkinter. -- Many small changes to Tkinter, to bring it more in line with Tk 4.0 (as +- Many small changes to Tkinter, to bring it more in line with Tk 4.0 (as well as Tk 4.1). -- New socket interfaces include ntohs(), ntohl(), htons(), htonl(), and -s.dup(). Sockets now work correctly on Windows. On Windows, the built-in -extension is called _socket and a wrapper module win/socket.py provides -"makefile()" and "dup()" functionality. On Windows, the select module +- New socket interfaces include ntohs(), ntohl(), htons(), htonl(), and +s.dup(). Sockets now work correctly on Windows. On Windows, the built-in +extension is called _socket and a wrapper module win/socket.py provides +"makefile()" and "dup()" functionality. On Windows, the select module works only with socket objects. - Bugs in bsddb module fixed (e.g. missing default argument values). - The curses extension now includes when available. -- The gdbm module now supports opening databases in "fast" mode by +- The gdbm module now supports opening databases in "fast" mode by specifying 'f' as the second character or the mode string. -- new variables sys.prefix and sys.exec_prefix pass corresponding +- new variables sys.prefix and sys.exec_prefix pass corresponding configuration options / Makefile variables to the Python programmer. -- The ``new'' module now supports creating new user-defined classes as well +- The ``new'' module now supports creating new user-defined classes as well as instances thereof. -- The soundex module now sports get_soundex() to get the soundex value for an -arbitrary string (formerly it would only do soundex-based string +- The soundex module now sports get_soundex() to get the soundex value for an +arbitrary string (formerly it would only do soundex-based string comparison) as well as doc strings. -- New object type "cobject" to safely wrap void pointers for passing them +- New object type "cobject" to safely wrap void pointers for passing them between various extension modules. - More efficient computation of float**smallint. -- The mysterious bug whereby "x.x" (two occurrences of the same -one-character name) typed from the commandline would sometimes fail +- The mysterious bug whereby "x.x" (two occurrences of the same +one-character name) typed from the commandline would sometimes fail mysteriously. -- The initialization of the readline function can now be invoked by a C +- The initialization of the readline function can now be invoked by a C extension through PyOS_ReadlineInit(). -- There's now an externally visible pointer PyImport_FrozenModules which +- There's now an externally visible pointer PyImport_FrozenModules which can be changed by an embedding application. -- The argument parsing functions now support a new format character 'D' to +- The argument parsing functions now support a new format character 'D' to specify complex numbers. - Various memory leaks plugged and bugs fixed. -- Improved support for posix threads (now that real implementations are +- Improved support for posix threads (now that real implementations are beginning to apepar). Still no fully functioning semaphores. -- Some various and sundry improvements and new entries in the Tools +- Some various and sundry improvements and new entries in the Tools directory. @@ -27417,7 +27450,7 @@ The limit on the size of the *run-time* stack has completely been removed -- this means that tuple or list displays can contain any number of elements (formerly more than 50 would crash the -interpreter). +interpreter). Changes to existing built-in functions and methods diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -4359,4137 +4359,4 @@ - Issue #18569: The installer now adds .py to the PATHEXT variable when extensions are registered. Patch by Paul Moore. - -What's New in Python 3.3.0? -=========================== - -*Release date: 29-Sep-2012* - -Core and Builtins ------------------ - -- Issue #16046: Fix loading sourceless legacy .pyo files. - -- Issue #16060: Fix refcounting bug when `__trunc__()` returns an object whose - `__int__()` gives a non-integer. Patch by Serhiy Storchaka. - -Extension Modules ------------------ - -- Issue #16012: Fix a regression in pyexpat. The parser's `UseForeignDTD()` - method doesn't require an argument again. - - -What's New in Python 3.3.0 Release Candidate 3? -=============================================== - -*Release date: 23-Sep-2012* - -Core and Builtins ------------------ - -- Issue #15900: Fix reference leak in `PyUnicode_TranslateCharmap()`. - -- Issue #15926: Fix crash after multiple reinitializations of the interpreter. - -- Issue #15895: Fix FILE pointer leak in one error branch of - `PyRun_SimpleFileExFlags()` when filename points to a pyc/pyo file, closeit is - false an and set_main_loader() fails. - -- Fixes for a few crash and memory leak regressions found by Coverity. - -Library -------- - -- Issue #15882: Change `_decimal` to accept any coefficient tuple when - constructing infinities. This is done for backwards compatibility with - decimal.py: Infinity coefficients are undefined in _decimal (in accordance - with the specification). - -- Issue #15925: Fix a regression in `email.util` where the `parsedate()` and - `parsedate_tz()` functions did not return None anymore when the argument could - not be parsed. - -Extension Modules ------------------ - -- Issue #15973: Fix a segmentation fault when comparing datetime timezone - objects. - -- Issue #15977: Fix memory leak in Modules/_ssl.c when the function - _set_npn_protocols() is called multiple times, thanks to Daniel Sommermann. - -- Issue #15969: `faulthandler` module: rename dump_tracebacks_later() to - dump_traceback_later() and cancel_dump_tracebacks_later() to - cancel_dump_traceback_later(). - -- _decimal module: use only C 89 style comments. - - -What's New in Python 3.3.0 Release Candidate 2? -=============================================== - -*Release date: 09-Sep-2012* - -Core and Builtins ------------------ - -- Issue #13992: The trashcan mechanism is now thread-safe. This eliminates - sporadic crashes in multi-thread programs when several long deallocator chains - ran concurrently and involved subclasses of built-in container types. - -- Issue #15784: Modify `OSError`.__str__() to better distinguish between errno - error numbers and Windows error numbers. - -- Issue #15781: Fix two small race conditions in import's module locking. - -Library -------- - -- Issue #17158: Add 'symbols' to help() welcome message; clarify - 'modules spam' messages. - -- Issue #15847: Fix a regression in argparse, which did not accept tuples as - argument lists anymore. - -- Issue #15828: Restore support for C extensions in `imp.load_module()`. - -- Issue #15340: Fix importing the random module when ``/dev/urandom`` cannot be - opened. This was a regression caused by the hash randomization patch. - -- Issue #10650: Deprecate the watchexp parameter of the `Decimal.quantize()` - method. - -- Issue #15785: Modify `window.get_wch()` API of the curses module: return a - character for most keys, and an integer for special keys, instead of always - returning an integer. So it is now possible to distinguish special keys like - keypad keys. - -- Issue #14223: Fix `window.addch()` of the curses module for special characters - like curses.ACS_HLINE: the Python function addch(int) and addch(bytes) is now - calling the C function waddch()/mvwaddch() (as it was done in Python 3.2), - instead of wadd_wch()/mvwadd_wch(). The Python function addch(str) is still - calling the C function wadd_wch()/mvwadd_wch() if the Python curses is linked - to libncursesw. - -Build ------ - -- Issue #15822: Really ensure 2to3 grammar pickles are properly installed - (replaces fixes for Issue #15645). - -Documentation -------------- - -- Issue #15814: The memoryview enhancements in 3.3.0 accidentally permitted the - hashing of multi-dimensional memorviews and memoryviews with multi-byte item - formats. The intended restrictions have now been documented - they will be - correctly enforced in 3.3.1. - - -What's New in Python 3.3.0 Release Candidate 1? -=============================================== - -*Release date: 25-Aug-2012* - -Core and Builtins ------------------ - -- Issue #15573: memoryview comparisons are now performed by value with full - support for any valid struct module format definition. - -- Issue #15316: When an item in the fromlist for `__import__()` doesn't exist, - don't raise an error, but if an exception is raised as part of an import do - let that propagate. - -- Issue #15778: Ensure that ``str(ImportError(msg))`` returns a str even when - msg isn't a str. - -- Issue #2051: Source file permission bits are once again correctly copied to - the cached bytecode file. (The migration to importlib reintroduced this - problem because these was no regression test. A test has been added as part of - this patch) - -- Issue #15761: Fix crash when ``PYTHONEXECUTABLE`` is set on Mac OS X. - -- Issue #15726: Fix incorrect bounds checking in PyState_FindModule. Patch by - Robin Schreiber. - -- Issue #15604: Update uses of `PyObject_IsTrue()` to check for and handle - errors correctly. Patch by Serhiy Storchaka. - -- Issue #14846: `importlib.FileFinder` now handles the case where the directory - being searched is removed after a previous import attempt. - -Library -------- - -- Issue #13370: Ensure that ctypes works on Mac OS X when Python is compiled - using the clang compiler. - -- Issue #13072: The array module's 'u' format code is now deprecated and will be - removed in Python 4.0. - -- Issue #15544: Fix Decimal.__float__ to work with payload-carrying NaNs. - -- Issue #15776: Allow pyvenv to work in existing directory with --clean. - -- Issue #15249: email's BytesGenerator now correctly mangles From lines (when - requested) even if the body contains undecodable bytes. - -- Issue #15777: Fix a refleak in _posixsubprocess. - -- Issue ##665194: Update `email.utils.localtime` to use datetime.astimezone and - correctly handle historic changes in UTC offsets. - -- Issue #15199: Fix JavaScript's default MIME type to application/javascript. - Patch by Bohuslav Kabrda. - -- Issue #12643: `code.InteractiveConsole` now respects `sys.excepthook` when - displaying exceptions. Patch by Aaron Iles. - -- Issue #13579: `string.Formatter` now understands the 'a' conversion specifier. - -- Issue #15595: Fix ``subprocess.Popen(universal_newlines=True)`` for certain - locales (utf-16 and utf-32 family). Patch by Chris Jerdonek. - -- Issue #15477: In cmath and math modules, add workaround for platforms whose - system-supplied log1p function doesn't respect signs of zeros. - -- Issue #15715: `importlib.__import__()` will silence an ImportError when the - use of fromlist leads to a failed import. - -- Issue #14669: Fix pickling of connections and sockets on Mac OS X by - sending/receiving an acknowledgment after file descriptor transfer. - TestPicklingConnection has been reenabled for Mac OS X. - -- Issue #11062: Fix adding a message from file to Babyl mailbox. - -- Issue #15646: Prevent equivalent of a fork bomb when using `multiprocessing` - on Windows without the ``if __name__ == '__main__'`` idiom. - -IDLE ----- - -- Issue #15678: Fix IDLE menus when started from OS X command line (3.3.0b2 - regression). - -Documentation -------------- - -- Touched up the Python 2 to 3 porting guide. - -- Issue #14674: Add a discussion of the `json` module's standard compliance. - Patch by Chris Rebert. - -- Create a 'Concurrent Execution' section in the docs, and split up the - 'Optional Operating System Services' section to use a more user-centric - classification scheme (splitting them across the new CE section, IPC and text - processing). Operating system limitations can be reflected with the Sphinx - ``:platform:`` tag, it doesn't make sense as part of the Table of Contents. - -- Issue #4966: Bring the sequence docs up to date for the Py3k transition and - the many language enhancements since they were original written. - -- The "path importer" misnomer has been replaced with Eric Snow's - more-awkward-but-at-least-not-wrong suggestion of "path based finder" in the - import system reference docs. - -- Issue #15640: Document `importlib.abc.Finder` as deprecated. - -- Issue #15630: Add an example for "continue" stmt in the tutorial. Patch by - Daniel Ellis. - -Tests ------ - -- Issue #15747: ZFS always returns EOPNOTSUPP when attempting to set the - UF_IMMUTABLE flag (via either chflags or lchflags); refactor affected tests in - test_posix.py to account for this. - -- Issue #15285: Refactor the approach for testing connect timeouts using two - external hosts that have been configured specifically for this type of test. - -- Issue #15743: Remove the deprecated method usage in `urllib` tests. Patch by - Jeff Knupp. - -- Issue #15615: Add some tests for the `json` module's handling of invalid input - data. Patch by Kushal Das. - -Build ------ - -- Output lib files for PGO build into PGO directory. - -- Pick up 32-bit launcher from PGO directory on 64-bit PGO build. - -- Drop ``PC\python_nt.h`` as it's not used. Add input dependency on custom - build step. - -- Issue #15511: Drop explicit dependency on pythonxy.lib from _decimal amd64 - configuration. - -- Add missing PGI/PGO configurations for pywlauncher. - -- Issue #15645: Ensure 2to3 grammar pickles are properly installed. - - -What's New in Python 3.3.0 Beta 2? -================================== - -*Release date: 12-Aug-2012* - -Core and Builtins ------------------ - -- Issue #15568: Fix the return value of ``yield from`` when StopIteration is - raised by a custom iterator. - -- Issue #13119: `sys.stdout` and `sys.stderr` are now using "\r\n" newline on - Windows, as Python 2. - -- Issue #15534: Fix the fast-search function for non-ASCII Unicode strings. - -- Issue #15508: Fix the docstring for `__import__()` to have the proper default - value of 0 for 'level' and to not mention negative levels since they are not - supported. - -- Issue #15425: Eliminated traceback noise from more situations involving - importlib. - -- Issue #14578: Support modules registered in the Windows registry again. - -- Issue #15466: Stop using TYPE_INT64 in marshal, to make importlib.h (and other - byte code files) equal between 32-bit and 64-bit systems. - -- Issue #1692335: Move initial exception args assignment to - `BaseException.__new__()` to help pickling of naive subclasses. - -- Issue #12834: Fix `PyBuffer_ToContiguous()` for non-contiguous arrays. - -- Issue #15456: Fix code `__sizeof__()` after #12399 change. Patch by Serhiy - Storchaka. - -- Issue #15404: Refleak in PyMethodObject repr. - -- Issue #15394: An issue in `PyModule_Create()` that caused references to be - leaked on some error paths has been fixed. Patch by Julia Lawall. - -- Issue #15368: An issue that caused bytecode generation to be non-deterministic - has been fixed. - -- Issue #15202: Consistently use the name "follow_symlinks" for new parameters - in os and shutil functions. - -- Issue #15314: ``__main__.__loader__`` is now set correctly during interpreter - startup. - -- Issue #15111: When a module imported using 'from import' has an ImportError - inside itself, don't mask that fact behind a generic ImportError for the - module itself. - -- Issue #15293: Add GC support to the AST base node type. - -- Issue #15291: Fix a memory leak where AST nodes where not properly - deallocated. - -- Issue #15110: Fix the tracebacks generated by "import xxx" to not show the - importlib stack frames. - -- Issue #16369: Global PyTypeObjects not initialized with PyType_Ready(...). - -- Issue #15020: The program name used to search for Python's path is now - "python3" under Unix, not "python". - -- Issue #15897: zipimport.c doesn't check return value of fseek(). - Patch by Felipe Cruz. - -- Issue #15033: Fix the exit status bug when modules invoked using -m switch, - return the proper failure return value (1). Patch contributed by Jeff Knupp. - -- Issue #15229: An `OSError` subclass whose __init__ doesn't call back - OSError.__init__ could produce incomplete instances, leading to crashes when - calling str() on them. - -- Issue #15307: Virtual environments now use symlinks with framework builds on - Mac OS X, like other POSIX builds. - -Library -------- - -- Issue #14590: configparser now correctly strips inline comments when delimiter - occurs earlier without preceding space. - -- Issue #15424: Add a `__sizeof__()` implementation for array objects. Patch by - Ludwig H?hne. - -- Issue #15576: Allow extension modules to act as a package's __init__ module. - -- Issue #15502: Have `importlib.invalidate_caches()` work on `sys.meta_path` - instead of `sys.path_importer_cache`. - -- Issue #15163: Pydoc shouldn't list __loader__ as module data. - -- Issue #15471: Do not use mutable objects as defaults for - `importlib.__import__()`. - -- Issue #15559: To avoid a problematic failure mode when passed to the bytes - constructor, objects in the ipaddress module no longer implement `__index__()` - (they still implement `__int__()` as appropriate). - -- Issue #15546: Fix handling of pathological input data in the peek() and - read1() methods of the BZ2File, GzipFile and LZMAFile classes. - -- Issue #12655: Instead of requiring a custom type, `os.sched_getaffinity()` and - `os.sched_setaffinity()` now use regular sets of integers to represent the - CPUs a process is restricted to. - -- Issue #15538: Fix compilation of the `socket.getnameinfo()` / - `socket.getaddrinfo()` emulation code. Patch by Philipp Hagemeister. - -- Issue #15519: Properly expose WindowsRegistryFinder in importlib (and use the - correct term for it). Original patch by Eric Snow. - -- Issue #15502: Bring the importlib ABCs into line with the current state of the - import protocols given PEP 420. Original patch by Eric Snow. - -- Issue #15499: Launching a webbrowser in Unix used to sleep for a few seconds. - Original patch by Anton Barkovsky. - -- Issue #15463: The faulthandler module truncates strings to 500 characters, - instead of 100, to be able to display long file paths. - -- Issue #6056: Make `multiprocessing` use setblocking(True) on the sockets it - uses. Original patch by J Derek Wilson. - -- Issue #15364: Fix sysconfig.get_config_var('srcdir') to be an absolute path. - -- Issue #15413: `os.times()` had disappeared under Windows. - -- Issue #15402: An issue in the struct module that caused `sys.getsizeof()` to - return incorrect results for struct.Struct instances has been fixed. Initial - patch by Serhiy Storchaka. - -- Issue #15232: When mangle_from is True, `email.Generator` now correctly - mangles lines that start with 'From ' that occur in a MIME preamble or - epilogue. - -- Issue #15094: Incorrectly placed #endif in _tkinter.c. Patch by Serhiy - Storchaka. - -- Issue #13922: `argparse` no longer incorrectly strips '--'s that appear after - the first one. - -- Issue #12353: `argparse` now correctly handles null argument values. - -- Issue #10017, issue #14998: Fix TypeError using pprint on dictionaries with - user-defined types as keys or other unorderable keys. - -- Issue #15397: `inspect.getmodulename()` is now based directly on importlib via - a new `importlib.machinery.all_suffixes()` API. - -- Issue #14635: `telnetlib` will use poll() rather than select() when possible to - avoid failing due to the select() file descriptor limit. - -- Issue #15180: Clarify posixpath.join() error message when mixing str & bytes. - -- Issue #15343: pkgutil now includes an iter_importer_modules implementation for - importlib.machinery.FileFinder (similar to the way it already handled - zipimport.zipimporter). - -- Issue #15314: runpy now sets __main__.__loader__ correctly. - -- Issue #15357: The import emulation in pkgutil is now deprecated. pkgutil uses - importlib internally rather than the emulation. - -- Issue #15233: Python now guarantees that callables registered with the atexit - module will be called in a deterministic order. - -- Issue #15238: `shutil.copystat()` now copies Linux "extended attributes". - -- Issue #15230: runpy.run_path now correctly sets __package__ as described in - the documentation. - -- Issue #15315: Support VS 2010 in distutils cygwincompiler. - -- Issue #15294: Fix a regression in pkgutil.extend_path()'s handling of nested - namespace packages. - -- Issue #15056: `imp.cache_from_source()` and `imp.source_from_cache()` raise - NotImplementedError when `sys.implementation.cache_tag` is set to None. - -- Issue #15256: Grammatical mistake in exception raised by `imp.find_module()`. - -- Issue #5931: `wsgiref` environ variable SERVER_SOFTWARE will specify an - implementation specific term like CPython, Jython instead of generic "Python". - -- Issue #13248: Remove obsolete argument "max_buffer_size" of BufferedWriter and - BufferedRWPair, from the io module. - -- Issue #13248: Remove obsolete argument "version" of `argparse.ArgumentParser`. - -- Issue #14814: Implement more consistent ordering and sorting behaviour for - ipaddress objects. - -- Issue #14814: `ipaddress` network objects correctly return NotImplemented when - compared to arbitrary objects instead of raising TypeError. - -- Issue #14990: Correctly fail with SyntaxError on invalid encoding declaration. - -- Issue #14814: `ipaddress` now provides more informative error messages when - constructing instances directly (changes permitted during beta due to - provisional API status). - -- Issue #15247: `io.FileIO` now raises an error when given a file descriptor - pointing to a directory. - -- Issue #15261: Stop os.stat(fd) crashing on Windows when fd not open. - -- Issue #15166: Implement `imp.get_tag()` using `sys.implementation.cache_tag`. - -- Issue #15210: Catch KeyError when `importlib.__init__()` can't find - _frozen_importlib in sys.modules, not ImportError. - -- Issue #15030: `importlib.abc.PyPycLoader` now supports the new source size - header field in .pyc files. - -- Issue #5346: Preserve permissions of mbox, MMDF and Babyl mailbox files on - flush(). - -- Issue #10571: Fix the "--sign" option of distutils' upload command. Patch by - Jakub Wilk. - -- Issue #9559: If messages were only added, a new file is no longer created and - renamed over the old file when flush() is called on an mbox, MMDF or Babyl - mailbox. - -- Issue #10924: Fixed `crypt.mksalt()` to use a RNG that is suitable for - cryptographic purpose. - -- Issue #15184: Ensure consistent results of OS X configuration tailoring for - universal builds by factoring out common OS X-specific customizations from - sysconfig, distutils.sysconfig, distutils.util, and distutils.unixccompiler - into a new module _osx_support. - -C API ------ - -- Issue #15610: `PyImport_ImportModuleEx()` now uses a 'level' of 0 instead of -1. - -- Issue #15169, issue #14599: Strip out the C implementation of - `imp.source_from_cache()` used by PyImport_ExecCodeModuleWithPathnames() and - used the Python code instead. Leads to PyImport_ExecCodeModuleObject() to not - try to infer the source path from the bytecode path as - PyImport_ExecCodeModuleWithPathnames() does. - -Extension Modules ------------------ - -- Issue #6493: An issue in ctypes on Windows that caused structure bitfields of - type `ctypes.c_uint32` and width 32 to incorrectly be set has been fixed. - -- Issue #15194: Update libffi to the 3.0.11 release. - -IDLE ----- - -- Issue #13052: Fix IDLE crashing when replace string in Search/Replace dialog - ended with ``\``. Patch by Roger Serwy. - -Tools/Demos ------------ - -- Issue #15458: python-config gets a new option --configdir to print the $LIBPL - value. - -- Move importlib.test.benchmark to Tools/importbench. - -- Issue #12605: The gdb hooks for debugging CPython (within Tools/gdb) have been - enhanced to show information on more C frames relevant to CPython within the - "py-bt" and "py-bt-full" commands: - - * C frames that are waiting on the GIL - * C frames that are garbage-collecting - * C frames that are due to the invocation of a PyCFunction - -Documentation -------------- - -- Issue #15041: Update "see also" list in tkinter documentation. - -- Issue #15444: Use proper spelling for non-ASCII contributor names. Patch by - Serhiy Storchaka. - -- Issue #15295: Reorganize and rewrite the documentation on the import system. - -- Issue #15230: Clearly document some of the limitations of the runpy module and - nudge readers towards importlib when appropriate. - -- Issue #15053: Copy Python 3.3 import lock change notice to all relevant - functions in imp instead of just at the top of the relevant section. - -- Issue #15288: Link to the term "loader" in notes in pkgutil about how things - won't work as expected in Python 3.3 and mark the requisite functions as - "changed" since they will no longer work with modules directly imported by - import itself. - -- Issue #13557: Clarify effect of giving two different namespaces to `exec()` or - `execfile()`. - -- Issue #15250: Document that `filecmp.dircmp()` compares files shallowly. Patch - contributed by Chris Jerdonek. - -- Issue #15442: Expose the default list of directories ignored by - `filecmp.dircmp()` as a module attribute, and expand the list to more modern - values. - -Tests ------ - -- Issue #15467: Move helpers for `__sizeof__()` tests into test_support. Patch - by Serhiy Storchaka. - -- Issue #15320: Make iterating the list of tests thread-safe when running tests - in multiprocess mode. Patch by Chris Jerdonek. - -- Issue #15168: Move `importlib.test` to `test.test_importlib`. - -- Issue #15091: Reactivate a test on UNIX which was failing thanks to a - forgotten `importlib.invalidate_caches()` call. - -- Issue #15230: Adopted a more systematic approach in the runpy tests. - -- Issue #15300: Ensure the temporary test working directories are in the same - parent folder when running tests in multiprocess mode from a Python build. - Patch by Chris Jerdonek. - -- Issue #15284: Skip {send,recv}msg tests in test_socket when IPv6 is not - enabled. Patch by Brian Brazil. - -- Issue #15277: Fix a resource leak in support.py when IPv6 is disabled. Patch - by Brian Brazil. - -Build ------ - -- Issue #11715: Fix multiarch detection without having Debian development tools - (dpkg-dev) installed. - -- Issue #15037: Build OS X installers with local copy of ncurses 5.9 libraries - to avoid curses.unget_wch bug present in older versions of ncurses such as - those shipped with OS X. - -- Issue #15560: Fix building _sqlite3 extension on OS X with an SDK. Also, for - OS X installers, ensure consistent sqlite3 behavior and feature availability - by building a local copy of libsqlite3 rather than depending on the wide range - of versions supplied with various OS X releases. - -- Issue #8847: Disable COMDAT folding in Windows PGO builds. - -- Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. - -- Issue #16256: OS X installer now sets correct permissions for doc directory. - -- Issue #15431: Add _freeze_importlib project to regenerate importlib.h on - Windows. Patch by Kristj?n Valur J?nsson. - -- Issue #14197: For OS X framework builds, ensure links to the shared library - are created with the proper ABI suffix. - -- Issue #14330: For cross builds, don't use host python, use host search paths - for host compiler. - -- Issue #15235: Allow Berkley DB versions up to 5.3 to build the dbm module. - -- Issue #15268: Search curses.h in /usr/include/ncursesw. - - -What's New in Python 3.3.0 Beta 1? -================================== - -*Release date: 27-Jun-2012* - -Core and Builtins ------------------ - -- Fix a (most likely) very rare memory leak when calling main() and not being - able to decode a command-line argument. - -- Issue #14815: Use Py_ssize_t instead of long for the object hash, to - preserve all 64 bits of hash on Win64. - -- Issue #12268: File readline, readlines and read() or readall() methods - no longer lose data when an underlying read system call is interrupted. - IOError is no longer raised due to a read system call returning EINTR - from within these methods. - -- Issue #11626: Add _SizeT functions to stable ABI. - -- Issue #15142: Fix reference leak when deallocating instances of types - created using PyType_FromSpec(). - -- Issue #10053: Don't close FDs when FileIO.__init__ fails. Loosely based on - the work by Hirokazu Yamamoto. - -- Issue #15096: Removed support for ur'' as the raw notation isn't - compatible with Python 2.x's raw unicode strings. - -- Issue #13783: Generator objects now use the identifier APIs internally - -- Issue #14874: Restore charmap decoding speed to pre-PEP 393 levels. - Patch by Serhiy Storchaka. - -- Issue #15026: utf-16 encoding is now significantly faster (up to 10x). - Patch by Serhiy Storchaka. - -- Issue #11022: open() and io.TextIOWrapper are now calling - locale.getpreferredencoding(False) instead of locale.getpreferredencoding() - in text mode if the encoding is not specified. Don't change temporary the - locale encoding using locale.setlocale(), use the current locale encoding - instead of the user preferred encoding. - -- Issue #14673: Add Eric Snow's sys.implementation implementation. - -- Issue #15038: Optimize python Locks on Windows. - -Library -------- - -- Issue #12288: Consider '0' and '0.0' as valid initialvalue - for tkinter SimpleDialog. - -- Issue #15512: Add a __sizeof__ implementation for parser. - Patch by Serhiy Storchaka. - -- Issue #15469: Add a __sizeof__ implementation for deque objects. - Patch by Serhiy Storchaka. - -- Issue #15489: Add a __sizeof__ implementation for BytesIO objects. - Patch by Serhiy Storchaka. - -- Issue #15487: Add a __sizeof__ implementation for buffered I/O objects. - Patch by Serhiy Storchaka. - -- Issue #15514: Correct __sizeof__ support for cpu_set. - Patch by Serhiy Storchaka. - -- Issue #15177: Added dir_fd parameter to os.fwalk(). - -- Issue #15061: Re-implemented hmac.compare_digest() in C to prevent further - timing analysis and to support all buffer protocol aware objects as well as - ASCII only str instances safely. - -- Issue #15164: Change return value of platform.uname() from a - plain tuple to a collections.namedtuple. - -- Support Mageia Linux in the platform module. - -- Issue #11678: Support Arch linux in the platform module. - -- Issue #15118: Change return value of os.uname() and os.times() from - plain tuples to immutable iterable objects with named attributes - (structseq objects). - -- Speed up _decimal by another 10-15% by caching the thread local context - that was last accessed. In the pi benchmark (64-bit platform, prec=9), - _decimal is now only 1.5x slower than float. - -- Remove the packaging module, which is not ready for prime time. - -- Issue #15154: Add "dir_fd" parameter to os.rmdir, remove "rmdir" - parameter from os.remove / os.unlink. - -- Issue #4489: Add a shutil.rmtree that isn't susceptible to symlink attacks. - It is used automatically on platforms supporting the necessary os.openat() - and os.unlinkat() functions. Main code by Martin von L?wis. - -- Issue #15156: HTMLParser now uses the new "html.entities.html5" dictionary. - -- Issue #11113: add a new "html5" dictionary containing the named character - references defined by the HTML5 standard and the equivalent Unicode - character(s) to the html.entities module. - -- Issue #15114: the strict mode of HTMLParser and the HTMLParseError exception - are deprecated now that the parser is able to parse invalid markup. - -- Issue #3665: \u and \U escapes are now supported in unicode regular - expressions. Patch by Serhiy Storchaka. - -- Issue #15153: Added inspect.getgeneratorlocals to simplify white box - testing of generator state updates - -- Issue #13062: Added inspect.getclosurevars to simplify testing stateful - closures - -- Issue #11024: Fixes and additional tests for Time2Internaldate. - -- Issue #14626: Large refactoring of functions / parameters in the os module. - Many functions now support "dir_fd" and "follow_symlinks" parameters; - some also support accepting an open file descriptor in place of a path - string. Added os.support_* collections as LBYL helpers. Removed many - functions only previously seen in 3.3 alpha releases (often starting with - "f" or "l", or ending with "at"). Originally suggested by Serhiy Storchaka; - implemented by Larry Hastings. - -- Issue #15008: Implement PEP 362 "Signature Objects". - Patch by Yury Selivanov. - -- Issue: #15138: base64.urlsafe_{en,de}code() are now 3-4x faster. - -- Issue #444582: Add shutil.which, for finding programs on the system path. - Original patch by Erik Demaine, with later iterations by Jan Killian - and Brian Curtin. - -- Issue #14837: SSL errors now have ``library`` and ``reason`` attributes - describing precisely what happened and in which OpenSSL submodule. The - str() of a SSLError is also enhanced accordingly. - -- Issue #9527: datetime.astimezone() method will now supply a class - timezone instance corresponding to the system local timezone when - called with no arguments. - -- Issue #14653: email.utils.mktime_tz() no longer relies on system - mktime() when timezone offest is supplied. - -- Issue #14684: zlib.compressobj() and zlib.decompressobj() now support the use - of predefined compression dictionaries. Original patch by Sam Rushing. - -- Fix GzipFile's handling of filenames given as bytes objects. - -- Issue #14772: Return destination values from some shutil functions. - -- Issue #15064: Implement context manager protocol for multiprocessing types - -- Issue #15101: Make pool finalizer avoid joining current thread. - -- Issue #14657: The frozen instance of importlib used for bootstrap is now - also the module imported as importlib._bootstrap. - -- Issue #14055: Add __sizeof__ support to _elementtree. - -- Issue #15054: A bug in tokenize.tokenize that caused string literals - with 'b' prefixes to be incorrectly tokenized has been fixed. - Patch by Serhiy Storchaka. - -- Issue #15006: Allow equality comparison between naive and aware - time or datetime objects. - -- Issue #15036: Mailbox no longer throws an error if a flush is done - between operations when removing or changing multiple items in mbox, - MMDF, or Babyl mailboxes. - -- Issue #14059: Implement multiprocessing.Barrier. - -- Issue #15061: The inappropriately named hmac.secure_compare has been - renamed to hmac.compare_digest, restricted to operating on bytes inputs - only and had its documentation updated to more accurately reflect both its - intent and its limitations - -- Issue #13841: Make child processes exit using sys.exit() on Windows. - -- Issue #14936: curses_panel was converted to PEP 3121 and PEP 384 API. - Patch by Robin Schreiber. - -- Issue #1667546: On platforms supporting tm_zone and tm_gmtoff fields - in struct tm, time.struct_time objects returned by time.gmtime(), - time.localtime() and time.strptime() functions now have tm_zone and - tm_gmtoff attributes. Original patch by Paul Boddie. - -- Rename adjusted attribute to adjustable in time.get_clock_info() result. - -- Issue #3518: Remove references to non-existent BaseManager.from_address() - method. - -- Issue #13857: Added textwrap.indent() function (initial patch by Ezra - Berch) - -- Issue #2736: Added datetime.timestamp() method. - -- Issue #13854: Make multiprocessing properly handle non-integer - non-string argument to SystemExit. - -- Issue #12157: Make pool.map() empty iterables correctly. Initial - patch by mouad. - -- Issue #11823: disassembly now shows argument counts on calls with keyword args. - -- Issue #14711: os.stat_float_times() has been deprecated. - -- LZMAFile now accepts the modes "rb"/"wb"/"ab" as synonyms of "r"/"w"/"a". - -- The bz2 and lzma modules now each contain an open() function, allowing - compressed files to readily be opened in text mode as well as binary mode. - -- BZ2File.__init__() and LZMAFile.__init__() now accept a file object as their - first argument, rather than requiring a separate "fileobj" argument. - -- gzip.open() now accepts file objects as well as filenames. - -- Issue #14992: os.makedirs(path, exist_ok=True) would raise an OSError - when the path existed and had the S_ISGID mode bit set when it was - not explicitly asked for. This is no longer an exception as mkdir - cannot control if the OS sets that bit for it or not. - -- Issue #14989: Make the CGI enable option to http.server available via command - line. - -- Issue #14987: Add a missing import statement to inspect. - -- Issue #1079: email.header.decode_header now correctly parses all the examples - in RFC2047. There is a necessary visible behavior change: the leading and/or - trailing whitespace on ASCII parts is now preserved. - -- Issue #14969: Better handling of exception chaining in contextlib.ExitStack - -- Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative - algorithm (Patch by Alon Horev) - -- Issue #14785: Add sys._debugmallocstats() to help debug low-level memory - allocation issues - -- Issue #14443: Ensure that .py files are byte-compiled with the correct Python - executable within bdist_rpm even on older versions of RPM - -C-API ------ - -- Issue #15146: Add PyType_FromSpecWithBases. Patch by Robin Schreiber. - -- Issue #15042: Add PyState_AddModule and PyState_RemoveModule. Add version - guard for Py_LIMITED_API additions. Patch by Robin Schreiber. - -- Issue #13783: Inadvertent additions to the public C API in the PEP 380 - implementation have either been removed or marked as private interfaces. - -Extension Modules ------------------ - -- Issue #15000: Support the "unique" x32 architecture in _posixsubprocess.c. - -IDLE ----- - -- Issue #9803: Don't close IDLE on saving if breakpoint is open. - Patch by Roger Serwy. - -- Issue #14962: Update text coloring in IDLE shell window after changing - options. Patch by Roger Serwy. - -Documentation -------------- - -- Issue #15176: Clarified behavior, documentation, and implementation - of os.listdir(). - -- Issue #14982: Document that pkgutil's iteration functions require the - non-standard iter_modules() method to be defined by an importer (something - the importlib importers do not define). - -- Issue #15081: Document PyState_FindModule. - Patch by Robin Schreiber. - -- Issue #14814: Added first draft of ipaddress module API reference - -Tests ------ - -- Issue #15187: Bugfix: remove temporary directories test_shutil was leaving - behind. - -- Issue #14769: test_capi now has SkipitemTest, which cleverly checks - for "parity" between PyArg_ParseTuple() and the Python/getargs.c static - function skipitem() for all possible "format units". - -- test_nntplib now tolerates being run from behind NNTP gateways that add - "X-Antivirus" headers to articles - -- Issue #15043: test_gdb is now skipped entirely if gdb security settings - block loading of the gdb hooks - -- Issue #14963: Add test cases for exception handling behaviour - in contextlib.ExitStack (Initial patch by Alon Horev) - -Build ------ - -- Issue #13590: Improve support for OS X Xcode 4: - * Try to avoid building Python or extension modules with problematic - llvm-gcc compiler. - * Since Xcode 4 removes ppc support, extension module builds now - check for ppc compiler support and automatically remove ppc and - ppc64 archs when not available. - * Since Xcode 4 no longer install SDKs in default locations, - extension module builds now revert to using installed headers - and libs if the SDK used to build the interpreter is not - available. - * Update ./configure to use better defaults for universal builds; - in particular, --enable-universalsdk=yes uses the Xcode default - SDK and --with-universal-archs now defaults to "intel" if ppc - not available. - -- Issue #14225: Fix Unicode support for curses (#12567) on OS X - -- Issue #14928: Fix importlib bootstrap issues by using a custom executable - (Modules/_freeze_importlib) to build Python/importlib.h. - - -What's New in Python 3.3.0 Alpha 4? -=================================== - -*Release date: 31-May-2012* - -Core and Builtins ------------------ - -- Issue #14835: Make plistlib output empty arrays & dicts like OS X. - Patch by Sidney San Mart?n. - -- Issue #14744: Use the new _PyUnicodeWriter internal API to speed up - str%args and str.format(args). - -- Issue #14930: Make memoryview objects weakrefable. - -- Issue #14775: Fix a potential quadratic dict build-up due to the garbage - collector repeatedly trying to untrack dicts. - -- Issue #14857: fix regression in references to PEP 3135 implicit __class__ - closure variable (Reopens issue #12370) - -- Issue #14712 (PEP 405): Virtual environments. Implemented by Vinay Sajip. - -- Issue #14660 (PEP 420): Namespace packages. Implemented by Eric Smith. - -- Issue #14494: Fix __future__.py and its documentation to note that - absolute imports are the default behavior in 3.0 instead of 2.7. - Patch by Sven Marnach. - -- Issue #9260: A finer-grained import lock. Most of the import sequence - now uses per-module locks rather than the global import lock, eliminating - well-known issues with threads and imports. - -- Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. - Patch by Serhiy Storchaka. - -- asdl_seq and asdl_int_seq are now Py_ssize_t sized. - -- Issue #14133 (PEP 415): Implement suppression of __context__ display with an - attribute on BaseException. This replaces the original mechanism of PEP 409. - -- Issue #14417: Mutating a dict during lookup now restarts the lookup instead - of raising a RuntimeError (undoes issue #14205). - -- Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy - Storchaka. - -- Issue #14700: Fix two broken and undefined-behaviour-inducing overflow checks - in old-style string formatting. - -Library -------- - -- Issue #14690: Use monotonic clock instead of system clock in the sched, - subprocess and trace modules. - -- Issue #14443: Tell rpmbuild to use the correct version of Python in - bdist_rpm. Initial patch by Ross Lagerwall. - -- Issue #12515: email now registers a defect if it gets to EOF while parsing - a MIME part without seeing the closing MIME boundary. - -- Issue #1672568: email now always decodes base64 payloads, adding padding and - ignoring non-base64-alphabet characters if needed, and registering defects - for any such problems. - -- Issue #14925: email now registers a defect when the parser decides that there - is a missing header/body separator line. MalformedHeaderDefect, which the - existing code would never actually generate, is deprecated. - -- Issue #10365: File open dialog now works instead of crashing even when - the parent window is closed before the dialog. Patch by Roger Serwy. - -- Issue #8739: Updated smtpd to support RFC 5321, and added support for the - RFC 1870 SIZE extension. - -- Issue #665194: Added a localtime function to email.utils to provide an - aware local datetime for use in setting Date headers. - -- Issue #12586: Added new provisional policies that implement convenient - unicode support for email headers. See What's New for details. - -- Issue #14731: Refactored email Policy framework to support full backward - compatibility with Python 3.2 by default yet allow for the introduction of - new features through new policies. Note that Policy.must_be_7bit is renamed - to cte_type. - -- Issue #14876: Use user-selected font for highlight configuration. - -- Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. - Have ascii characters in help. - -- Issue #14548: Make multiprocessing finalizers check pid before - running to cope with possibility of gc running just after fork. - -- Issue #14036: Add an additional check to validate that port in urlparse does - not go in illegal range and returns None. - -- Issue #14862: Add missing names to os.__all__ - -- Issue #14875: Use float('inf') instead of float('1e66666') in the json module. - -- Issue #13585: Added contextlib.ExitStack - -- PEP 3144, Issue #14814: Added the ipaddress module - -- Issue #14426: Correct the Date format in Expires attribute of Set-Cookie - Header in Cookie.py. - -- Issue #14588: The types module now provide new_class() and prepare_class() - functions to support PEP 3115 compliant dynamic class creation. Patch by - Daniel Urban and Nick Coghlan. - -- Issue #13152: Allow to specify a custom tabsize for expanding tabs in - textwrap. Patch by John Feuerstein. - -- Issue #14721: Send the correct 'Content-length: 0' header when the body is an - empty string ''. Initial Patch contributed by Arve Knudsen. - -- Issue #14072: Fix parsing of 'tel' URIs in urlparse by making the check for - ports stricter. - -- Issue #9374: Generic parsing of query and fragment portions of url for any - scheme. Supported both by RFC3986 and RFC2396. - -- Issue #14798: Fix the functions in pyclbr to raise an ImportError - when the first part of a dotted name is not a package. Patch by - Xavier de Gaye. - -- Issue #12098: multiprocessing on Windows now starts child processes - using the same sys.flags as the current process. Initial patch by - Sergey Mezentsev. - -- Issue #13031: Small speed-up for tarfile when unzipping tarfiles. - Patch by Justin Peel. - -- Issue #14780: urllib.request.urlopen() now has a ``cadefault`` argument - to use the default certificate store. Initial patch by James Oakley. - -- Issue #14829: Fix bisect and range() indexing with large indices - (>= 2 ** 32) under 64-bit Windows. - -- Issue #14732: The _csv module now uses PEP 3121 module initialization. - Patch by Robin Schreiber. - -- Issue #14809: Add HTTP status codes introduced by RFC 6585 to http.server - and http.client. Patch by EungJun Yi. - -- Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when - accessing the Tk clipboard. Modify clipboad_get() to first request type - UTF8_STRING when no specific type is requested in an X11 windowing - environment, falling back to the current default type STRING if that fails. - Original patch by Thomas Kluyver. - -- Issue #14773: Fix os.fwalk() failing on dangling symlinks. - -- Issue #12541: Be lenient with quotes around Realm field of HTTP Basic - Authentation in urllib2. - -- Issue #14807: move undocumented tarfile.filemode() to stat.filemode() and add - doc entry. Add tarfile.filemode alias with deprecation warning. - -- Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. - -- Issue #14532: Add a secure_compare() helper to the hmac module, to mitigate - timing attacks. Patch by Jon Oberheide. - -- Add importlib.util.resolve_name(). - -- Issue #14366: Support lzma compression in zip files. - Patch by Serhiy Storchaka. - -- Issue #13959: Introduce importlib.find_loader() and document - imp.find_module/load_module as deprecated. - -- Issue #14082: shutil.copy2() now copies extended attributes, if possible. - Patch by Hynek Schlawack. - -- Issue #13959: Make importlib.abc.FileLoader.load_module()/get_filename() and - importlib.machinery.ExtensionFileLoader.load_module() have their single - argument be optional. Allows for the replacement (and thus deprecation) of - imp.load_source()/load_package()/load_compiled(). - -- Issue #13959: imp.get_suffixes() has been deprecated in favour of the new - attributes on importlib.machinery: SOURCE_SUFFIXES, DEBUG_BYTECODE_SUFFIXES, - OPTIMIZED_BYTECODE_SUFFIXES, BYTECODE_SUFFIXES, and EXTENSION_SUFFIXES. This - led to an indirect deprecation of inspect.getmoduleinfo(). - -- Issue #14662: Prevent shutil failures on OS X when destination does not - support chflag operations. Patch by Hynek Schlawack. - -- Issue #14157: Fix time.strptime failing without a year on February 29th. - Patch by Hynek Schlawack. - -- Issue #14753: Make multiprocessing's handling of negative timeouts - the same as it was in Python 3.2. - -- Issue #14583: Fix importlib bug when a package's __init__.py would first - import one of its modules then raise an error. - -- Issue #14741: Fix missing support for Ellipsis ('...') in parser module. - -- Issue #14697: Fix missing support for set displays and set comprehensions in - parser module. - -- Issue #14701: Fix missing support for 'raise ... from' in parser module. - -- Add support for timeouts to the acquire() methods of - multiprocessing's lock/semaphore/condition proxies. - -- Issue #13989: Add support for text mode to gzip.open(). - -- Issue #14127: The os.stat() result object now provides three additional - fields: st_ctime_ns, st_mtime_ns, and st_atime_ns, providing those times as an - integer with nanosecond resolution. The functions os.utime(), os.lutimes(), - and os.futimes() now accept a new parameter, ns, which accepts mtime and atime - as integers with nanosecond resolution. - -- Issue #14127 and #10148: shutil.copystat now preserves exact mtime and atime - on filesystems providing nanosecond resolution. - -IDLE ----- - -- Issue #14958: Change IDLE systax highlighting to recognize all string and - byte literals supported in Python 3.3. - -- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. - -- Issue #14929: Stop IDLE 3.x from closing on Unicode decode errors when - grepping. Patch by Roger Serwy. - -- Issue #12510: Attempting to get invalid tooltip no longer closes IDLE. - Other tooltipss have been corrected or improved and the number of tests - has been tripled. Original patch by Roger Serwy. - -Tools/Demos ------------ - -- Issue #14695: Bring Tools/parser/unparse.py support up to date with - the Python 3.3 Grammar. - -Build ------ - -- Issue #14472: Update .gitignore. Patch by Matej Cepl. - -- Upgrade Windows library versions: bzip 1.0.6, OpenSSL 1.0.1c. - -- Issue #14693: Under non-Windows platforms, hashlib's fallback modules are - always compiled, even if OpenSSL is present at build time. - -- Issue #13210: Windows build now uses VS2010, ported from VS2008. - -C-API ------ - -- Issue #14705: The PyArg_Parse() family of functions now support the 'p' format - unit, which accepts a "boolean predicate" argument. It converts any Python - value into an integer--0 if it is "false", and 1 otherwise. - -Documentation -------------- - -- Issue #14863: Update the documentation of os.fdopen() to reflect the - fact that it's only a thin wrapper around open() anymore. - -- Issue #14588: The language reference now accurately documents the Python 3 - class definition process. Patch by Nick Coghlan. - -- Issue #14943: Correct a default argument value for winreg.OpenKey - and correctly list the argument names in the function's explanation. - - -What's New in Python 3.3.0 Alpha 3? -=================================== - -*Release date: 01-May-2012* - -Core and Builtins ------------------ - -- Issue #14699: Fix calling the classmethod descriptor directly. - -- Issue #14433: Prevent msvcrt crash in interactive prompt when stdin is closed. - -- Issue #14521: Make result of float('nan') and float('-nan') more consistent - across platforms. - -- Issue #14646: __import__() sets __loader__ if the loader did not. - -- Issue #14605: No longer have implicit entries in sys.meta_path. If - sys.meta_path is found to be empty, raise ImportWarning. - -- Issue #14605: No longer have implicit entries in sys.path_hooks. If - sys.path_hooks is found to be empty, a warning will be raised. None is now - inserted into sys.path_importer_cache if no finder was discovered. This also - means imp.NullImporter is no longer implicitly used. - -- Issue #13903: Implement PEP 412. Individual dictionary instances can now share - their keys with other dictionaries. Classes take advantage of this to share - their instance dictionary keys for improved memory and performance. - -- Issue #11603 (again): Setting __repr__ to __str__ now raises a RuntimeError - when repr() or str() is called on such an object. - -- Issue #14658: Fix binding a special method to a builtin implementation of a - special method with a different name. - -- Issue #14630: Fix a memory access bug for instances of a subclass of int - with value 0. - -- Issue #14339: Speed improvements to bin, oct and hex functions. Patch by - Serhiy Storchaka. - -- Issue #14385: It is now possible to use a custom type for the __builtins__ - namespace, instead of a dict. It can be used for sandboxing for example. - Raise also a NameError instead of ImportError if __build_class__ name if not - found in __builtins__. - -- Issue #12599: Be more strict in accepting None compared to a false-like - object for importlib.util.module_for_loader and - importlib.machinery.PathFinder. - -- Issue #14612: Fix jumping around with blocks by setting f_lineno. - -- Issue #14592: Attempting a relative import w/o __package__ or __name__ set in - globals raises a KeyError. - -- Issue #14607: Fix keyword-only arguments which started with ``__``. - -- Issue #10854: The ImportError raised when an extension module on Windows - fails to import now uses the new path and name attributes from - Issue #1559549. - -- Issue #13889: Check and (if necessary) set FPU control word before calling - any of the dtoa.c string <-> float conversion functions, on MSVC builds of - Python. This fixes issues when embedding Python in a Delphi app. - -- __import__() now matches PEP 328 and documentation by defaulting 'index' to 0 - instead of -1 and removing support for negative values. - -- Issue #2377: Make importlib the implementation of __import__(). - -- Issue #1559549: ImportError now has 'name' and 'path' attributes that are set - using keyword arguments to its constructor. They are currently not set by - import as they are meant for use by importlib. - -- Issue #14474: Save and restore exception state in thread.start_new_thread() - while writing error message if the thread leaves a unhandled exception. - -- Issue #13019: Fix potential reference leaks in bytearray.extend(). Patch - by Suman Saha. - -Library -------- - -- Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. - -- Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. - -- Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running - step. Patch by Xavier de Gaye. - -- Issue #14696: Fix parser module to understand 'nonlocal' declarations. - -- Issue #10941: Fix imaplib.Internaldate2tuple to produce correct result near - the DST transition. Patch by Joe Peterson. - -- Issue #9154: Fix parser module to understand function annotations. - -- Issue #6085: In http.server.py SimpleHTTPServer.address_string returns the - client ip address instead client hostname. Patch by Charles-Fran?ois Natali. - -- Issue #14309: Deprecate time.clock(), use time.perf_counter() or - time.process_time() instead. - -- Issue #14428: Implement the PEP 418. Add time.get_clock_info(), - time.perf_counter() and time.process_time() functions, and rename - time.steady() to time.monotonic(). - -- Issue #14646: importlib.util.module_for_loader() now sets __loader__ and - __package__ (when possible). - -- Issue #14664: It is now possible to use @unittest.skip{If,Unless} on a - test class that doesn't inherit from TestCase (i.e. a mixin). - -- Issue #4892: multiprocessing Connections can now be transferred over - multiprocessing Connections. Patch by Richard Oudkerk (sbt). - -- Issue #14160: TarFile.extractfile() failed to resolve symbolic links when - the links were not located in an archive subdirectory. - -- Issue #14638: pydoc now treats non-string __name__ values as if they - were missing, instead of raising an error. - -- Issue #13684: Fix httplib tunnel issue of infinite loops for certain sites - which send EOF without trailing \r\n. - -- Issue #14605: Add importlib.abc.FileLoader, importlib.machinery.(FileFinder, - SourceFileLoader, SourcelessFileLoader, ExtensionFileLoader). - -- Issue #13959: imp.cache_from_source()/source_from_cache() now follow - os.path.join()/split() semantics for path manipulation instead of its prior, - custom semantics of caring the right-most path separator forward in path - joining. - -- Issue #2193: Allow ":" character in Cookie NAME values. - -- Issue #14629: tokenizer.detect_encoding will specify the filename in the - SyntaxError exception if found at readline.__self__.name. - -- Issue #14629: Raise SyntaxError in tokenizer.detect_encoding if the - first two lines have non-UTF-8 characters without an encoding declaration. - -- Issue #14308: Fix an exception when a "dummy" thread is in the threading - module's active list after a fork(). - -- Issue #11750: The Windows API functions scattered in the _subprocess and - _multiprocessing.win32 modules now live in a single module "_winapi". - Patch by sbt. - -- Issue #14087: multiprocessing: add Condition.wait_for(). Patch by sbt. - -- Issue #14538: HTMLParser can now parse correctly start tags that contain - a bare '/'. - -- Issue #14452: SysLogHandler no longer inserts a UTF-8 BOM into the message. - -- Issue #14386: Expose the dict_proxy internal type as types.MappingProxyType. - -- Issue #13959: Make imp.reload() always use a module's __loader__ to perform - the reload. - -- Issue #13959: Add imp.py and rename the built-in module to _imp, allowing for - re-implementing parts of the module in pure Python. - -- Issue #13496: Fix potential overflow in bisect.bisect algorithm when applied - to a collection of size > sys.maxsize / 2. - -- Have importlib take advantage of ImportError's new 'name' and 'path' - attributes. - -- Issue #14399: zipfile now recognizes that the archive has been modified even - if only the comment is changed. In addition, the TypeError that results from - trying to set a non-binary value as a comment is now raised at the time - the comment is set rather than at the time the zipfile is written. - -- trace.CoverageResults.is_ignored_filename() now ignores any name that starts - with "<" and ends with ">" instead of special-casing "" and - ". - -- Issue #14295: Add unittest.mock - -- Issue #7652: Add --with-system-libmpdec option to configure for linking - the _decimal module against an installed libmpdec. - -- Issue #14380: MIMEText now defaults to utf-8 when passed non-ASCII unicode - with no charset specified. - -- Issue #10340: asyncore - properly handle EINVAL in dispatcher constructor on - OSX; avoid to call handle_connect in case of a disconnected socket which - was not meant to connect. - -- Issue #14204: The ssl module now has support for the Next Protocol - Negotiation extension, if available in the underlying OpenSSL library. - Patch by Colin Marc. - -- Issue #3035: Unused functions from tkinter are marked as pending deprecated. - -- Issue #12757: Fix the skipping of doctests when python is run with -OO so - that it works in unittest's verbose mode as well as non-verbose mode. - -- Issue #7652: Integrate the decimal floating point libmpdec library to speed - up the decimal module. Performance gains of the new C implementation are - between 10x and 100x, depending on the application. - -- Issue #14269: SMTPD now conforms to the RFC and requires a HELO command - before MAIL, RCPT, or DATA. - -- Issue #13694: asynchronous connect in asyncore.dispatcher does not set addr - attribute. - -- Issue #14344: fixed the repr of email.policy objects. - -- Issue #11686: Added missing entries to email package __all__ lists - (mostly the new Bytes classes). - -- Issue #14335: multiprocessing's custom Pickler subclass now inherits from - the C-accelerated implementation. Patch by sbt. - -- Issue #10484: Fix the CGIHTTPServer's PATH_INFO handling problem. - -- Issue #11199: Fix the with urllib which hangs on particular ftp urls. - -- Improve the memory utilization and speed of functools.lru_cache. - -- Issue #14222: Use the new time.steady() function instead of time.time() for - timeout in queue and threading modules to not be affected of system time - update. - -- Issue #13248: Remove lib2to3.pytree.Base.get_prefix/set_prefix. - -- Issue #14234: CVE-2012-0876: Randomize hashes of xml attributes in the hash - table internal to the pyexpat module's copy of the expat library to avoid a - denial of service due to hash collisions. Patch by David Malcolm with some - modifications by the expat project. - -- Issue #12818: format address no longer needlessly \ escapes ()s in names when - the name ends up being quoted. - -- Issue #14062: BytesGenerator now correctly folds Header objects, - including using linesep when folding. - -- Issue #13839: When invoked on the command-line, the pstats module now - accepts several filenames of profile stat files and merges them all. - Patch by Matt Joiner. - -- Issue #14291: Email now defaults to utf-8 for non-ASCII unicode headers - instead of raising an error. This fixes a regression relative to 2.7. - -- Issue #989712: Support using Tk without a mainloop. - -- Issue #3835: Refuse to use unthreaded Tcl in threaded Python. - -- Issue #2843: Add new Tk API to Tkinter. - -- Issue #14184: Increase the default stack size for secondary threads on - Mac OS X to avoid interpreter crashes when using threads on 10.7. - -- Issue #14180: datetime.date.fromtimestamp(), - datetime.datetime.fromtimestamp() and datetime.datetime.utcfromtimestamp() - now raise an OSError instead of ValueError if localtime() or gmtime() failed. - -- Issue #14180: time.ctime(), gmtime(), time.localtime(), - datetime.date.fromtimestamp(), datetime.datetime.fromtimestamp() and - datetime.datetime.utcfromtimestamp() now raises an OverflowError, instead of - a ValueError, if the timestamp does not fit in time_t. - -- Issue #14180: datetime.datetime.fromtimestamp() and - datetime.datetime.utcfromtimestamp() now round microseconds towards zero - instead of rounding to nearest with ties going away from zero. - -- Issue #10543: Fix unittest test discovery with Jython bytecode files. - -- Issue #1178863: Separate initialisation from setting when initializing - Tkinter.Variables; harmonize exceptions to ValueError; only delete variables - that have not been deleted; assert that variable names are strings. - -- Issue #14104: Implement time.monotonic() on Mac OS X, patch written by - Nicholas Riley. - -- Issue #13394: the aifc module now uses warnings.warn() to signal warnings. - -- Issue #14252: Fix subprocess.Popen.terminate() to not raise an error under - Windows when the child process has already exited. - -- Issue #14223: curses.addch() is no more limited to the range 0-255 when the - Python curses is not linked to libncursesw. It was a regression introduced - in Python 3.3a1. - -- Issue #14168: Check for presence of Element._attrs in minidom before - accessing it. - -- Issue #12328: Fix multiprocessing's use of overlapped I/O on Windows. - Also, add a multiprocessing.connection.wait(rlist, timeout=None) function - for polling multiple objects at once. Patch by sbt. - -- Issue #14007: Accept incomplete TreeBuilder objects (missing start, end, - data or close method) for the Python implementation as well. - Drop the no-op TreeBuilder().xml() method from the C implementation. - -- Issue #14210: pdb now has tab-completion not only for command names, but - also for their arguments, wherever possible. - -- Issue #14310: Sockets can now be with other processes on Windows using - the api socket.socket.share() and socket.fromshare(). - -- Issue #10576: The gc module now has a 'callbacks' member that will get - called when garbage collection takes place. - -Build ------ - -- Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. - -- Issue #14387: Do not include accu.h from Python.h. - -- Issue #14359: Only use O_CLOEXEC in _posixmodule.c if it is defined. - Based on patch from Herv? Coatanhay. - -- Issue #14321: Do not run pgen during the build if files are up to date. - -Documentation -------------- - -- Issue #14034: added the argparse tutorial. - -- Issue #14324: Fix configure tests for cross builds. - -- Issue #14327: Call AC_CANONICAL_HOST in configure.ac and check in - config.{guess,sub}. Don't use uname calls for cross builds. - -Extension Modules ------------------ - -- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and - ctypes.c_float that caused an incorrect exception to be returned in the - case of overflow has been fixed. - -- Issue #14212: The re module didn't retain a reference to buffers it was - scanning, resulting in segfaults. - -- Issue #14259: The finditer() method of re objects did not take any - keyword arguments, contrary to the documentation. - -- Issue #10142: Support for SEEK_HOLE/SEEK_DATA (for example, under ZFS). - -Tests ------ - -- Issue #14442: Add missing errno import in test_smtplib. - -- Issue #8315: (partial fix) python -m unittest test.test_email now works. - - -What's New in Python 3.3.0 Alpha 1? -=================================== - -*Release date: 05-Mar-2012* - -Core and Builtins ------------------ - -- Issue #14172: Fix reference leak when marshalling a buffer-like object - (other than a bytes object). - -- Issue #13521: dict.setdefault() now does only one lookup for the given key, - making it "atomic" for many purposes. Patch by Filip Gruszczy?ski. - -- PEP 409, Issue #6210: "raise X from None" is now supported as a means of - suppressing the display of the chained exception context. The chained - context still remains available as the __context__ attribute. - -- Issue #10181: New memoryview implementation fixes multiple ownership - and lifetime issues of dynamically allocated Py_buffer members (#9990) - as well as crashes (#8305, #7433). Many new features have been added - (See whatsnew/3.3), and the documentation has been updated extensively. - The ndarray test object from _testbuffer.c implements all aspects of - PEP-3118, so further development towards the complete implementation - of the PEP can proceed in a test-driven manner. - - Thanks to Nick Coghlan, Antoine Pitrou and Pauli Virtanen for review - and many ideas. - -- Issue #12834: Fix incorrect results of memoryview.tobytes() for - non-contiguous arrays. - -- Issue #5231: Introduce memoryview.cast() method that allows changing - format and shape without making a copy of the underlying memory. - -- Issue #14084: Fix a file descriptor leak when importing a module with a - bad encoding. - -- Upgrade Unicode data to Unicode 6.1. - -- Issue #14040: Remove rarely used file name suffixes for C extensions - (under POSIX mainly). - -- Issue #14051: Allow arbitrary attributes to be set of classmethod and - staticmethod. - -- Issue #13703: oCERT-2011-003: Randomize hashes of str and bytes to protect - against denial of service attacks due to hash collisions within the dict and - set types. Patch by David Malcolm, based on work by Victor Stinner. - -- Issue #13020: Fix a reference leak when allocating a structsequence object - fails. Patch by Suman Saha. - -- Issue #13908: Ready types returned from PyType_FromSpec. - -- Issue #11235: Fix OverflowError when trying to import a source file whose - modification time doesn't fit in a 32-bit timestamp. - -- Issue #12705: A SyntaxError exception is now raised when attempting to - compile multiple statements as a single interactive statement. - -- Fix the builtin module initialization code to store the init function for - future reinitialization. - -- Issue #8052: The posix subprocess module would take a long time closing - all possible file descriptors in the child process rather than just open - file descriptors. It now closes only the open fds if possible for the - default close_fds=True behavior. - -- Issue #13629: Renumber the tokens in token.h so that they match the indexes - into _PyParser_TokenNames. - -- Issue #13752: Add a casefold() method to str. - -- Issue #13761: Add a "flush" keyword argument to the print() function, - used to ensure flushing the output stream. - -- Issue #13645: pyc files now contain the size of the corresponding source - code, to avoid timestamp collisions (especially on filesystems with a low - timestamp resolution) when checking for freshness of the bytecode. - -- PEP 380, Issue #11682: Add "yield from " to support easy delegation to - subgenerators (initial patch by Greg Ewing, integration into 3.3 by - Renaud Blanch, Ryan Kelly, Zbigniew J?drzejewski-Szmek and Nick Coghlan) - -- Issue #13748: Raw bytes literals can now be written with the ``rb`` prefix - as well as ``br``. - -- Issue #12736: Use full unicode case mappings for upper, lower, and title case. - -- Issue #12760: Add a create mode to open(). Patch by David Townshend. - -- Issue #13738: Simplify implementation of bytes.lower() and bytes.upper(). - -- Issue #13577: Built-in methods and functions now have a __qualname__. - Patch by sbt. - -- Issue #6695: Full garbage collection runs now clear the freelist of set - objects. Initial patch by Matthias Troffaes. - -- Fix OSError.__init__ and OSError.__new__ so that each of them can be - overriden and take additional arguments (followup to issue #12555). - -- Fix the fix for issue #12149: it was incorrect, although it had the side - effect of appearing to resolve the issue. Thanks to Mark Shannon for - noticing. - -- Issue #13505: Pickle bytes objects in a way that is compatible with - Python 2 when using protocols <= 2. - -- Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix - given by Campbell Barton). - -- Issue #13503: Use a more efficient reduction format for bytearrays with - pickle protocol >= 3. The old reduction format is kept with older protocols - in order to allow unpickling under Python 2. Patch by Irmen de Jong. - -- Issue #7111: Python can now be run without a stdin, stdout or stderr - stream. It was already the case with Python 2. However, the corresponding - sys module entries are now set to None (instead of an unusable file object). - -- Issue #11849: Ensure that free()d memory arenas are really released - on POSIX systems supporting anonymous memory mappings. Patch by - Charles-Fran?ois Natali. - -- PEP 3155 / issue #13448: Qualified name for classes and functions. - -- Issue #13436: Fix a bogus error message when an AST object was passed - an invalid integer value. - -- Issue #13411: memoryview objects are now hashable when the underlying - object is hashable. - -- Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER - to allow compiling extension modules with -Wswitch-enum on gcc. - Initial patch by Floris Bruynooghe. - -- Issue #10227: Add an allocation cache for a single slice object. Patch by - Stefan Behnel. - -- Issue #13393: BufferedReader.read1() now asks the full requested size to - the raw stream instead of limiting itself to the buffer size. - -- Issue #13392: Writing a pyc file should now be atomic under Windows as well. - -- Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder - already accepts them). - -- Issue #13389: Full garbage collection passes now clear the freelists for - list and dict objects. They already cleared other freelists in the - interpreter. - -- Issue #13327: Remove the need for an explicit None as the second argument - to os.utime, os.lutimes, os.futimes, os.futimens, os.futimesat, in - order to update to the current time. Also added keyword argument - handling to os.utimensat in order to remove the need for explicit None. - -- Issue #13350: Simplify some C code by replacing most usages of - PyUnicode_Format by PyUnicode_FromFormat. - -- Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode - error handler in interactive mode (when calling into PyOS_Readline()). - -- Issue #9896: Add start, stop, and step attributes to range objects. - -- Issue #13343: Fix a SystemError when a lambda expression uses a global - variable in the default value of a keyword-only argument: ``lambda *, - arg=GLOBAL_NAME: None`` - -- Issue #12797: Added custom opener parameter to builtin open() and - FileIO.open(). - -- Issue #10519: Avoid unnecessary recursive function calls in - setobject.c. - -- Issue #10363: Deallocate global locks in Py_Finalize(). - -- Issue #13018: Fix reference leaks in error paths in dictobject.c. - Patch by Suman Saha. - -- Issue #13201: Define '==' and '!=' to compare range objects based on - the sequence of values they define (instead of comparing based on - object identity). - -- Issue #1294232: In a few cases involving metaclass inheritance, the - interpreter would sometimes invoke the wrong metaclass when building a new - class object. These cases now behave correctly. Patch by Daniel Urban. - -- Issue #12753: Add support for Unicode name aliases and named sequences. - Both ``unicodedata.lookup()`` and '\N{...}' now resolve aliases, - and ``unicodedata.lookup()`` resolves named sequences too. - -- Issue #12170: The count(), find(), rfind(), index() and rindex() methods - of bytes and bytearray objects now accept an integer between 0 and 255 - as their first argument. Patch by Petri Lehtinen. - -- Issue #12604: VTRACE macro expanded to no-op in _sre.c to avoid compiler - warnings. Patch by Josh Triplett and Petri Lehtinen. - -- Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore - error handlers on all Windows versions. The MBCS codec is now supporting all - error handlers, instead of only replace to encode and ignore to decode. - -- Issue #13188: When called without an explicit traceback argument, - generator.throw() now gets the traceback from the passed exception's - ``__traceback__`` attribute. Patch by Petri Lehtinen. - -- Issue #13146: Writing a pyc file is now atomic under POSIX. - -- Issue #7833: Extension modules built using distutils on Windows will no - longer include a "manifest" to prevent them failing at import time in some - embedded situations. - -- PEP 3151 / issue #12555: reworking the OS and IO exception hierarchy. - -- Add internal API for static strings (_Py_identifier et al.). - -- Issue #13063: the Windows error ERROR_NO_DATA (numbered 232 and described - as "The pipe is being closed") is now mapped to POSIX errno EPIPE - (previously EINVAL). - -- Issue #12911: Fix memory consumption when calculating the repr() of huge - tuples or lists. - -- PEP 393: flexible string representation. Thanks to Torsten Becker for the - initial implementation, and Victor Stinner for various bug fixes. - -- Issue #14081: The 'sep' and 'maxsplit' parameter to str.split, bytes.split, - and bytearray.split may now be passed as keyword arguments. - -- Issue #13012: The 'keepends' parameter to str.splitlines may now be passed - as a keyword argument: "my_string.splitlines(keepends=True)". The same - change also applies to bytes.splitlines and bytearray.splitlines. - -- Issue #7732: Don't open a directory as a file anymore while importing a - module. Ignore the direcotry if its name matchs the module name (e.g. - "__init__.py") and raise a ImportError instead. - -- Issue #13021: Missing decref on an error path. Thanks to Suman Saha for - finding the bug and providing a patch. - -- Issue #12973: Fix overflow checks that relied on undefined behaviour in - list_repeat (listobject.c) and islice_next (itertoolsmodule.c). These bugs - caused test failures with recent versions of Clang. - -- Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write - atime and mtime with nanosecond precision on modern POSIX platforms. - -- Issue #12802: the Windows error ERROR_DIRECTORY (numbered 267) is now - mapped to POSIX errno ENOTDIR (previously EINVAL). - -- Issue #9200: The str.is* methods now work with strings that contain non-BMP - characters even in narrow Unicode builds. - -- Issue #12791: Break reference cycles early when a generator exits with - an exception. - -- Issue #12773: Make __doc__ mutable on user-defined classes. - -- Issue #12766: Raise a ValueError when creating a class with a class variable - that conflicts with a name in __slots__. - -- Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase - titlecased and cased non-letter characters. - -- Issue #12732: In narrow unicode builds, allow Unicode identifiers which fall - outside the BMP. - -- Issue #12575: Validate user-generated AST before it is compiled. - -- Make type(None), type(Ellipsis), and type(NotImplemented) callable. They - return the respective singleton instances. - -- Forbid summing bytes with sum(). - -- Verify the types of AST strings and identifiers provided by the user before - compiling them. - -- Issue #12647: The None object now has a __bool__() method that returns False. - Formerly, bool(None) returned False only because of special case logic - in PyObject_IsTrue(). - -- Issue #12579: str.format_map() now raises a ValueError if used on a - format string that contains positional fields. Initial patch by - Julian Berman. - -- Issue #10271: Allow warnings.showwarning() be any callable. - -- Issue #11627: Fix segfault when __new__ on a exception returns a - non-exception class. - -- Issue #12149: Update the method cache after a type's dictionary gets - cleared by the garbage collector. This fixes a segfault when an instance - and its type get caught in a reference cycle, and the instance's - deallocator calls one of the methods on the type (e.g. when subclassing - IOBase). Diagnosis and patch by Davide Rizzo. - -- Issue #9611, Issue #9015: FileIO.read() clamps the length to INT_MAX on Windows. - -- Issue #9642: Uniformize the tests on the availability of the mbcs codec, add - a new HAVE_MBCS define. - -- Issue #9642: Fix filesystem encoding initialization: use the ANSI code page - on Windows if the mbcs codec is not available, and fail with a fatal error if - we cannot get the locale encoding (if nl_langinfo(CODESET) is not available) - instead of using UTF-8. - -- When a generator yields, do not retain the caller's exception state on the - generator. - -- Issue #12475: Prevent generators from leaking their exception state into the - caller's frame as they return for the last time. - -- Issue #12291: You can now load multiple marshalled objects from a stream, - with other data interleaved between marshalled objects. - -- Issue #12356: When required positional or keyword-only arguments are not - given, produce a informative error message which includes the name(s) of the - missing arguments. - -- Issue #12370: Fix super with no arguments when __class__ is overriden in the - class body. - -- Issue #12084: os.stat on Windows now works properly with relative symbolic - links when called from any directory. - -- Loosen type restrictions on the __dir__ method. __dir__ can now return any - sequence, which will be converted to a list and sorted by dir(). - -- Issue #12265: Make error messages produced by passing an invalid set of - arguments to a function more informative. - -- Issue #12225: Still allow Python to build if Python is not in its hg repo or - mercurial is not installed. - -- Issue #1195: my_fgets() now always clears errors before calling fgets(). Fix - the following case: sys.stdin.read() stopped with CTRL+d (end of file), - raw_input() interrupted by CTRL+c. - -- Issue #12216: Allow unexpected EOF errors to happen on any line of the file. - -- Issue #12199: The TryExcept and TryFinally and AST nodes have been unified - into a Try node. - -- Issue #9670: Increase the default stack size for secondary threads on - Mac OS X and FreeBSD to reduce the chances of a crash instead of a - "maximum recursion depth" RuntimeError exception. - (patch by Ronald Oussoren) - -- Issue #12106: The use of the multiple-with shorthand syntax is now reflected - in the AST. - -- Issue #12190: Try to use the same filename object when compiling unmarshalling - a code objects in the same file. - -- Issue #12166: Move implementations of dir() specialized for various types into - the __dir__() methods of those types. - -- Issue #5715: In socketserver, close the server socket in the child process. - -- Correct lookup of __dir__ on objects. Among other things, this causes errors - besides AttributeError found on lookup to be propagated. - -- Issue #12060: Use sig_atomic_t type and volatile keyword in the signal - module. Patch written by Charles-Fran?ois Natali. - -- Issue #1746656: Added the if_nameindex, if_indextoname, if_nametoindex - methods to the socket module. - -- Issue #12044: Fixed subprocess.Popen when used as a context manager to - wait for the process to end when exiting the context to avoid unintentionally - leaving zombie processes around. - -- Issue #1195: Fix input() if it is interrupted by CTRL+d and then CTRL+c, - clear the end-of-file indicator after CTRL+d. - -- Issue #1856: Avoid crashes and lockups when daemon threads run while the - interpreter is shutting down; instead, these threads are now killed when - they try to take the GIL. - -- Issue #9756: When calling a method descriptor or a slot wrapper descriptor, - the check of the object type doesn't read the __class__ attribute anymore. - Fix a crash if a class override its __class__ attribute (e.g. a proxy of the - str type). Patch written by Andreas St?hrk. - -- Issue #10517: After fork(), reinitialize the TLS used by the PyGILState_* - APIs, to avoid a crash with the pthread implementation in RHEL 5. Patch - by Charles-Fran?ois Natali. - -- Issue #10914: Initialize correctly the filesystem codec when creating a new - subinterpreter to fix a bootstrap issue with codecs implemented in Python, as - the ISO-8859-15 codec. - -- Issue #11918: OS/2 and VMS are no more supported because of the lack of - maintainer. - -- Issue #6780: fix starts/endswith error message to mention that tuples are - accepted too. - -- Issue #5057: fix a bug in the peepholer that led to non-portable pyc files - between narrow and wide builds while optimizing BINARY_SUBSCR on non-BMP - chars (e.g. "\U00012345"[0]). - -- Issue #11845: Fix typo in rangeobject.c that caused a crash in - compute_slice_indices. Patch by Daniel Urban. - -- Issue #5673: Added a `timeout` keyword argument to subprocess.Popen.wait, - subprocess.Popen.communicated, subprocess.call, subprocess.check_call, and - subprocess.check_output. If the blocking operation takes more than `timeout` - seconds, the `subprocess.TimeoutExpired` exception is raised. - -- Issue #11650: PyOS_StdioReadline() retries fgets() if it was interrupted - (EINTR), for example if the program is stopped with CTRL+z on Mac OS X. Patch - written by Charles-Francois Natali. - -- Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error. - -- Issue #10785: Store the filename as Unicode in the Python parser. - -- Issue #11619: _PyImport_LoadDynamicModule() doesn't encode the path to bytes - on Windows. - -- Issue #10998: Remove mentions of -Q, sys.flags.division_warning and - Py_DivisionWarningFlag left over from Python 2. - -- Issue #11244: Remove an unnecessary peepholer check that was preventing - negative zeros from being constant-folded properly. - -- Issue #11395: io.FileIO().write() clamps the data length to 32,767 bytes on - Windows if the file is a TTY to workaround a Windows bug. The Windows console - returns an error (12: not enough space error) on writing into stdout if - stdout mode is binary and the length is greater than 66,000 bytes (or less, - depending on heap usage). - -- Issue #11320: fix bogus memory management in Modules/getpath.c, leading to - a possible crash when calling Py_SetPath(). - -- Issue #11432: A bug was introduced in subprocess.Popen on posix systems with - 3.2.0 where the stdout or stderr file descriptor being the same as the stdin - file descriptor would raise an exception. webbrowser.open would fail. fixed. - -- Issue #9856: Change object.__format__ with a non-empty format string - to be a DeprecationWarning. In 3.2 it was a PendingDeprecationWarning. - In 3.4 it will be a TypeError. - -- Issue #11244: The peephole optimizer is now able to constant-fold - arbitrarily complex expressions. This also fixes a 3.2 regression where - operations involving negative numbers were not constant-folded. - -- Issue #11450: Don't truncate hg version info in Py_GetBuildInfo() when - there are many tags (e.g. when using mq). Patch by Nadeem Vawda. - -- Issue #11335: Fixed a memory leak in list.sort when the key function - throws an exception. - -- Issue #8923: When a string is encoded to UTF-8 in strict mode, the result is - cached into the object. Examples: str.encode(), str.encode('utf-8'), - PyUnicode_AsUTF8String() and PyUnicode_AsEncodedString(unicode, "utf-8", - NULL). - -- Issue #10829: Refactor PyUnicode_FromFormat(), use the same function to parse - the format string in the 3 steps, fix crashs on invalid format strings. - -- Issue #13007: whichdb should recognize gdbm 1.9 magic numbers. - -- Issue #11286: Raise a ValueError from calling PyMemoryView_FromBuffer with - a buffer struct having a NULL data pointer. - -- Issue #11272: On Windows, input() strips '\r' (and not only '\n'), and - sys.stdin uses universal newline (replace '\r\n' by '\n'). - -- Issue #11828: startswith and endswith now accept None as slice index. - Patch by Torsten Becker. - -- Issue #11168: Remove filename debug variable from PyEval_EvalFrameEx(). - It encoded the Unicode filename to UTF-8, but the encoding fails on - undecodable filename (on surrogate characters) which raises an unexpected - UnicodeEncodeError on recursion limit. - -- Issue #11187: Remove bootstrap code (use ASCII) of - PyUnicode_AsEncodedString(), it was replaced by a better fallback (use the - locale encoding) in PyUnicode_EncodeFSDefault(). - -- Check for NULL result in PyType_FromSpec. - -- Issue #10516: New copy() and clear() methods for lists and bytearrays. - -- Issue #11386: bytearray.pop() now throws IndexError when the bytearray is - empty, instead of OverflowError. - -- Issue #12380: The rjust, ljust and center methods of bytes and bytearray - now accept a bytearray argument. - -Library -------- - -- Issue #14195: An issue that caused weakref.WeakSet instances to incorrectly - return True for a WeakSet instance 'a' in both 'a < a' and 'a > a' has been - fixed. - -- Issue #14166: Pickler objects now have an optional ``dispatch_table`` - attribute which allows to set custom per-pickler reduction functions. - Patch by sbt. - -- Issue #14177: marshal.loads() now raises TypeError when given an unicode - string. Patch by Guilherme Gon?alves. - -- Issue #13550: Remove the debug machinery from the threading module: remove - verbose arguments from all threading classes and functions. - -- Issue #14159: Fix the len() of weak containers (WeakSet, WeakKeyDictionary, - WeakValueDictionary) to return a better approximation when some objects - are dead or dying. Moreover, the implementation is now O(1) rather than - O(n). - -- Issue #11841: Fix comparison bug with 'rc' versions in packaging.version. - Patch by Filip Gruszczy?ski. - -- Issue #6884: Fix long-standing bugs with MANIFEST.in parsing in distutils - on Windows. Also fixed in packaging. - -- Issue #8033: sqlite3: Fix 64-bit integer handling in user functions - on 32-bit architectures. Initial patch by Philippe Devalkeneer. - -- HTMLParser is now able to handle slashes in the start tag. - -- Issue #13641: Decoding functions in the base64 module now accept ASCII-only - unicode strings. Patch by Catalin Iacob. - -- Issue #14043: Speed up importlib's _FileFinder by at least 8x, and add a - new importlib.invalidate_caches() function. - -- Issue #14001: CVE-2012-0845: xmlrpc: Fix an endless loop in - SimpleXMLRPCServer upon malformed POST request. - -- Issue #13961: Move importlib over to using os.replace() for atomic renaming. - -- Do away with ambiguous level values (as suggested by PEP 328) in - importlib.__import__() by raising ValueError when level < 0. - -- Issue #2489: pty.spawn could consume 100% cpu when it encountered an EOF. - -- Issue #13014: Fix a possible reference leak in SSLSocket.getpeercert(). - -- Issue #13777: Add PF_SYSTEM sockets on OS X. - Patch by Michael Goderbauer. - -- Issue #13015: Fix a possible reference leak in defaultdict.__repr__. - Patch by Suman Saha. - -- Issue #1326113: distutils' and packaging's build_ext commands option now - correctly parses multiple values (separated by whitespace or commas) given - to their --libraries option. - -- Issue #10287: nntplib now queries the server's CAPABILITIES first before - sending MODE READER, and only sends it if not already in READER mode. - Patch by Hynek Schlawack. - -- Issue #13993: HTMLParser is now able to handle broken end tags when - strict=False. - -- Issue #13930: lib2to3 now supports writing converted output files to another - directory tree as well as copying unchanged files and altering the file - suffix. - -- Issue #9750: Fix sqlite3.Connection.iterdump on tables and fields - with a name that is a keyword or contains quotes. Patch by Marko - Kohtala. - -- Issue #10287: nntplib now queries the server's CAPABILITIES again after - authenticating (since the result may change, according to RFC 4643). - Patch by Hynek Schlawack. - -- Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building - Distutils-based packages with C extension modules may fail because - Apple has removed gcc-4.2, the version used to build python.org - 64-bit/32-bit Pythons. If the user does not explicitly override - the default C compiler by setting the CC environment variable, - Distutils will now attempt to compile extension modules with clang - if gcc-4.2 is required but not found. Also as a convenience, if - the user does explicitly set CC, substitute its value as the default - compiler in the Distutils LDSHARED configuration variable for OS X. - (Note, the python.org 32-bit-only Pythons use gcc-4.0 and the 10.4u - SDK, neither of which are available in Xcode 4. This change does not - attempt to override settings to support their use with Xcode 4.) - -- Issue #13960: HTMLParser is now able to handle broken comments when - strict=False. - -- When '' is a path (e.g. in sys.path), make sure __file__ uses the current - working directory instead of '' in importlib. - -- Issue #13609: Add two functions to query the terminal size: - os.get_terminal_size (low level) and shutil.get_terminal_size (high level). - Patch by Zbigniew J?drzejewski-Szmek. - -- Issue #13845: On Windows, time.time() now uses GetSystemTimeAsFileTime() - instead of ftime() to have a resolution of 100 ns instead of 1 ms (the clock - accuracy is between 0.5 ms and 15 ms). - -- Issue #13846: Add time.monotonic(), monotonic clock. - -- Issue #8184: multiprocessing: On Windows, don't set SO_REUSEADDR on - Connection sockets, and set FILE_FLAG_FIRST_PIPE_INSTANCE on named pipes, to - make sure two listeners can't bind to the same socket/pipe (or any existing - socket/pipe). - -- Issue #10811: Fix recursive usage of cursors. Instead of crashing, - raise a ProgrammingError now. - -- Issue #13734: Add os.fwalk(), a directory walking function yielding file - descriptors. - -- Issue #2945: Make the distutils upload command aware of bdist_rpm products. - -- Issue #13712: pysetup create should not convert package_data to extra_files. - -- Issue #11805: package_data in setup.cfg should allow more than one value. - -- Issue #13676: Handle strings with embedded zeros correctly in sqlite3. - -- Issue #8828: Add new function os.replace(), for cross-platform renaming - with overwriting. - -- Issue #13848: open() and the FileIO constructor now check for NUL - characters in the file name. Patch by Hynek Schlawack. - -- Issue #13806: The size check in audioop decompression functions was too - strict and could reject valid compressed data. Patch by Oleg Plakhotnyuk. - -- Issue #13812: When a multiprocessing Process child raises an exception, - flush stderr after printing the exception traceback. - -- Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC - IV attack countermeasure. - -- Issue #13847: time.localtime() and time.gmtime() now raise an OSError instead - of ValueError on failure. time.ctime() and time.asctime() now raises an - OSError if localtime() failed. time.clock() now raises a RuntimeError if the - processor time used is not available or its value cannot be represented - -- Issue #13772: In os.symlink() under Windows, do not try to guess the link - target's type (file or directory). The detection was buggy and made the - call non-atomic (therefore prone to race conditions). - -- Issue #6631: Disallow relative file paths in urllib urlopen methods. - -- Issue #13722: Avoid silencing ImportErrors when initializing the codecs - registry. - -- Issue #13781: Fix GzipFile bug that caused an exception to be raised when - opening for writing using a fileobj returned by os.fdopen(). - -- Issue #13803: Under Solaris, distutils doesn't include bitness - in the directory name. - -- Issue #10278: Add time.wallclock() function, monotonic clock. - -- Issue #13809: Fix regression where bz2 module wouldn't work when threads are - disabled. Original patch by Amaury Forgeot d'Arc. - -- Issue #13589: Fix some serialization primitives in the aifc module. - Patch by Oleg Plakhotnyuk. - -- Issue #13642: Unquote before b64encoding user:password during Basic - Authentication. Patch contributed by Joonas Kuorilehto. - -- Issue #12364: Fix a hang in concurrent.futures.ProcessPoolExecutor. - The hang would occur when retrieving the result of a scheduled future after - the executor had been shut down. - -- Issue #13502: threading: Fix a race condition in Event.wait() that made it - return False when the event was set and cleared right after. - -- Issue #9993: When the source and destination are on different filesystems, - and the source is a symlink, shutil.move() now recreates a symlink on the - destination instead of copying the file contents. Patch by Jonathan Niehof - and Hynek Schlawack. - -- Issue #12926: Fix a bug in tarfile's link extraction. - -- Issue #13696: Fix the 302 Relative URL Redirection problem. - -- Issue #13636: Weak ciphers are now disabled by default in the ssl module - (except when SSLv2 is explicitly asked for). - -- Issue #12715: Add an optional symlinks argument to shutil functions - (copyfile, copymode, copystat, copy, copy2). When that parameter is - true, symlinks aren't dereferenced and the operation instead acts on the - symlink itself (or creates one, if relevant). Patch by Hynek Schlawack. - -- Add a flags parameter to select.epoll. - -- Issue #13626: Add support for SSL Diffie-Hellman key exchange, through the - SSLContext.load_dh_params() method and the ssl.OP_SINGLE_DH_USE option. - -- Issue #11006: Don't issue low level warning in subprocess when pipe2() fails. - -- Issue #13620: Support for Chrome browser in webbrowser. Patch contributed - by Arnaud Calmettes. - -- Issue #11829: Fix code execution holes in inspect.getattr_static for - metaclasses with metaclasses. Patch by Andreas St?hrk. - -- Issue #12708: Add starmap() and starmap_async() methods (similar to - itertools.starmap()) to multiprocessing.Pool. Patch by Hynek Schlawack. - -- Issue #1785: Fix inspect and pydoc with misbehaving descriptors. - -- Issue #13637: "a2b" functions in the binascii module now accept ASCII-only - unicode strings. - -- Issue #13634: Add support for querying and disabling SSL compression. - -- Issue #13627: Add support for SSL Elliptic Curve-based Diffie-Hellman - key exchange, through the SSLContext.set_ecdh_curve() method and the - ssl.OP_SINGLE_ECDH_USE option. - -- Issue #13635: Add ssl.OP_CIPHER_SERVER_PREFERENCE, so that SSL servers - choose the cipher based on their own preferences, rather than on the - client's. - -- Issue #11813: Fix inspect.getattr_static for modules. Patch by Andreas - St?hrk. - -- Issue #7502: Fix equality comparison for DocTestCase instances. Patch by - C?dric Krier. - -- Issue #11870: threading: Properly reinitialize threads internal locks and - condition variables to avoid deadlocks in child processes. - -- Issue #8035: urllib: Fix a bug where the client could remain stuck after a - redirection or an error. - -- Issue #13560: os.strerror() now uses the current locale encoding instead of - UTF-8. - -- Issue #8373: The filesystem path of AF_UNIX sockets now uses the filesystem - encoding and the surrogateescape error handler, rather than UTF-8. Patch - by David Watson. - -- Issue #10350: Read and save errno before calling a function which might - overwrite it. Original patch by Hallvard B Furuseth. - -- Issue #11610: Introduce a more general way to declare abstract properties. - -- Issue #13591: A bug in importlib has been fixed that caused import_module - to load a module twice. - -- Issue #13449 sched.scheduler.run() method has a new "blocking" parameter which - when set to False makes run() execute the scheduled events due to expire - soonest (if any) and then return. Patch by Giampaolo Rodol?. - -- Issue #8684 sched.scheduler class can be safely used in multi-threaded - environments. Patch by Josiah Carlson and Giampaolo Rodol?. - -- Alias resource.error to OSError ala PEP 3151. - -- Issue #5689: Add support for lzma compression to the tarfile module. - -- Issue #13248: Turn 3.2's PendingDeprecationWarning into 3.3's - DeprecationWarning. It covers 'cgi.escape', 'importlib.abc.PyLoader', - 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', - and private attributes of 'smtpd.SMTPChannel'. - -- Issue #5905, Issue #13560: time.strftime() is now using the current locale - encoding, instead of UTF-8, if the wcsftime() function is not available. - -- Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch - by Jon Kuhn. - -- tarfile.py: Correctly detect bzip2 compressed streams with blocksizes - other than 900k. - -- Issue #13439: Fix many errors in turtle docstrings. - -- Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. - Thanks to Per ?yvind Karlsen for the initial implementation. - -- Issue #13487: Make inspect.getmodule robust against changes done to - sys.modules while it is iterating over it. - -- Issue #12618: Fix a bug that prevented py_compile from creating byte - compiled files in the current directory. Initial patch by Sjoerd de Vries. - -- Issue #13444: When stdout has been closed explicitly, we should not attempt - to flush it at shutdown and print an error. - -- Issue #12567: The curses module uses Unicode functions for Unicode arguments - when it is linked to the ncurses library. It encodes also Unicode strings to - the locale encoding instead of UTF-8. - -- Issue #12856: Ensure child processes do not inherit the parent's random - seed for filename generation in the tempfile module. Patch by Brian - Harring. - -- Issue #9957: SpooledTemporaryFile.truncate() now accepts an optional size - parameter, as other file-like objects. Patch by Ryan Kelly. - -- Issue #13458: Fix a memory leak in the ssl module when decoding a - certificate with a subjectAltName. Patch by Robert Xiao. - -- Issue #13415: os.unsetenv() doesn't ignore errors anymore. - -- Issue #13245: sched.scheduler class constructor's timefunc and - delayfunct parameters are now optional. - scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter. - Patch contributed by Chris Clark. - -- Issue #12328: Under Windows, refactor handling of Ctrl-C events and - make _multiprocessing.win32.WaitForMultipleObjects interruptible when - the wait_flag parameter is false. Patch by sbt. - -- Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is - raised when the wrapped raw file is non-blocking and the write would block. - Previous code assumed that the raw write() would raise BlockingIOError, but - RawIOBase.write() is defined to returned None when the call would block. - Patch by sbt. - -- Issue #13358: HTMLParser now calls handle_data only once for each CDATA. - -- Issue #4147: minidom's toprettyxml no longer adds whitespace around a text - node when it is the only child of an element. Initial patch by Dan - Kenigsberg. - -- Issue #13374: The Windows bytes API has been deprecated in the os module. Use - Unicode filenames instead of bytes filenames to not depend on the ANSI code - page anymore and to support any filename. - -- Issue #13297: Use bytes type to send and receive binary data through XMLRPC. - -- Issue #6397: Support "/dev/poll" polling objects in select module, - under Solaris & derivatives. - -- Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly - handles non-valid attributes, including adjacent and unquoted attributes. - -- Issue #13193: Fix distutils.filelist.FileList and packaging.manifest.Manifest - under Windows. - -- Issue #13384: Remove unnecessary __future__ import in Lib/random.py - -- Issue #13149: Speed up append-only StringIO objects. - -- Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely - when called with a timeout. Patch by Arnaud Ysmal. - -- Issue #13254: Fix Maildir initialization so that maildir contents - are read correctly. - -- Issue #3067: locale.setlocale() now raises TypeError if the second - argument is an invalid iterable. Its documentation and docstring - were also updated. Initial patch by Jyrki Pulliainen. - -- Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn. - -- Issue #13339: Fix compile error in posixmodule.c due to missing semicolon. - Thanks to Robert Xiao. - -- Byte compilation in packaging is now isolated from the calling Python -B or - -O options, instead of being disallowed under -B or buggy under -O. - -- Issue #10570: curses.putp() and curses.tparm() are now expecting a byte - string, instead of a Unicode string. - -- Issue #13295: http.server now produces valid HTML 4.01 strict. - -- Issue #2892: preserve iterparse events in case of SyntaxError. - -- Issue #13287: urllib.request and urllib.error now contains an __all__ - attribute to expose only relevant classes and functions. Patch by Florent - Xicluna. - -- Issue #670664: Fix HTMLParser to correctly handle the content of - ```` and ````. - -- Issue #10817: Fix urlretrieve function to raise ContentTooShortError even - when reporthook is None. Patch by Jyrki Pulliainen. - -- Fix the xmlrpc.client user agent to return something similar to - urllib.request user agent: "Python-xmlrpc/3.3". - -- Issue #13293: Better error message when trying to marshal bytes using - xmlrpc.client. - -- Issue #13291: NameError in xmlrpc package. - -- Issue #13258: Use callable() built-in in the standard library. - -- Issue #13273: fix a bug that prevented HTMLParser to properly detect some - tags when strict=False. - -- Issue #11183: Add finer-grained exceptions to the ssl module, so that - you don't have to inspect the exception's attributes in the common case. - -- Issue #13216: Add cp65001 codec, the Windows UTF-8 (CP_UTF8). - -- Issue #13226: Add RTLD_xxx constants to the os module. These constants can be - used with sys.setdlopenflags(). - -- Issue #10278: Add clock_getres(), clock_gettime() and CLOCK_xxx constants to - the time module. time.clock_gettime(time.CLOCK_MONOTONIC) provides a - monotonic clock - -- Issue #10332: multiprocessing: fix a race condition when a Pool is closed - before all tasks have completed. - -- Issue #13255: wrong docstrings in array module. - -- Issue #8540: Remove deprecated Context._clamp attribute in Decimal module. - -- Issue #13235: Added DeprecationWarning to logging.warn() method and function. - -- Issue #9168: now smtpd is able to bind privileged port. - -- Issue #12529: fix cgi.parse_header issue on strings with double-quotes and - semicolons together. Patch by Ben Darnell and Petri Lehtinen. - -- Issue #13227: functools.lru_cache() now has a option to distinguish - calls with different argument types. - -- Issue #6090: zipfile raises a ValueError when a document with a timestamp - earlier than 1980 is provided. Patch contributed by Petri Lehtinen. - -- Issue #13150: sysconfig no longer parses the Makefile and config.h files - when imported, instead doing it at build time. This makes importing - sysconfig faster and reduces Python startup time by 20%. - -- Issue #12448: smtplib now flushes stdout while running ``python -m smtplib`` - in order to display the prompt correctly. - -- Issue #12454: The mailbox module is now using ASCII, instead of the locale - encoding, to read and write .mh_sequences files. - -- Issue #13194: zlib.compressobj().copy() and zlib.decompressobj().copy() are - now available on Windows. - -- Issue #1673007: urllib.request now supports HEAD request via new method argument. - Patch contributions by David Stanek, Patrick Westerhoff and Ezio Melotti. - -- Issue #12386: packaging does not fail anymore when writing the RESOURCES - file. - -- Issue #13158: Fix decoding and encoding of GNU tar specific base-256 number - fields in tarfile. - -- Issue #13025: mimetypes is now reading MIME types using the UTF-8 encoding, - instead of the locale encoding. - -- Issue #10653: On Windows, use strftime() instead of wcsftime() because - wcsftime() doesn't format time zone correctly. - -- Issue #13150: The tokenize module doesn't compile large regular expressions - at startup anymore. - -- Issue #11171: Fix distutils.sysconfig.get_makefile_filename when Python was - configured with different prefix and exec-prefix. - -- Issue #11254: Teach distutils and packaging to compile .pyc and .pyo files in - PEP 3147-compliant __pycache__ directories. - -- Issue #7367: Fix pkgutil.walk_paths to skip directories whose - contents cannot be read. - -- Issue #3163: The struct module gets new format characters 'n' and 'N' - supporting C integer types ``ssize_t`` and ``size_t``, respectively. - -- Issue #13099: Fix sqlite3.Cursor.lastrowid under a Turkish locale. - Reported and diagnosed by Thomas Kluyver. - -- Issue #13087: BufferedReader.seek() now always raises UnsupportedOperation - if the underlying raw stream is unseekable, even if the seek could be - satisfied using the internal buffer. Patch by John O'Connor. - -- Issue #7689: Allow pickling of dynamically created classes when their - metaclass is registered with copyreg. Patch by Nicolas M. Thi?ry and Craig - Citro. - -- Issue #13034: When decoding some SSL certificates, the subjectAltName - extension could be unreported. - -- Issue #12306: Expose the runtime version of the zlib C library as a constant, - ZLIB_RUNTIME_VERSION, in the zlib module. Patch by Torsten Landschoff. - -- Issue #12959: Add collections.ChainMap to collections.__all__. - -- Issue #8933: distutils' PKG-INFO files and packaging's METADATA files will - now correctly report Metadata-Version: 1.1 instead of 1.0 if a Classifier or - Download-URL field is present. - -- Issue #12567: Add curses.unget_wch() function. Push a character so the next - get_wch() will return it. - -- Issue #9561: distutils and packaging now writes egg-info files using UTF-8, - instead of the locale encoding. - -- Issue #8286: The distutils command sdist will print a warning message instead - of crashing when an invalid path is given in the manifest template. - -- Issue #12841: tarfile unnecessarily checked the existence of numerical user - and group ids on extraction. If one of them did not exist the respective id - of the current user (i.e. root) was used for the file and ownership - information was lost. - -- Issue #12888: Fix a bug in HTMLParser.unescape that prevented it to escape - more than 128 entities. Patch by Peter Otten. - -- Issue #12878: Expose a __dict__ attribute on io.IOBase and its subclasses. - -- Issue #12494: On error, call(), check_call(), check_output() and - getstatusoutput() functions of the subprocess module now kill the process, - read its status (to avoid zombis) and close pipes. - -- Issue #12720: Expose low-level Linux extended file attribute functions in os. - -- Issue #10946: The distutils commands bdist_dumb, bdist_wininst and bdist_msi - now respect a --skip-build option given to bdist. The packaging commands - were fixed too. - -- Issue #12847: Fix a crash with negative PUT and LONG_BINPUT arguments in - the C pickle implementation. - -- Issue #11564: Avoid crashes when trying to pickle huge objects or containers - (more than 2**31 items). Instead, in most cases, an OverflowError is raised. - -- Issue #12287: Fix a stack corruption in ossaudiodev module when the FD is - greater than FD_SETSIZE. - -- Issue #12839: Fix crash in zlib module due to version mismatch. - Fix by Richard M. Tew. - -- Issue #9923: The mailcap module now correctly uses the platform path - separator for the MAILCAP environment variable on non-POSIX platforms. - -- Issue #12835: Follow up to #6560 that unconditionally prevents use of the - unencrypted sendmsg/recvmsg APIs on SSL wrapped sockets. Patch by David - Watson. - -- Issue #12803: SSLContext.load_cert_chain() now accepts a password argument - to be used if the private key is encrypted. Patch by Adam Simpkins. - -- Issue #11657: Fix sending file descriptors over 255 over a multiprocessing - Pipe. - -- Issue #12811: tabnanny.check() now promptly closes checked files. Patch by - Anthony Briggs. - -- Issue #6560: The sendmsg/recvmsg API is now exposed by the socket module - when provided by the underlying platform, supporting processing of - ancillary data in pure Python code. Patch by David Watson and Heiko Wundram. - -- Issue #12326: On Linux, sys.platform doesn't contain the major version - anymore. It is now always 'linux', instead of 'linux2' or 'linux3' depending - on the Linux version used to build Python. - -- Issue #12213: Fix a buffering bug with interleaved reads and writes that - could appear on BufferedRandom streams. - -- Issue #12778: Reduce memory consumption when JSON-encoding a large - container of many small objects. - -- Issue #12650: Fix a race condition where a subprocess.Popen could leak - resources (FD/zombie) when killed at the wrong time. - -- Issue #12744: Fix inefficient representation of integers between 2**31 and - 2**63 on systems with a 64-bit C "long". - -- Issue #12646: Add an 'eof' attribute to zlib.Decompress, to make it easier to - detect truncated input streams. - -- Issue #11513: Fix exception handling ``tarfile.TarFile.gzopen()`` when - the file cannot be opened. - -- Issue #12687: Fix a possible buffering bug when unpickling text mode - (protocol 0, mostly) pickles. - -- Issue #10087: Fix the html output format of the calendar module. - -- Issue #13121: add support for inplace math operators to collections.Counter. - -- Add support for unary plus and unary minus to collections.Counter. - -- Issue #12683: urlparse updated to include svn as schemes that uses relative - paths. (svn from 1.5 onwards support relative path). - -- Issue #12655: Expose functions from sched.h in the os module: sched_yield(), - sched_setscheduler(), sched_getscheduler(), sched_setparam(), - sched_get_min_priority(), sched_get_max_priority(), sched_rr_get_interval(), - sched_getaffinity(), sched_setaffinity(). - -- Add ThreadError to threading.__all__. - -- Issues #11104, #8688: Fix the behavior of distutils' sdist command with - manually-maintained MANIFEST files. - -- Issue #11281: smtplib.STMP gets source_address parameter, which adds the - ability to bind to specific source address on a machine with multiple - interfaces. Patch by Paulo Scardine. - -- Issue #12464: tempfile.TemporaryDirectory.cleanup() should not follow - symlinks: fix it. Patch by Petri Lehtinen. - -- Issue #8887: "pydoc somebuiltin.somemethod" (or help('somebuiltin.somemethod') - in Python code) now finds the doc of the method. - -- Issue #10968: Remove indirection in threading. The public names (Event, - Condition, etc.) used to be factory functions returning instances of hidden - classes (_Event, _Condition, etc.), because (if Guido recalls correctly) this - code pre-dates the ability to subclass extension types. It is now possible - to inherit from these classes, without having to import the private - underscored names like multiprocessing did. - -- Issue #9723: Add shlex.quote functions, to escape filenames and command - lines. - -- Issue #12603: Fix pydoc.synopsis() on files with non-negative st_mtime. - -- Issue #12514: Use try/finally to assure the timeit module restores garbage - collections when it is done. - -- Issue #12607: In subprocess, fix issue where if stdin, stdout or stderr is - given as a low fd, it gets overwritten. - -- Issue #12576: Fix urlopen behavior on sites which do not send (or obfuscates) - ``Connection: close`` header. - -- Issue #12560: Build libpython.so on OpenBSD. Patch by Stefan Sperling. - -- Issue #1813: Fix codec lookup under Turkish locales. - -- Issue #12591: Improve support of "universal newlines" in the subprocess - module: the piped streams can now be properly read from or written to. - -- Issue #12591: Allow io.TextIOWrapper to work with raw IO objects (without - a read1() method), and add a *write_through* parameter to mandate - unbuffered writes. - -- Issue #10883: Fix socket leaks in urllib.request when using FTP. - -- Issue #12592: Make Python build on OpenBSD 5 (and future major releases). - -- Issue #12372: POSIX semaphores are broken on AIX: don't use them. - -- Issue #12551: Provide a get_channel_binding() method on SSL sockets so as - to get channel binding data for the current SSL session (only the - "tls-unique" channel binding is implemented). This allows the implementation - of certain authentication mechanisms such as SCRAM-SHA-1-PLUS. Patch by - Jacek Konieczny. - -- Issue #665194: email.utils now has format_datetime and parsedate_to_datetime - functions, allowing for round tripping of RFC2822 format dates. - -- Issue #12571: Add a plat-linux3 directory mirroring the plat-linux2 - directory, so that "import DLFCN" and other similar imports work on - Linux 3.0. - -- Issue #7484: smtplib no longer puts <> around addresses in VRFY and EXPN - commands; they aren't required and in fact postfix doesn't support that form. - -- Issue #12273: Remove ast.__version__. AST changes can be accounted for by - checking sys.version_info or sys._mercurial. - -- Silence spurious "broken pipe" tracebacks when shutting down a - ProcessPoolExecutor. - -- Fix potential resource leaks in concurrent.futures.ProcessPoolExecutor - by joining all queues and processes when shutdown() is called. - -- Issue #11603: Fix a crash when __str__ is rebound as __repr__. Patch by - Andreas St?hrk. - -- Issue #11321: Fix a crash with multiple imports of the _pickle module when - embedding Python. Patch by Andreas St?hrk. - -- Issue #6755: Add get_wch() method to curses.window class. Patch by I?igo - Serna. - -- Add cgi.closelog() function to close the log file. - -- Issue #12502: asyncore: fix polling loop with AF_UNIX sockets. - -- Issue #4376: ctypes now supports nested structures in a endian different than - the parent structure. Patch by Vlad Riscutia. - -- Raise ValueError when attempting to set the _CHUNK_SIZE attribute of a - TextIOWrapper to a huge value, not TypeError. - -- Issue #12504: Close file handles in a timely manner in packaging.database. - This fixes a bug with the remove (uninstall) feature on Windows. - -- Issues #12169 and #10510: Factor out code used by various packaging commands - to make HTTP POST requests, and make sure it uses CRLF. - -- Issue #12016: Multibyte CJK decoders now resynchronize faster. They only - ignore the first byte of an invalid byte sequence. For example, - b'\xff\n'.decode('gb2312', 'replace') gives '\ufffd\n' instead of '\ufffd'. - -- Issue #12459: time.sleep() now raises a ValueError if the sleep length is - negative, instead of an infinite sleep on Windows or raising an IOError on - Linux for example, to have the same behaviour on all platforms. - -- Issue #12451: pydoc: html_getfile() now uses tokenize.open() to support - Python scripts using a encoding different than UTF-8 (read the coding cookie - of the script). - -- Issue #12493: subprocess: Popen.communicate() now also handles EINTR errors - if the process has only one pipe. - -- Issue #12467: warnings: fix a race condition if a warning is emitted at - shutdown, if globals()['__file__'] is None. - -- Issue #12451: pydoc: importfile() now opens the Python script in binary mode, - instead of text mode using the locale encoding, to avoid encoding issues. - -- Issue #12451: runpy: run_path() now opens the Python script in binary mode, - instead of text mode using the locale encoding, to support other encodings - than UTF-8 (scripts using the coding cookie). - -- Issue #12451: xml.dom.pulldom: parse() now opens files in binary mode instead - of the text mode (using the locale encoding) to avoid encoding issues. - -- Issue #12147: Adjust the new-in-3.2 smtplib.send_message method for better - conformance to the RFCs: correctly handle Sender and Resent- headers. - -- Issue #12352: Fix a deadlock in multiprocessing.Heap when a block is freed by - the garbage collector while the Heap lock is held. - -- Issue #12462: time.sleep() now immediately calls the (Python) signal handler - if it is interrupted by a signal, instead of having to wait until the next - instruction. - -- Issue #12442: new shutil.disk_usage function, providing total, used and free - disk space statistics. - -- Issue #12451: The XInclude default loader of xml.etree now decodes files from - UTF-8 instead of the locale encoding if the encoding is not specified. It now - also opens XML files for the parser in binary mode instead of the text mode - to avoid encoding issues. - -- Issue #12451: doctest.debug_script() doesn't create a temporary file - anymore to avoid encoding issues. - -- Issue #12451: pydoc.synopsis() now reads the encoding cookie if available, - to read the Python script from the right encoding. - -- Issue #12451: distutils now opens the setup script in binary mode to read the - encoding cookie, instead of opening it in UTF-8. - -- Issue #9516: On Mac OS X, change Distutils to no longer globally attempt to - check or set the MACOSX_DEPLOYMENT_TARGET environment variable for the - interpreter process. This could cause failures in non-Distutils subprocesses - and was unreliable since tests or user programs could modify the interpreter - environment after Distutils set it. Instead, have Distutils set the - deployment target only in the environment of each build subprocess. It is - still possible to globally override the default by setting - MACOSX_DEPLOYMENT_TARGET before launching the interpreter; its value must be - greater or equal to the default value, the value with which the interpreter - was built. Also, implement the same handling in packaging. - -- Issue #12422: In the copy module, don't store objects that are their own copy - in the memo dict. - -- Issue #12303: Add sigwaitinfo() and sigtimedwait() to the signal module. - -- Issue #12404: Remove C89 incompatible code from mmap module. Patch by Akira - Kitada. - -- Issue #1874: email now detects and reports as a defect the presence of - any CTE other than 7bit, 8bit, or binary on a multipart. - -- Issue #12383: Fix subprocess module with env={}: don't copy the environment - variables, start with an empty environment. - -- Issue #11637: Fix support for importing packaging setup hooks from the - project directory. - -- Issue #6771: Moved the curses.wrapper function from the single-function - wrapper module into __init__, eliminating the module. Since __init__ was - already importing the function to curses.wrapper, there is no API change. - -- Issue #11584: email.header.decode_header no longer fails if the header - passed to it is a Header object, and Header/make_header no longer fail - if given binary unknown-8bit input. - -- Issue #11700: mailbox proxy object close methods can now be called multiple - times without error. - -- Issue #11767: Correct file descriptor leak in mailbox's __getitem__ method. - -- Issue #12133: AbstractHTTPHandler.do_open() of urllib.request closes the HTTP - connection if its getresponse() method fails with a socket error. Patch - written by Ezio Melotti. - -- Issue #12240: Allow multiple setup hooks in packaging's setup.cfg files. - Original patch by Erik Bray. - -- Issue #9284: Allow inspect.findsource() to find the source of doctest - functions. - -- Issue #11595: Fix assorted bugs in packaging.util.cfg_to_args, a - compatibility helper for the distutils-packaging transition. Original patch - by Erik Bray. - -- Issue #12287: In ossaudiodev, check that the device isn't closed in several - methods. - -- Issue #12009: Fixed regression in netrc file comment handling. - -- Issue #12246: Warn and fail when trying to install a third-party project from - an uninstalled Python (built in a source checkout). Original patch by - Tshepang Lekhonkhobe. - -- Issue #10694: zipfile now ignores garbage at the end of a zipfile. - -- Issue #12283: Fixed regression in smtplib quoting of leading dots in DATA. - -- Issue #10424: Argparse now includes the names of the missing required - arguments in the missing arguments error message. - -- Issue #12168: SysLogHandler now allows NUL termination to be controlled using - a new 'append_nul' attribute on the handler. - -- Issue #11583: Speed up os.path.isdir on Windows by using GetFileAttributes - instead of os.stat. - -- Issue #12021: Make mmap's read() method argument optional. Patch by Petri - Lehtinen. - -- Issue #9205: concurrent.futures.ProcessPoolExecutor now detects killed - children and raises BrokenProcessPool in such a situation. Previously it - would reliably freeze/deadlock. - -- Issue #12040: Expose a new attribute ``sentinel`` on instances of - ``multiprocessing.Process``. Also, fix Process.join() to not use polling - anymore, when given a timeout. - -- Issue #11893: Remove obsolete internal wrapper class ``SSLFakeFile`` in the - smtplib module. Patch by Catalin Iacob. - -- Issue #12080: Fix a Decimal.power() case that took an unreasonably long time - to compute. - -- Issue #12221: Remove __version__ attributes from pyexpat, pickle, tarfile, - pydoc, tkinter, and xml.parsers.expat. This were useless version constants - left over from the Mercurial transition - -- Named tuples now work correctly with vars(). - -- Issue #12085: Fix an attribute error in subprocess.Popen destructor if the - constructor has failed, e.g. because of an undeclared keyword argument. Patch - written by Oleg Oshmyan. - -- Issue #12028: Make threading._get_ident() public, rename it to - threading.get_ident() and document it. This function was already used using - _thread.get_ident(). - -- Issue #12171: IncrementalEncoder.reset() of CJK codecs (multibytecodec) calls - encreset() instead of decreset(). - -- Issue #12218: Removed wsgiref.egg-info. - -- Issue #12196: Add pipe2() to the os module. - -- Issue #985064: Make plistlib more resilient to faulty input plists. - Patch by Mher Movsisyan. - -- Issue #1625: BZ2File and bz2.decompress() now support multi-stream files. - Initial patch by Nir Aides. - -- Issue #12175: BufferedReader.read(-1) now calls raw.readall() if available. - -- Issue #12175: FileIO.readall() now only reads the file position and size - once. - -- Issue #12175: RawIOBase.readall() now returns None if read() returns None. - -- Issue #12175: FileIO.readall() now raises a ValueError instead of an IOError - if the file is closed. - -- Issue #11109: New service_action method for BaseServer, used by ForkingMixin - class for cleanup. Initial Patch by Justin Warkentin. - -- Issue #12045: Avoid duplicate execution of command in - ctypes.util._get_soname(). Patch by Sijin Joseph. - -- Issue #10818: Remove the Tk GUI and the serve() function of the pydoc module, - pydoc -g has been deprecated in Python 3.2 and it has a new enhanced web - server. - -- Issue #1441530: In imaplib, read the data in one chunk to speed up large - reads and simplify code. - -- Issue #12070: Fix the Makefile parser of the sysconfig module to handle - correctly references to "bogus variable" (e.g. "prefix=$/opt/python"). - -- Issue #12100: Don't reset incremental encoders of CJK codecs at each call to - their encode() method anymore, but continue to call the reset() method if the - final argument is True. - -- Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl - module. - -- Issue #6501: os.device_encoding() returns None on Windows if the application - has no console. - -- Issue #12105: Add O_CLOEXEC to the os module. - -- Issue #12079: Decimal('Infinity').fma(Decimal('0'), (3.91224318126786e+19+0j)) - now raises TypeError (reflecting the invalid type of the 3rd argument) rather - than Decimal.InvalidOperation. - -- Issue #12124: zipimport doesn't keep a reference to zlib.decompress() anymore - to be able to unload the module. - -- Add the packaging module, an improved fork of distutils (also known as - distutils2). - -- Issue #12065: connect_ex() on an SSL socket now returns the original errno - when the socket's timeout expires (it used to return None). - -- Issue #8809: The SMTP_SSL constructor and SMTP.starttls() now support - passing a ``context`` argument pointing to an ssl.SSLContext instance. - Patch by Kasun Herath. - -- Issue #9516: Issue #9516: avoid errors in sysconfig when MACOSX_DEPLOYMENT_TARGET - is set in shell. - -- Issue #8650: Make zlib module 64-bit clean. compress(), decompress() and - their incremental counterparts now raise OverflowError if given an input - larger than 4GB, instead of silently truncating the input and returning - an incorrect result. - -- Issue #12050: zlib.decompressobj().decompress() now clears the unconsumed_tail - attribute when called without a max_length argument. - -- Issue #12062: Fix a flushing bug when doing a certain type of I/O sequence - on a file opened in read+write mode (namely: reading, seeking a bit forward, - writing, then seeking before the previous write but still within buffered - data, and writing again). - -- Issue #9971: Write an optimized implementation of BufferedReader.readinto(). - Patch by John O'Connor. - -- Issue #11799: urllib.request Authentication Handlers will raise a ValueError - when presented with an unsupported Authentication Scheme. Patch contributed - by Yuval Greenfield. - -- Issue #10419, #6011: build_scripts command of distutils handles correctly - non-ASCII path (path to the Python executable). Open and write the script in - binary mode, but ensure that the shebang is decodable from UTF-8 and from the - encoding of the script. - -- Issue #8498: In socket.accept(), allow to specify 0 as a backlog value in - order to accept exactly one connection. Patch by Daniel Evers. - -- Issue #12011: signal.signal() and signal.siginterrupt() raise an OSError, - instead of a RuntimeError: OSError has an errno attribute. - -- Issue #3709: add a flush_headers method to BaseHTTPRequestHandler, which - manages the sending of headers to output stream and flushing the internal - headers buffer. Patch contribution by Andrew Schaaf - -- Issue #11743: Rewrite multiprocessing connection classes in pure Python. - -- Issue #11164: Stop trying to use _xmlplus in the xml module. - -- Issue #11888: Add log2 function to math module. Patch written by Mark - Dickinson. - -- Issue #12012: ssl.PROTOCOL_SSLv2 becomes optional. - -- Issue #8407: The signal handler writes the signal number as a single byte - instead of a nul byte into the wakeup file descriptor. So it is possible to - wait more than one signal and know which signals were raised. - -- Issue #8407: Add pthread_kill(), sigpending() and sigwait() functions to the - signal module. - -- Issue #11927: SMTP_SSL now uses port 465 by default as documented. Patch - by Kasun Herath. - -- Issue #12002: ftplib's abort() method raises TypeError. - -- Issue #11916: Add a number of MacOSX specific definitions to the errno module. - Patch by Pierre Carrier. - -- Issue #11999: fixed sporadic sync failure mailbox.Maildir due to its trying to - detect mtime changes by comparing to the system clock instead of to the - previous value of the mtime. - -- Issue #11072: added MLSD command (RFC-3659) support to ftplib. - -- Issue #8808: The IMAP4_SSL constructor now allows passing an SSLContext - parameter to control parameters of the secure channel. Patch by Sijin - Joseph. - -- ntpath.samefile failed to notice that "a.txt" and "A.TXT" refer to the same - file on Windows XP. As noticed in issue #10684. - -- Issue #12000: When a SSL certificate has a subjectAltName without any - dNSName entry, ssl.match_hostname() should use the subject's commonName. - Patch by Nicolas Bareil. - -- Issue #10775: assertRaises, assertRaisesRegex, assertWarns, and - assertWarnsRegex now accept a keyword argument 'msg' when used as context - managers. Initial patch by Winston Ewert. - -- Issue #10684: shutil.move used to delete a folder on case insensitive - filesystems when the source and destination name where the same except - for the case. - -- Issue #11647: objects created using contextlib.contextmanager now support - more than one call to the function when used as a decorator. Initial patch - by Ysj Ray. - -- Issue #11930: Removed deprecated time.accept2dyear variable. - Removed year >= 1000 restriction from datetime.strftime. - -- logging: don't define QueueListener if Python has no thread support. - -- functools.cmp_to_key() now works with collections.Hashable(). - -- Issue #11277: mmap.mmap() calls fcntl(fd, F_FULLFSYNC) on Mac OS X to get - around a mmap bug with sparse files. Patch written by Steffen Daode Nurpmeso. - -- Issue #8407: Add signal.pthread_sigmask() function to fetch and/or change the - signal mask of the calling thread. - -- Issue #11858: configparser.ExtendedInterpolation expected lower-case section - names. - -- Issue #11324: ConfigParser(interpolation=None) now works correctly. - -- Issue #11811: ssl.get_server_certificate() is now IPv6-compatible. Patch - by Charles-Fran?ois Natali. - -- Issue #11763: don't use difflib in TestCase.assertMultiLineEqual if the - strings are too long. - -- Issue #11236: getpass.getpass responds to ctrl-c or ctrl-z on terminal. - -- Issue #11856: Speed up parsing of JSON numbers. - -- Issue #11005: threading.RLock()._release_save() raises a RuntimeError if the - lock was not acquired. - -- Issue #11258: Speed up ctypes.util.find_library() under Linux by a factor - of 5 to 10. Initial patch by Jonas H. - -- Issue #11382: Trivial system calls, such as dup() or pipe(), needn't - release the GIL. Patch by Charles-Fran?ois Natali. - -- Issue #11223: Add threading._info() function providing informations about - the thread implementation. - -- Issue #11731: simplify/enhance email parser/generator API by introducing - policy objects. - -- Issue #11768: The signal handler of the signal module only calls - Py_AddPendingCall() for the first signal to fix a deadlock on reentrant or - parallel calls. PyErr_SetInterrupt() writes also into the wake up file. - -- Issue #11492: fix several issues with header folding in the email package. - -- Issue #11852: Add missing imports and update tests. - -- Issue #11875: collections.OrderedDict's __reduce__ was temporarily - mutating the object instead of just working on a copy. - -- Issue #11467: Fix urlparse behavior when handling urls which contains scheme - specific part only digits. Patch by Santoso Wijaya. - -- collections.Counter().copy() now works correctly for subclasses. - -- Issue #11474: Fix the bug with url2pathname() handling of '/C|/' on Windows. - Patch by Santoso Wijaya. - -- Issue #11684: complete email.parser bytes API by adding BytesHeaderParser. - -- The bz2 module now handles 4GiB+ input buffers correctly. - -- Issue #9233: Fix json.loads('{}') to return a dict (instead of a list), when - _json is not available. - -- Issue #11830: Remove unnecessary introspection code in the decimal module. - -- Issue #11703: urllib2.geturl() does not return correct url when the original - url contains #fragment. - -- Issue #10019: Fixed regression in json module where an indent of 0 stopped - adding newlines and acted instead like 'None'. - -- Issue #11186: pydoc ignores a module if its name contains a surrogate - character in the index of modules. - -- Issue #11815: Use a light-weight SimpleQueue for the result queue in - concurrent.futures.ProcessPoolExecutor. - -- Issue #5162: Treat services like frozen executables to allow child spawning - from multiprocessing.forking on Windows. - -- logging.basicConfig now supports an optional 'handlers' argument taking an - iterable of handlers to be added to the root logger. Additional parameter - checks were also added to basicConfig. - -- Issue #11814: Fix likely typo in multiprocessing.Pool._terminate(). - -- Issue #11747: Fix range formatting in difflib.context_diff() and - difflib.unified_diff(). - -- Issue #8428: Fix a race condition in multiprocessing.Pool when terminating - worker processes: new processes would be spawned while the pool is being - shut down. Patch by Charles-Fran?ois Natali. - -- Issue #2650: re.escape() no longer escapes the '_'. - -- Issue #11757: select.select() now raises ValueError when a negative timeout - is passed (previously, a select.error with EINVAL would be raised). Patch - by Charles-Fran?ois Natali. - -- Issue #7311: fix html.parser to accept non-ASCII attribute values. - -- Issue #11605: email.parser.BytesFeedParser was incorrectly converting - multipart subparts with an 8-bit CTE into unicode instead of preserving the - bytes. - -- Issue #1690608: email.util.formataddr is now RFC 2047 aware: it now has a - charset parameter that defaults to utf-8 and is used as the charset for RFC - 2047 encoding when the realname contains non-ASCII characters. - -- Issue #10963: Ensure that subprocess.communicate() never raises EPIPE. - -- Issue #10791: Implement missing method GzipFile.read1(), allowing GzipFile - to be wrapped in a TextIOWrapper. Patch by Nadeem Vawda. - -- Issue #11707: Added a fast C version of functools.cmp_to_key(). - Patch by Filip Gruszczy?ski. - -- Issue #11688: Add sqlite3.Connection.set_trace_callback(). Patch by - Torsten Landschoff. - -- Issue #11746: Fix SSLContext.load_cert_chain() to accept elliptic curve - private keys. - -- Issue #5863: Rewrite BZ2File in pure Python, and allow it to accept - file-like objects using a new ``fileobj`` constructor argument. Patch by - Nadeem Vawda. - -- unittest.TestCase.assertSameElements has been removed. - -- sys.getfilesystemencoding() raises a RuntimeError if initfsencoding() was not - called yet: detect bootstrap (startup) issues earlier. - -- Issue #11393: Add the new faulthandler module. - -- Issue #11618: Fix the timeout logic in threading.Lock.acquire() under Windows. - -- Removed the 'strict' argument to email.parser.Parser, which has been - deprecated since Python 2.4. - -- Issue #11256: Fix inspect.getcallargs on functions that take only keyword - arguments. - -- Issue #11696: Fix ID generation in msilib. - -- itertools.accumulate now supports an optional *func* argument for - a user-supplied binary function. - -- Issue #11692: Remove unnecessary demo functions in subprocess module. - -- Issue #9696: Fix exception incorrectly raised by xdrlib.Packer.pack_int when - trying to pack a negative (in-range) integer. - -- Issue #11675: multiprocessing.[Raw]Array objects created from an integer size - are now zeroed on creation. This matches the behaviour specified by the - documentation. - -- Issue #7639: Fix short file name generation in bdist_msi - -- Issue #11635: Don't use polling in worker threads and processes launched by - concurrent.futures. - -- Issue #5845: Automatically read readline configuration to enable completion - in interactive mode. - -- Issue #6811: Allow importlib to change a code object's co_filename attribute - to match the path to where the source code currently is, not where the code - object originally came from. - -- Issue #8754: Have importlib use the repr of a module name in error messages. - -- Issue #11591: Prevent "import site" from modifying sys.path when python - was started with -S. - -- collections.namedtuple() now adds a _source attribute to the generated - class. This make the source more accessible than the outdated - "verbose" option which prints to stdout but doesn't make the source - string available. - -- Issue #11371: Mark getopt error messages as localizable. Patch by Filip - Gruszczy?ski. - -- Issue #11333: Add __slots__ to collections ABCs. - -- Issue #11628: cmp_to_key generated class should use __slots__. - -- Issue #11666: let help() display named tuple attributes and methods - that start with a leading underscore. - -- Issue #11662: Make urllib and urllib2 ignore redirections if the - scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). - -- Issue #5537: Fix time2isoz() and time2netscape() functions of - httplib.cookiejar for expiration year greater than 2038 on 32-bit systems. - -- Issue #4391: Use proper gettext plural forms in optparse. - -- Issue #11127: Raise a TypeError when trying to pickle a socket object. - -- Issue #11563: ``Connection: close`` header is sent by requests using URLOpener - class which helps in closing of sockets after connection is over. Patch - contributions by Jeff McNeil and Nadeem Vawda. - -- Issue #11459: A ``bufsize`` value of 0 in subprocess.Popen() really creates - unbuffered pipes, such that select() works properly on them. - -- Issue #5421: Fix misleading error message when one of socket.sendto()'s - arguments has the wrong type. Patch by Nikita Vetoshkin. - -- Issue #10812: Add some extra posix functions to the os module. - -- Issue #10979: unittest stdout buffering now works with class and module - setup and teardown. - -- Issue #11243: fix the parameter querying methods of Message to work if - the headers contain un-encoded non-ASCII data. - -- Issue #11401: fix handling of headers with no value; this fixes a regression - relative to Python2 and the result is now the same as it was in Python2. - -- Issue #9298: base64 bodies weren't being folded to line lengths less than 78, - which was a regression relative to Python2. Unlike Python2, the last line - of the folded body now ends with a carriage return. - -- Issue #11560: shutil.unpack_archive now correctly handles the format - parameter. Patch by Evan Dandrea. - -- Issue #5870: Add `subprocess.DEVNULL` constant. - -- Issue #11133: fix two cases where inspect.getattr_static can trigger code - execution. Patch by Andreas St?hrk. - -- Issue #11569: use absolute path to the sysctl command in multiprocessing to - ensure that it will be found regardless of the shell PATH. This ensures - that multiprocessing.cpu_count works on default installs of MacOSX. - -- Issue #11501: disutils.archive_utils.make_zipfile no longer fails if zlib is - not installed. Instead, the zipfile.ZIP_STORED compression is used to create - the ZipFile. Patch by Natalia B. Bidart. - -- Issue #11289: `smtp.SMTP` class is now a context manager so it can be used - in a `with` statement. Contributed by Giampaolo Rodola. - -- Issue #11554: Fixed support for Japanese codecs; previously the body output - encoding was not done if euc-jp or shift-jis was specified as the charset. - -- Issue #11407: `TestCase.run` returns the result object used or created. - Contributed by Janathan Hartley. - -- Issue #11500: Fixed a bug in the OS X proxy bypass code for fully qualified - IP addresses in the proxy exception list. - -- Issue #11491: dbm.error is no longer raised when dbm.open is called with - the "n" as the flag argument and the file exists. The behavior matches - the documentation and general logic. - -- Issue #1162477: Postel Principle adjustment to email date parsing: handle the - fact that some non-compliant MUAs use '.' instead of ':' in time specs. - -- Issue #11131: Fix sign of zero in decimal.Decimal plus and minus - operations when the rounding mode is ROUND_FLOOR. - -- Issue #9935: Speed up pickling of instances of user-defined classes. - -- Issue #5622: Fix curses.wrapper to raise correct exception if curses - initialization fails. - -- Issue #11408: In threading.Lock.acquire(), only call gettimeofday() when - really necessary. Patch by Charles-Fran?ois Natali. - -- Issue #11391: Writing to a mmap object created with - ``mmap.PROT_READ|mmap.PROT_EXEC`` would segfault instead of raising a - TypeError. Patch by Charles-Fran?ois Natali. - -- Issue #9795: add context manager protocol support for nntplib.NNTP class. - -- Issue #11306: mailbox in certain cases adapts to an inability to open - certain files in read-write mode. Previously it detected this by - checking for EACCES, now it also checks for EROFS. - -- Issue #11265: asyncore now correctly handles EPIPE, EBADF and EAGAIN errors - on accept(), send() and recv(). - -- Issue #11377: Deprecate platform.popen() and reimplement it with os.popen(). - -- Issue #8513: On UNIX, subprocess supports bytes command string. - -- Issue #10866: Add socket.sethostname(). Initial patch by Ross Lagerwall. - -- Issue #11140: Lock.release() now raises a RuntimeError when attempting - to release an unacquired lock, as claimed in the threading documentation. - The _thread.error exception is now an alias of RuntimeError. Patch by - Filip Gruszczy?ski. Patch for _dummy_thread by Aymeric Augustin. - -- Issue #8594: ftplib now provides a source_address parameter to specify which - (address, port) to bind to before connecting. - -- Issue #11326: Add the missing connect_ex() implementation for SSL sockets, - and make it work for non-blocking connects. - -- Issue #11297: Add collections.ChainMap(). - -- Issue #10755: Add the posix.flistdir() function. Patch by Ross Lagerwall. - -- Issue #4761: Add the ``*at()`` family of functions (openat(), etc.) to the - posix module. Patch by Ross Lagerwall. - -- Issue #7322: Trying to read from a socket's file-like object after a timeout - occurred now raises an error instead of silently losing data. - -- Issue #11291: poplib.POP no longer suppresses errors on quit(). - -- Issue #11177: asyncore's create_socket() arguments can now be omitted. - -- Issue #6064: Add a ``daemon`` keyword argument to the threading.Thread - and multiprocessing.Process constructors in order to override the - default behaviour of inheriting the daemonic property from the current - thread/process. - -- Issue #10956: Buffered I/O classes retry reading or writing after a signal - has arrived and the handler returned successfully. - -- Issue #10784: New os.getpriority() and os.setpriority() functions. - -- Issue #11114: Fix catastrophic performance of tell() on text files (up - to 1000x faster in some cases). It is still one to two order of magnitudes - slower than binary tell(). - -- Issue #10882: Add os.sendfile function. - -- Issue #10868: Allow usage of the register method of an ABC as a class - decorator. - -- Issue #11224: Fixed a regression in tarfile that affected the file-like - objects returned by TarFile.extractfile() regarding performance, memory - consumption and failures with the stream interface. - -- Issue #10924: Adding salt and Modular Crypt Format to crypt library. - Moved old C wrapper to _crypt, and added a Python wrapper with - enhanced salt generation and simpler API for password generation. - -- Issue #11074: Make 'tokenize' so it can be reloaded. - -- Issue #11085: Moved collections abstract base classes into a separate - module called collections.abc, following the pattern used by importlib.abc. - For backwards compatibility, the names are imported into the collections - module. - -- Issue #4681: Allow mmap() to work on file sizes and offsets larger than - 4GB, even on 32-bit builds. Initial patch by Ross Lagerwall, adapted for - 32-bit Windows. - -- Issue #11169: compileall module uses repr() to format filenames and paths to - escape surrogate characters and show spaces. - -- Issue #11089: Fix performance issue limiting the use of ConfigParser() - with large config files. - -- Issue #10276: Fix the results of zlib.crc32() and zlib.adler32() on buffers - larger than 4GB. Patch by Nadeem Vawda. - -- Issue #11388: Added a clear() method to MutableSequence - -- Issue #11174: Add argparse.MetavarTypeHelpFormatter, which uses type names - for the names of optional and positional arguments in help messages. - -- Issue #9348: Raise an early error if argparse nargs and metavar don't match. - -- Issue #9026: Fix order of argparse sub-commands in help messages. - -- Issue #9347: Fix formatting for tuples in argparse type= error messages. - -- Issue #12191: Added shutil.chown() to change user and/or group owner of a - given path also specifying their names. - -- Issue #13988: The _elementtree accelerator is used whenever available. - Now xml.etree.cElementTree becomes a deprecated alias to ElementTree. - -Build ------ - -- Issue #6807: Run msisupport.mak earlier. - -- Issue #10580: Minor grammar change in Windows installer. - -- Issue #13326: Clean __pycache__ directories correctly on OpenBSD. - -- PEP 393: the configure option --with-wide-unicode is removed. - -- Issue #12852: Set _XOPEN_SOURCE to 700, instead of 600, to get POSIX 2008 - functions on OpenBSD (e.g. fdopendir). - -- Issue #11863: Remove support for legacy systems deprecated in Python 3.2 - (following PEP 11). These systems are systems using Mach C Threads, - SunOS lightweight processes, GNU pth threads and IRIX threads. - -- Issue #8746: Correct faulty configure checks so that os.chflags() and - os.lchflags() are once again built on systems that support these - functions (BSD and OS X). Also add new stat file flags for OS X - (UF_HIDDEN and UF_COMPRESSED). - -- Issue #10645: Installing Python no longer creates a - Python-X.Y.Z-pyX.Y.egg-info file in the lib-dynload directory. - -- Do not accidentally include the directory containing sqlite.h twice when - building sqlite3. - -- Issue #11217: For 64-bit/32-bit Mac OS X universal framework builds, - ensure "make install" creates symlinks in --prefix bin for the "-32" - files in the framework bin directory like the installer does. - -- Issue #11347: Use --no-as-needed when linking libpython3.so. - -- Issue #11411: Fix 'make DESTDIR=' with a relative destination. - -- Issue #11268: Prevent Mac OS X Installer failure if Documentation - package had previously been installed. - -- Issue #11495: OSF support is eliminated. It was deprecated in Python 3.2. - -IDLE ----- - -- Issue #14409: IDLE now properly executes commands in the Shell window - when it cannot read the normal config files on startup and - has to use the built-in default key bindings. - There was previously a bug in one of the defaults. - -- IDLE can be launched as python -m idlelib - -- Issue #3573: IDLE hangs when passing invalid command line args - (directory(ies) instead of file(s)) (Patch by Guilherme Polo) - -- Issue #14200: IDLE shell crash on printing non-BMP unicode character. - -- Issue #5219: Prevent event handler cascade in IDLE. - -- Issue #964437: Make IDLE help window non-modal. - Patch by Guilherme Polo and Roger Serwy. - -- Issue #13933: IDLE auto-complete did not work with some imported - module, like hashlib. (Patch by Roger Serwy) - -- Issue #13506: Add '' to path for IDLE Shell when started and restarted with Restart Shell. - Original patches by Marco Scataglini and Roger Serwy. - -- Issue #4625: If IDLE cannot write to its recent file or breakpoint files, - display a message popup and continue rather than crash. Original patch by - Roger Serwy. - -- Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". - Patch by Tal Einat. - -- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart. - (Patch by Roger Serwy) - -- Issue #9871: Prevent IDLE 3 crash when given byte stings - with invalid hex escape sequences, like b'\x0'. - (Original patch by Claudiu Popa.) - -- Issue #12636: IDLE reads the coding cookie when executing a Python script. - -- Issue #12540: Prevent zombie IDLE processes on Windows due to changes - in os.kill(). - -- Issue #12590: IDLE editor window now always displays the first line - when opening a long file. With Tk 8.5, the first line was hidden. - -- Issue #11088: don't crash when using F5 to run a script in IDLE on MacOSX - with Tk 8.5. - -- Issue #1028: Tk returns invalid Unicode null in %A: UnicodeDecodeError. - With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused - IDLE to exit. Converted to valid Unicode null in PythonCmd(). - -- Issue #11718: IDLE's open module dialog couldn't find the __init__.py - file in a package. - -Tools/Demos ------------ - -- Issue #14053: patchcheck.py ("make patchcheck") now works with MQ patches. - Patch by Francisco Mart?n Brugu?. - -- Issue #13930: 2to3 is now able to write its converted output files to another - directory tree as well as copying unchanged files and altering the file - suffix. See its new -o, -W and --add-suffix options. This makes it more - useful in many automated code translation workflows. - -- Issue #13628: python-gdb.py is now able to retrieve more frames in the Python - traceback if Python is optimized. - -- Issue #11996: libpython (gdb), replace "py-bt" command by "py-bt-full" and - add a smarter "py-bt" command printing a classic Python traceback. - -- Issue #11179: Make ccbench work under Python 3.1 and 2.7 again. - -- Issue #10639: reindent.py no longer converts newlines and will raise - an error if attempting to convert a file with mixed newlines. - "--newline" option added to specify new line character. - -Extension Modules ------------------ - -- Issue #16847: Fixed improper use of _PyUnicode_CheckConsistency() in - non-pydebug builds. Several extension modules now compile cleanly when - assert()s are enabled in standard builds (-DDEBUG flag). - -- Issue #13840: The error message produced by ctypes.create_string_buffer - when given a Unicode string has been fixed. - -- Issue #9975: socket: Fix incorrect use of flowinfo and scope_id. Patch by - Vilmos Nebehaj. - -- Issue #7777: socket: Add Reliable Datagram Sockets (PF_RDS) support. - -- Issue #13159: FileIO and BZ2Compressor/BZ2Decompressor now use a linear-time - buffer growth strategy instead of a quadratic-time one. - -- Issue #10141: socket: Add SocketCAN (PF_CAN) support. Initial patch by - Matthias Fuchs, updated by Tiago Gon?alves. - -- Issue #13070: Fix a crash when a TextIOWrapper caught in a reference cycle - would be finalized after the reference to its underlying BufferedRWPair's - writer got cleared by the GC. - -- Issue #12881: ctypes: Fix segfault with large structure field names. - -- Issue #13058: ossaudiodev: fix a file descriptor leak on error. Patch by - Thomas Jarosch. - -- Issue #13013: ctypes: Fix a reference leak in PyCArrayType_from_ctype. - Thanks to Suman Saha for finding the bug and providing a patch. - -- Issue #13022: Fix: _multiprocessing.recvfd() doesn't check that - file descriptor was actually received. - -- Issue #1172711: Add 'long long' support to the array module. - Initial patch by Oren Tirosh and Hirokazu Yamamoto. - -- Issue #12483: ctypes: Fix a crash when the destruction of a callback - object triggers the garbage collector. - -- Issue #12950: Fix passing file descriptors in multiprocessing, under - OpenIndiana/Illumos. - -- Issue #12764: Fix a crash in ctypes when the name of a Structure field is not - a string. - -- Issue #11241: subclasses of ctypes.Array can now be subclassed. - -- Issue #9651: Fix a crash when ctypes.create_string_buffer(0) was passed to - some functions like file.write(). - -- Issue #10309: Define _GNU_SOURCE so that mremap() gets the proper - signature. Without this, architectures where sizeof void* != sizeof int are - broken. Patch given by Hallvard B Furuseth. - -- Issue #12051: Fix segfault in json.dumps() while encoding highly-nested - objects using the C accelerations. - -- Issue #12017: Fix segfault in json.loads() while decoding highly-nested - objects using the C accelerations. - -- Issue #1838: Prevent segfault in ctypes, when _as_parameter_ on a class is set - to an instance of the class. - -Tests ------ - -- Issue #13125: Silence spurious test_lib2to3 output when in non-verbose mode. - Patch by Mikhail Novikov. - -- Issue #13447: Add a test file to host regression tests for bugs in the - scripts found in the Tools directory. - -- Issue #10881: Fix test_site failure with OS X framework builds. - -- Issue #13901: Prevent test_distutils failures on OS X with --enable-shared. - -- Issue #13862: Fix spurious failure in test_zlib due to runtime/compile time - minor versions not matching. - -- Issue #12804: Fix test_socket and test_urllib2net failures when running tests - on a system without internet access. - -- Issue #13726: Fix the ambiguous -S flag in regrtest. It is -o/--slow for slow - tests. - -- Issue #11659: Fix ResourceWarning in test_subprocess introduced by #11459. - Patch by Ben Hayden. - -- Issue #11577: fix ResourceWarning triggered by improved binhex test coverage - -- Issue #11509: Significantly increase test coverage of fileinput. - Patch by Denver Coneybeare at PyCon 2011 Sprints. - -- Issue #11689: Fix a variable scoping error in an sqlite3 test - -- Issue #13786: Remove unimplemented 'trace' long option from regrtest.py. - -- Issue #13725: Fix regrtest to recognize the documented -d flag. - Patch by Erno Tukia. - -- Issue #13304: Skip test case if user site-packages disabled (-s or - PYTHONNOUSERSITE). (Patch by Carl Meyer) - -- Issue #5661: Add a test for ECONNRESET/EPIPE handling to test_asyncore. Patch - by Xavier de Gaye. - -- Issue #13218: Fix test_ssl failures on Debian/Ubuntu. - -- Re-enable lib2to3's test_parser.py tests, though with an expected failure - (see issue 13125). - -- Issue #12656: Add tests for IPv6 and Unix sockets to test_asyncore. - -- Issue #6484: Add unit tests for mailcap module (patch by Gregory Nofi) - -- Issue #11651: Improve the Makefile test targets to run more of the test suite - more quickly. The --multiprocess option is now enabled by default, reducing - the amount of time needed to run the tests. "make test" and "make quicktest" - now include some resource-intensive tests, but no longer run the test suite - twice to check for bugs in .pyc generation. Tools/scripts/run_test.py provides - an easy platform-independent way to run test suite with sensible defaults. - -- Issue #12331: The test suite for the packaging module can now run from an - installed Python. - -- Issue #12331: The test suite for lib2to3 can now run from an installed - Python. - -- Issue #12626: In regrtest, allow to filter tests using a glob filter - with the ``-m`` (or ``--match``) option. This works with all test cases - using the unittest module. This is useful with long test suites - such as test_io or test_subprocess. - -- Issue #12624: It is now possible to fail after the first failure when - running in verbose mode (``-v`` or ``-W``), by using the ``--failfast`` - (or ``-G``) option to regrtest. This is useful with long test suites - such as test_io or test_subprocess. - -- Issue #12587: Correct faulty test file and reference in test_tokenize. - (Patch by Robert Xiao) - -- Issue #12573: Add resource checks for dangling Thread and Process objects. - -- Issue #12549: Correct test_platform to not fail when OS X returns 'x86_64' - as the processor type on some Mac systems. - -- Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary - failure in name resolution. - -- Issue #11812: Solve transient socket failure to connect to 'localhost' - in test_telnetlib.py. - -- Solved a potential deadlock in test_telnetlib.py. Related to issue #11812. - -- Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and - an overzealous DNS service (e.g. OpenDNS) redirects to a placeholder - Web site. - -- Avoid failing in test_urllibnet.test_bad_address when some overzealous - DNS service (e.g. OpenDNS) resolves a non-existent domain name. The test - is now skipped instead. - -- Issue #12440: When testing whether some bits in SSLContext.options can be - reset, check the version of the OpenSSL headers Python was compiled against, - rather than the runtime version of the OpenSSL library. - -- Issue #11512: Add a test suite for the cgitb module. Patch by Robbie Clemons. - -- Issue #12497: Install test/data to prevent failures of the various codecmaps - tests. - -- Issue #12496: Install test/capath directory to prevent test_connect_capath - testcase failure in test_ssl. - -- Issue #12469: Run wakeup and pending signal tests in a subprocess to run the - test in a fresh process with only one thread and to not change signal - handling of the parent process. - -- Issue #8716: Avoid crashes caused by Aqua Tk on OSX when attempting to run - test_tk or test_ttk_guionly under a username that is not currently logged - in to the console windowserver (as may be the case under buildbot or ssh). - -- Issue #12407: Explicitly skip test_capi.EmbeddingTest under Windows. - -- Issue #12400: regrtest -W doesn't rerun the tests twice anymore, but captures - the output and displays it on failure instead. regrtest -v doesn't print the - error twice anymore if there is only one error. - -- Issue #12141: Install copies of template C module file so that - test_build_ext of test_distutils and test_command_build_ext of - test_packaging are no longer silently skipped when - run outside of a build directory. - -- Issue #8746: Add additional tests for os.chflags() and os.lchflags(). - Patch by Garrett Cooper. - -- Issue #10736: Fix test_ttk test_widgets failures with Cocoa Tk 8.5.9 - 2.8 + on Mac OS X. (Patch by Ronald Oussoren) - -- Issue #12057: Add tests for ISO 2022 codecs (iso2022_jp, iso2022_jp_2, - iso2022_kr). - -- Issue #12096: Fix a race condition in test_threading.test_waitfor(). Patch - written by Charles-Fran?ois Natali. - -- Issue #11614: import __hello__ prints "Hello World!". Patch written by - Andreas St?hrk. - -- Issue #5723: Improve json tests to be executed with and without accelerations. - -- Issue #12041: Make test_wait3 more robust. - -- Issue #11873: Change regex in test_compileall to fix occasional failures when - when the randomly generated temporary path happened to match the regex. - -- Issue #11958: Fix FTP tests for IPv6, bind to "::1" instead of "localhost". - Patch written by Charles-Francois Natali. - -- Issue #8407, #11859: Fix tests of test_io using threads and an alarm: use - pthread_sigmask() to ensure that the SIGALRM signal is received by the main - thread. - -- Issue #11811: Factor out detection of IPv6 support on the current host - and make it available as ``test.support.IPV6_ENABLED``. Patch by - Charles-Fran?ois Natali. - -- Issue #10914: Add a minimal embedding test to test_capi. - -- Issue #11223: Skip test_lock_acquire_interruption() and - test_rlock_acquire_interruption() of test_threadsignals if a thread lock is - implemented using a POSIX mutex and a POSIX condition variable. A POSIX - condition variable cannot be interrupted by a signal (e.g. on Linux, the - futex system call is restarted). - -- Issue #11790: Fix sporadic failures in test_multiprocessing.WithProcessesTestCondition. - -- Fix possible "file already exists" error when running the tests in parallel. - -- Issue #11719: Fix message about unexpected test_msilib skip on non-Windows - platforms. Patch by Nadeem Vawda. - -- Issue #11727: Add a --timeout option to regrtest: if a test takes more than - TIMEOUT seconds, dumps the traceback of all threads and exits. - -- Issue #11653: fix -W with -j in regrtest. - -- The email test suite now lives in the Lib/test/test_email package. The test - harness code has also been modernized to allow use of new unittest features. - -- regrtest now discovers test packages as well as test modules. - -- Issue #11577: improve test coverage of binhex.py. Patch by Arkady Koplyarov. - -- New test_crashers added to exercise the scripts in the Lib/test/crashers - directory and confirm they fail as expected - -- Issue #11578: added test for the timeit module. Patch by Michael Henry. - -- Issue #11503: improve test coverage of posixpath.py. Patch by Evan Dandrea. - -- Issue #11505: improves test coverage of string.py, increases granularity of - string.Formatter tests. Initial patch by Alicia Arlen. - -- Issue #11548: Improve test coverage of the shutil module. Patch by - Evan Dandrea. - -- Issue #11554: Reactivated test_email_codecs. - -- Issue #11505: improves test coverage of string.py. Patch by Alicia - Arlen - -- Issue #11490: test_subprocess.test_leaking_fds_on_error no longer gives a - false positive if the last directory in the path is inaccessible. - -- Issue #11223: Fix test_threadsignals to fail, not hang, when the - non-semaphore implementation of locks is used under POSIX. - -- Issue #10911: Add tests on CGI with non-ASCII characters. Patch written by - Pierre Quentel. - -- Issue #9931: Fix hangs in GUI tests under Windows in certain conditions. - Patch by Hirokazu Yamamoto. - -- Issue #10512: Properly close sockets under test.test_cgi. - -- Issue #10992: Make tests pass under coverage. - -- Issue #10826: Prevent sporadic failure in test_subprocess on Solaris due - to open door files. - -- Issue #10990: Prevent tests from clobbering a set trace function. - -C-API ------ - -- Issue #13452: PyUnicode_EncodeDecimal() doesn't support error handlers - different than "strict" anymore. The caller was unable to compute the - size of the output buffer: it depends on the error handler. - -- Issue #13560: Add PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() - and PyUnicode_EncodeLocale() functions to the C API to decode/encode from/to - the current locale encoding. - -- Issue #10831: PyUnicode_FromFormat() supports %li, %lli and %zi formats. - -- Issue #11246: Fix PyUnicode_FromFormat("%V") to decode the byte string from - UTF-8 (with replace error handler) instead of ISO-8859-1 (in strict mode). - Patch written by Ray Allen. - -- Issue #10830: Fix PyUnicode_FromFormatV("%c") for non-BMP characters on - narrow build. - -- Add PyObject_GenericGetDict and PyObject_GeneriSetDict. They are generic - implementations for the getter and setter of a ``__dict__`` descriptor of C - types. - -- Issue #13727: Add 3 macros to access PyDateTime_Delta members: - PyDateTime_DELTA_GET_DAYS, PyDateTime_DELTA_GET_SECONDS, - PyDateTime_DELTA_GET_MICROSECONDS. - -- Issue #10542: Add 4 macros to work with surrogates: Py_UNICODE_IS_SURROGATE, - Py_UNICODE_IS_HIGH_SURROGATE, Py_UNICODE_IS_LOW_SURROGATE, - Py_UNICODE_JOIN_SURROGATES. - -- Issue #12724: Add Py_RETURN_NOTIMPLEMENTED macro for returning NotImplemented. - -- PY_PATCHLEVEL_REVISION has been removed, since it's meaningless with - Mercurial. - -- Issue #12173: The first argument of PyImport_ImportModuleLevel is now `const - char *` instead of `char *`. - -- Issue #12380: PyArg_ParseTuple now accepts a bytearray for the 'c' format. - -Documentation -------------- - -- Issue #13989: Document that GzipFile does not support text mode, and give a - more helpful error message when opened with an invalid mode string. - -- Issue #13921: Undocument and clean up sqlite3.OptimizedUnicode, - which is obsolete in Python 3.x. It's now aliased to str for - backwards compatibility. - -- Issue #12102: Document that buffered files must be flushed before being used - with mmap. Patch by Steffen Daode Nurpmeso. - -- Issue #8982: Improve the documentation for the argparse Namespace object. - -- Issue #9343: Document that argparse parent parsers must be configured before - their children. - -- Issue #13498: Clarify docs of os.makedirs()'s exist_ok argument. Done with - great native-speaker help from R. David Murray. - -- Issues #13491 and #13995: Fix many errors in sqlite3 documentation. - Initial patch for #13491 by Johannes Vogel. - -- Issue #13402: Document absoluteness of sys.executable. - -- Issue #13883: PYTHONCASEOK also works on OS X. - -- Issue #9021: Add an introduction to the copy module documentation. - -- Issue #6005: Examples in the socket library documentation use sendall, where - relevant, instead send method. - -- Issue #12798: Updated the mimetypes documentation. - -- Issue #12949: Document the kwonlyargcount argument for the PyCode_New - C API function. - -- Issue #13513: Fix io.IOBase documentation to correctly link to the - io.IOBase.readline method instead of the readline module. - -- Issue #13237: Reorganise subprocess documentation to emphasise convenience - functions and the most commonly needed arguments to Popen. - -- Issue #13141: Demonstrate recommended style for socketserver examples. - -- Issue #11818: Fix tempfile examples for Python 3. - - **(For information about older versions, consult the HISTORY file.)** -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:35:16 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 22:35:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Drop_Python_3=2E4=2E1_sect?= =?utf-8?q?ion_from_Misc/NEWS?= Message-ID: <3fnrQX22c2z7LjQ@mail.python.org> http://hg.python.org/cpython/rev/cb161cd94e6e changeset: 89829:cb161cd94e6e user: Victor Stinner date: Mon Mar 17 22:34:59 2014 +0100 summary: Drop Python 3.4.1 section from Misc/NEWS Changes of 3.4 after 3.4.0 must already be present in the 3.4 branch. files: Misc/NEWS | 71 ------------------------------------------- 1 files changed, 0 insertions(+), 71 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,77 +13,6 @@ Library ------- -What's New in Python 3.4.1rc1? -============================== - -Release date: TBA - -Core and Builtins ------------------ - -- Issue #20929: Add a type cast to avoid shifting a negative number. - -- Issue #20731: Properly position in source code files even if they - are opened in text mode. Patch by Serhiy Storchaka. - -- Issue #20637: Key-sharing now also works for instance dictionaries of - subclasses. Patch by Peter Ingebretson. - -Library -------- - -- Issue #19157: Include the broadcast address in the usuable hosts for IPv6 - in ipaddress. - -- Issue #11599: When an external command (e.g. compiler) fails, distutils now - prints out the whole command line (instead of just the command name) if the - environment variable DISTUTILS_DEBUG is set. - -- Issue #4931: distutils should not produce unhelpful "error: None" messages - anymore. distutils.util.grok_environment_error is kept but doc-deprecated. - -- Issue #20875: Prevent possible gzip "'read' is not defined" NameError. - Patch by Claudiu Popa. - -- Issue #11558: ``email.message.Message.attach`` now returns a more - useful error message if ``attach`` is called on a message for which - ``is_multipart`` is False. - -- Issue #20283: RE pattern methods now accept the string keyword parameters - as documented. The pattern and source keyword parameters are left as - deprecated aliases. - -- Issue #20778: Fix modulefinder to work with bytecode-only modules. - -- Issue #20791: copy.copy() now doesn't make a copy when the input is - a bytes object. Initial patch by Peter Otten. - -- Issue #19748: On AIX, time.mktime() now raises an OverflowError for year - outsize range [1902; 2037]. - -Documentation -------------- - -- Issue #20765: Add missing documentation for PurePath.with_name() and - PurePath.with_suffix(). - -- Issue #19407: New package installation and distribution guides based on - the Python Packaging Authority tools. Existing guides have been retained - as legacy links from the distutils docs, as they still contain some - required reference material for tool developers that isn't recorded - anywhere else. - -Tests ------ - -- Issue #20743: Fix a reference leak in test_tcl. - -Tools/Demos ------------ - -- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. - Patch by Arfrever Frehtes Taifersar Arahesis. - What's New in Python 3.4.0? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:40:07 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 22:40:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwODc5?= =?utf-8?q?=3A_Delay_the_initialization_of_encoding_and_decoding_tables_fo?= =?utf-8?q?r?= Message-ID: <3fnrX76yC2z7LjW@mail.python.org> http://hg.python.org/cpython/rev/7093d5758954 changeset: 89830:7093d5758954 branch: 3.4 parent: 89826:1724846a8676 user: Victor Stinner date: Mon Mar 17 22:38:41 2014 +0100 summary: Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the initialization of the unquote_to_bytes() table of the urllib.parse module, to not waste memory if these modules are not used. files: Lib/base64.py | 59 +++++++++++++++++++++++--------- Lib/urllib/parse.py | 9 +++- Misc/NEWS | 5 ++ 3 files changed, 54 insertions(+), 19 deletions(-) diff --git a/Lib/base64.py b/Lib/base64.py --- a/Lib/base64.py +++ b/Lib/base64.py @@ -138,15 +138,22 @@ # Base32 encoding/decoding must be done in Python _b32alphabet = b'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567' -_b32tab = [bytes([i]) for i in _b32alphabet] -_b32tab2 = [a + b for a in _b32tab for b in _b32tab] -_b32rev = {v: k for k, v in enumerate(_b32alphabet)} +_b32tab2 = None +_b32rev = None def b32encode(s): """Encode a byte string using Base32. s is the byte string to encode. The encoded byte string is returned. """ + global _b32tab2 + # Delay the initialization of the table to not waste memory + # if the function is never called + if _b32tab2 is None: + b32tab = [bytes((i,)) for i in _b32alphabet] + _b32tab2 = [a + b for a in b32tab for b in b32tab] + b32tab = None + if not isinstance(s, bytes_types): s = memoryview(s).tobytes() leftover = len(s) % 5 @@ -193,6 +200,11 @@ the input is incorrectly padded or if there are non-alphabet characters present in the input. """ + global _b32rev + # Delay the initialization of the table to not waste memory + # if the function is never called + if _b32rev is None: + _b32rev = {v: k for k, v in enumerate(_b32alphabet)} s = _bytes_from_decode_data(s) if len(s) % 8: raise binascii.Error('Incorrect padding') @@ -274,6 +286,11 @@ # Ascii85 encoding/decoding # +_a85chars = None +_a85chars2 = None +_A85START = b"<~" +_A85END = b"~>" + def _85encode(b, chars, chars2, pad=False, foldnuls=False, foldspaces=False): # Helper function for a85encode and b85encode if not isinstance(b, bytes_types): @@ -284,8 +301,6 @@ b = b + b'\0' * padding words = struct.Struct('!%dI' % (len(b) // 4)).unpack(b) - a85chars2 = _a85chars2 - a85chars = _a85chars chunks = [b'z' if foldnuls and not word else b'y' if foldspaces and word == 0x20202020 else (chars2[word // 614125] + @@ -300,11 +315,6 @@ return b''.join(chunks) -_A85START = b"<~" -_A85END = b"~>" -_a85chars = [bytes([i]) for i in range(33, 118)] -_a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] - def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False): """Encode a byte string using Ascii85. @@ -324,6 +334,13 @@ adobe controls whether the encoded byte sequence is framed with <~ and ~>, which is used by the Adobe implementation. """ + global _a85chars, _a85chars2 + # Delay the initialization of tables to not waste memory + # if the function is never called + if _a85chars is None: + _a85chars = [bytes((i,)) for i in range(33, 118)] + _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] + result = _85encode(b, _a85chars, _a85chars2, pad, True, foldspaces) if adobe: @@ -408,10 +425,10 @@ # The following code is originally taken (with permission) from Mercurial -_b85chars = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" \ - b"abcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~" -_b85chars = [bytes([i]) for i in _b85chars] -_b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] +_b85alphabet = (b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" + b"abcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~") +_b85chars = None +_b85chars2 = None _b85dec = None def b85encode(b, pad=False): @@ -420,17 +437,25 @@ If pad is true, the input is padded with "\0" so its length is a multiple of 4 characters before encoding. """ + global _b85chars, _b85chars2 + # Delay the initialization of tables to not waste memory + # if the function is never called + if _b85chars is None: + _b85chars = [bytes((i,)) for i in _b85alphabet] + _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] return _85encode(b, _b85chars, _b85chars2, pad) def b85decode(b): """Decode base85-encoded byte array""" - b = _bytes_from_decode_data(b) global _b85dec + # Delay the initialization of tables to not waste memory + # if the function is never called if _b85dec is None: _b85dec = [None] * 256 - for i, c in enumerate(_b85chars): - _b85dec[c[0]] = i + for i, c in enumerate(_b85alphabet): + _b85dec[c] = i + b = _bytes_from_decode_data(b) padding = (-len(b)) % 5 b = b + b'~' * padding out = [] diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -472,8 +472,7 @@ return _coerce_result(DefragResult(defrag, frag)) _hexdig = '0123456789ABCDEFabcdef' -_hextobyte = {(a + b).encode(): bytes([int(a + b, 16)]) - for a in _hexdig for b in _hexdig} +_hextobyte = None def unquote_to_bytes(string): """unquote_to_bytes('abc%20def') -> b'abc def'.""" @@ -490,6 +489,12 @@ return string res = [bits[0]] append = res.append + # Delay the initialization of the table to not waste memory + # if the function is never called + global _hextobyte + if _hextobyte is None: + _hextobyte = {(a + b).encode(): bytes([int(a + b, 16)]) + for a in _hexdig for b in _hexdig} for item in bits[1:]: try: append(_hextobyte[item[:2]]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,11 @@ Library ------- +- Issue #20879: Delay the initialization of encoding and decoding tables for + base32, ascii85 and base85 codecs in the base64 module, and delay the + initialization of the unquote_to_bytes() table of the urllib.parse module, to + not waste memory if these modules are not used. + - Issue #19157: Include the broadcast address in the usuable hosts for IPv6 in ipaddress. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:40:09 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 22:40:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2320879=3A_Delay_the_initializa?= =?utf-8?q?tion_of_encoding_and_decoding?= Message-ID: <3fnrX949pqz7Lkb@mail.python.org> http://hg.python.org/cpython/rev/06d646935c9a changeset: 89831:06d646935c9a parent: 89829:cb161cd94e6e parent: 89830:7093d5758954 user: Victor Stinner date: Mon Mar 17 22:39:49 2014 +0100 summary: (Merge 3.4) Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the initialization of the unquote_to_bytes() table of the urllib.parse module, to not waste memory if these modules are not used. files: Lib/base64.py | 59 +++++++++++++++++++++++--------- Lib/urllib/parse.py | 9 +++- Misc/NEWS | 5 ++ 3 files changed, 54 insertions(+), 19 deletions(-) diff --git a/Lib/base64.py b/Lib/base64.py --- a/Lib/base64.py +++ b/Lib/base64.py @@ -138,15 +138,22 @@ # Base32 encoding/decoding must be done in Python _b32alphabet = b'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567' -_b32tab = [bytes([i]) for i in _b32alphabet] -_b32tab2 = [a + b for a in _b32tab for b in _b32tab] -_b32rev = {v: k for k, v in enumerate(_b32alphabet)} +_b32tab2 = None +_b32rev = None def b32encode(s): """Encode a byte string using Base32. s is the byte string to encode. The encoded byte string is returned. """ + global _b32tab2 + # Delay the initialization of the table to not waste memory + # if the function is never called + if _b32tab2 is None: + b32tab = [bytes((i,)) for i in _b32alphabet] + _b32tab2 = [a + b for a in b32tab for b in b32tab] + b32tab = None + if not isinstance(s, bytes_types): s = memoryview(s).tobytes() leftover = len(s) % 5 @@ -193,6 +200,11 @@ the input is incorrectly padded or if there are non-alphabet characters present in the input. """ + global _b32rev + # Delay the initialization of the table to not waste memory + # if the function is never called + if _b32rev is None: + _b32rev = {v: k for k, v in enumerate(_b32alphabet)} s = _bytes_from_decode_data(s) if len(s) % 8: raise binascii.Error('Incorrect padding') @@ -274,6 +286,11 @@ # Ascii85 encoding/decoding # +_a85chars = None +_a85chars2 = None +_A85START = b"<~" +_A85END = b"~>" + def _85encode(b, chars, chars2, pad=False, foldnuls=False, foldspaces=False): # Helper function for a85encode and b85encode if not isinstance(b, bytes_types): @@ -284,8 +301,6 @@ b = b + b'\0' * padding words = struct.Struct('!%dI' % (len(b) // 4)).unpack(b) - a85chars2 = _a85chars2 - a85chars = _a85chars chunks = [b'z' if foldnuls and not word else b'y' if foldspaces and word == 0x20202020 else (chars2[word // 614125] + @@ -300,11 +315,6 @@ return b''.join(chunks) -_A85START = b"<~" -_A85END = b"~>" -_a85chars = [bytes([i]) for i in range(33, 118)] -_a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] - def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False): """Encode a byte string using Ascii85. @@ -324,6 +334,13 @@ adobe controls whether the encoded byte sequence is framed with <~ and ~>, which is used by the Adobe implementation. """ + global _a85chars, _a85chars2 + # Delay the initialization of tables to not waste memory + # if the function is never called + if _a85chars is None: + _a85chars = [bytes((i,)) for i in range(33, 118)] + _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] + result = _85encode(b, _a85chars, _a85chars2, pad, True, foldspaces) if adobe: @@ -408,10 +425,10 @@ # The following code is originally taken (with permission) from Mercurial -_b85chars = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" \ - b"abcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~" -_b85chars = [bytes([i]) for i in _b85chars] -_b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] +_b85alphabet = (b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" + b"abcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~") +_b85chars = None +_b85chars2 = None _b85dec = None def b85encode(b, pad=False): @@ -420,17 +437,25 @@ If pad is true, the input is padded with "\0" so its length is a multiple of 4 characters before encoding. """ + global _b85chars, _b85chars2 + # Delay the initialization of tables to not waste memory + # if the function is never called + if _b85chars is None: + _b85chars = [bytes((i,)) for i in _b85alphabet] + _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] return _85encode(b, _b85chars, _b85chars2, pad) def b85decode(b): """Decode base85-encoded byte array""" - b = _bytes_from_decode_data(b) global _b85dec + # Delay the initialization of tables to not waste memory + # if the function is never called if _b85dec is None: _b85dec = [None] * 256 - for i, c in enumerate(_b85chars): - _b85dec[c[0]] = i + for i, c in enumerate(_b85alphabet): + _b85dec[c] = i + b = _bytes_from_decode_data(b) padding = (-len(b)) % 5 b = b + b'~' * padding out = [] diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -472,8 +472,7 @@ return _coerce_result(DefragResult(defrag, frag)) _hexdig = '0123456789ABCDEFabcdef' -_hextobyte = {(a + b).encode(): bytes([int(a + b, 16)]) - for a in _hexdig for b in _hexdig} +_hextobyte = None def unquote_to_bytes(string): """unquote_to_bytes('abc%20def') -> b'abc def'.""" @@ -490,6 +489,12 @@ return string res = [bits[0]] append = res.append + # Delay the initialization of the table to not waste memory + # if the function is never called + global _hextobyte + if _hextobyte is None: + _hextobyte = {(a + b).encode(): bytes([int(a + b, 16)]) + for a in _hexdig for b in _hexdig} for item in bits[1:]: try: append(_hextobyte[item[:2]]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,11 @@ Library ------- +- Issue #20879: Delay the initialization of encoding and decoding tables for + base32, ascii85 and base85 codecs in the base64 module, and delay the + initialization of the unquote_to_bytes() table of the urllib.parse module, to + not waste memory if these modules are not used. + What's New in Python 3.4.0? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 22:45:44 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 17 Mar 2014 22:45:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_add_clang_howto_by_Jeffre?= =?utf-8?q?y_Walton?= Message-ID: <3fnrfc3kWJz7LjT@mail.python.org> http://hg.python.org/devguide/rev/aa1f1e41d8d5 changeset: 678:aa1f1e41d8d5 user: Benjamin Peterson date: Mon Mar 17 16:45:43 2014 -0500 summary: add clang howto by Jeffrey Walton files: clang.rst | 260 ++++++++++++++++++++++++++++++++++++++++++ index.rst | 2 + 2 files changed, 262 insertions(+), 0 deletions(-) diff --git a/clang.rst b/clang.rst new file mode 100644 --- /dev/null +++ b/clang.rst @@ -0,0 +1,260 @@ +********************************** + Dynamic Analysis with Clang +********************************** + +This document describes how to use Clang to perform analysis on Python and its +libraries. In addition to performing the analysis, the document will cover +downloading, building and installing the the latest Clang/LLVM combination +(which is currently 3.4). + +What is Clang? +============== + +Clang is the C, C++ and Objective C front-end for the LLVM compiler. The +front-end provides access to LLVM's optimizer and code generator. + +What are Sanitizers? +==================== + +Clang sanitizers are runtime checkers used to identify suspicious and undefined +behavior. The checking occurs at runtime with actual runtime parameters so false +positives are kept to a minimum. + +There are a number of sanitizers available, but two that should be used on a +regular basis are the Address Sanitizer (or Asan) and the Undefined Behavior +Sanitizer (or UBsan). Asan is invoked with the compiler option +``-fsanitize=address``, and UBsan is invoked with ``-fsanitize=undefined``. + +A complete list of sanitizers can be found at `Controlling Code Generation +`_. + +Clang/LLVM Setup +================ + +This portion of the document covers downloading, building and installing Clang +and LLVM. There are three components to download and build. They are the LLVM +compiler, the compiler front end and the compiler runtime library. + +In preparation you should create a scratch directory. Also ensure you are using +Python 2 and not Python 3. Python 3 will cause the build to fail. + +Download, Build and Install +--------------------------- + +Perform the following to download, build and install the Clang/LLVM 3.4. :: + + # Download + wget http://llvm.org/releases/3.4/llvm-3.4.src.tar.gz + wget http://llvm.org/releases/3.4/clang-3.4.src.tar.gz + wget http://llvm.org/releases/3.4/compiler-rt-3.4.src.tar.gz + + # LLVM + tar xvf llvm-3.4.src.tar.gz + cd llvm-3.4/tools + + # Clang Front End + tar xvf ../../clang-3.4.src.tar.gz + mv clang-3.4 clang + + # Compiler RT + cd ../projects + tar xvf ../../compiler-rt-3.4.src.tar.gz + mv compiler-rt-3.4/ compiler-rt + + # Build + cd .. + ./configure --enable-optimized --prefix=/usr/local + make -j4 + sudo make install + +.. note:: + + If you receive an error ``'LibraryDependencies.inc' file not found``, then + ensure you are utilizing Python 2 and not Python 3. If you encounter the + error after switching to Python 2, then delete everything and start over. If + the issue still persists, then see + http://llvm.org/bugs/show_bug.cgi?id=19158. + +After ``make install`` executes, the compilers will be installed in +``/usr/local/bin`` and the various librarioes will be installed in +``/usr/local/lib/clang/3.4/lib/linux/``: :: + + $ ls /usr/local/lib/clang/3.4/lib/linux/ + libclang_rt.asan-x86_64.a libclang_rt.profile-x86_64.a + libclang_rt.dfsan-x86_64.a libclang_rt.san-x86_64.a + libclang_rt.full-x86_64.a libclang_rt.tsan-x86_64.a + libclang_rt.lsan-x86_64.a libclang_rt.ubsan_cxx-x86_64.a + libclang_rt.msan-x86_64.a libclang_rt.ubsan-x86_64.a + +You should never have to add the libraries to a project. Clang should handle it +for you. + +The installer does not install all the components needed on occasion. For +example, you might want to run a ``scan-build`` or examine the results with +``scan-veiw``. You can copy the components by hand with: :: + + sudo mkdir /usr/local/bin/scan-build + sudo cp -r llvm-3.4/tools/clang/tools/scan-build /usr/local/bin + sudo mkdir /usr/local/bin/scan-view + sudo cp -r llvm-3.4/tools/clang/tools/scan-view /usr/local/bin + +.. note:: + + Because the installer does not install all the components needed on + occasion, you should not delete the scratch directory until you are sure + things work as expected. If a library is missing, then you should search for + it in the Clang/LLVM build directory. + +Python Build Setup +================== + +This portion of the document covers invoking Clang and LLVM with the options +required so the sanitizers analyze Python with under its test suite. Two +checkers are used - Asan and UBsan. + +Because the sanitizers are runtime checkers, its best to have as many positive +and negative self tests as possible. You can never have enough self tests. + +The general idea is to compile and link with the sanitizer flags. At link time, +Clang will include the needed runtime libraries. However, you can't use +``CFLAGS`` and ``CXXFLAGS`` to pass the options through the compiler to the +linker because the makefile rules for ``BUILDPYTHON``, ``_testembed`` and +``_freeze_importlib`` don't use the flags. + +As a workaround to the absence of flags to the linker, you can pass the +sanitizer options by way of the compilers - ``CC`` and ``CXX`. Passing the +flags though the compiler is used below, but passing them through ``LDFLAGS`` is +also reported to work. + +Building Python +--------------- + +To begin, export the variables of interest with the desired sanitizers. Its OK +to specify both sanitizers: :: + + # Asan + export CC="/usr/local/bin/clang -fsanitize=address" + export CXX="/usr/local/bin/clang++ -fsanitize=address -fno-sanitize=vptr" + +Or: + + # UBsan + export CC="/usr/local/bin/clang -fsanitize=undefined" + export CXX="/usr/local/bin/clang++ -fsanitize=undefined -fno-sanitize=vptr" + +The ``-fno-sanitize=vptr`` removes vtable checks that are part of UBsan from C++ +projects due to noise. + +Clang also accepts a blacklist of functions to ignore. There is no documentation +on the blacklist, but the check-in is available at +http://thread.gmane.org/gmane.comp.compilers.clang.scm/62388. You specify the +blacklist with ``-fsanitize-blacklist=XXX``. For example: :: + + -fsanitize-blacklist=my_blacklist.txt + +``my_blacklist.txt`` would then contain entries such as: :: + + fun:_Ios_Fmtflags + +Unfortunately, you won't know what to blacklist until you run the sanitizer. + +After exporting ``CC`` and ``CXX``, ``configure`` as normal: :: + + $ ./configure + checking build system type... x86_64-unknown-linux-gnu + checking host system type... x86_64-unknown-linux-gnu + checking for --enable-universalsdk... no + checking for --with-universal-archs... 32-bit + checking MACHDEP... linux + checking for --without-gcc... no + checking for gcc... /usr/local/bin/clang -fsanitize=undefined + checking whether the C compiler works... yes + ... + +Next is a standard ``make`` (formatting added for clarity): :: + + $ make + /usr/local/bin/clang -fsanitize=undefined -c -Wno-unused-result + -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. + -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o + ./Modules/python.c + /usr/local/bin/clang -fsanitize=undefined -c -Wno-unused-result + -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. + -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o + Parser/acceler.c + ... + +Finally is ``make test`` (formatting added for clarity): :: + + Objects/longobject.c:39:42: runtime error: index -1 out of bounds + for type 'PyLongObject [262]' + Objects/tupleobject.c:188:13: runtime error: member access within + misaligned address 0x2b76be018078 for type 'PyGC_Head' (aka + 'union _gc_head'), which requires 16 byte alignment + 0x2b76be018078: note: pointer points here + 00 00 00 00 40 53 5a b6 76 2b 00 00 60 52 5a b6 ... + ^ + ... + +If you are using the address sanitizer, its important to pipe the output through +``asan_symbolize.py`` to get a good trace. For example, from Issue 20953 during +compile (formatting added for clarity): :: + + $ /usr/local/bin/clang -fsanitize=address -Xlinker -export-dynamic + -o python Modules/python.o libpython3.3m.a -ldl -lutil + /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -lm + ./python -E -S -m sysconfig --generate-posix-vars + ================================================================= + ==24064==ERROR: AddressSanitizer: heap-buffer-overflow on address + 0x619000004020 at pc 0x4ed4b2 bp 0x7fff80fff010 sp 0x7fff80fff008 + READ of size 4 at 0x619000004020 thread T0 + #0 0x4ed4b1 in PyObject_Free Python-3.3.5/./Objects/obmalloc.c:987 + #1 0x7a2141 in code_dealloc Python-3.3.5/./Objects/codeobject.c:359 + #2 0x620c00 in PyImport_ImportFrozenModuleObject + Python-3.3.5/./Python/import.c:1098 + #3 0x620d5c in PyImport_ImportFrozenModule + Python-3.3.5/./Python/import.c:1114 + #4 0x63fd07 in import_init Python-3.3.5/./Python/pythonrun.c:206 + #5 0x63f636 in _Py_InitializeEx_Private + Python-3.3.5/./Python/pythonrun.c:369 + #6 0x681d77 in Py_Main Python-3.3.5/./Modules/main.c:648 + #7 0x4e6894 in main Python-3.3.5/././Modules/python.c:62 + #8 0x2abf9a525eac in __libc_start_main + /home/aurel32/eglibc/eglibc-2.13/csu/libc-start.c:244 + #9 0x4e664c in _start (Python-3.3.5/./python+0x4e664c) + + AddressSanitizer can not describe address in more detail (wild + memory access suspected). + SUMMARY: AddressSanitizer: heap-buffer-overflow + Python-3.3.5/./Objects/obmalloc.c:987 PyObject_Free + Shadow bytes around the buggy address: + 0x0c327fff87b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff87c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff87d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff87e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff87f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + =>0x0c327fff8800: fa fa fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff8810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff8820: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff8830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff8840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + 0x0c327fff8850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa + Shadow byte legend (one shadow byte represents 8 application bytes): + Addressable: 00 + Partially addressable: 01 02 03 04 05 06 07 + Heap left redzone: fa + Heap right redzone: fb + Freed heap region: fd + Stack left redzone: f1 + Stack mid redzone: f2 + Stack right redzone: f3 + Stack partial redzone: f4 + Stack after return: f5 + Stack use after scope: f8 + Global redzone: f9 + Global init order: f6 + Poisoned by user: f7 + ASan internal: fe + ==24064==ABORTING + make: *** [pybuilddir.txt] Error 1 + diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -171,6 +171,7 @@ * Tool support * :doc:`emacs` * :doc:`gdb` + * :doc:`clang` * Various tools with configuration files as found in the `Misc directory`_ * `python.org maintenance`_ @@ -210,6 +211,7 @@ grammar compiler coverity + clang faq -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon Mar 17 23:12:27 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 23:12:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320574=3A_Implemen?= =?utf-8?q?t_incremental_decoder_for_cp65001_code?= Message-ID: <3fnsFR0jCBz7LjT@mail.python.org> http://hg.python.org/cpython/rev/08f9b881f78c changeset: 89832:08f9b881f78c user: Victor Stinner date: Mon Mar 17 23:08:06 2014 +0100 summary: Issue #20574: Implement incremental decoder for cp65001 code (Windows code page 65001, Microsoft UTF-8). files: Lib/encodings/cp65001.py | 9 ++++-- Lib/test/test_codecs.py | 12 ++----- Misc/NEWS | 3 ++ Objects/unicodeobject.c | 41 ++++++--------------------- 4 files changed, 22 insertions(+), 43 deletions(-) diff --git a/Lib/encodings/cp65001.py b/Lib/encodings/cp65001.py --- a/Lib/encodings/cp65001.py +++ b/Lib/encodings/cp65001.py @@ -11,20 +11,23 @@ ### Codec APIs encode = functools.partial(codecs.code_page_encode, 65001) -decode = functools.partial(codecs.code_page_decode, 65001) +_decode = functools.partial(codecs.code_page_decode, 65001) + +def decode(input, errors='strict'): + return codecs.code_page_decode(65001, input, errors, True) class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return encode(input, self.errors)[0] class IncrementalDecoder(codecs.BufferedIncrementalDecoder): - _buffer_decode = decode + _buffer_decode = _decode class StreamWriter(codecs.StreamWriter): encode = encode class StreamReader(codecs.StreamReader): - decode = decode + decode = _decode ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -890,10 +890,6 @@ "\U00010fff\uD800") self.assertTrue(codecs.lookup_error("surrogatepass")) - def test_readline(self): - self.skipTest("issue #20571: code page 65001 codec does not " - "support partial decoder yet") - class UTF7Test(ReadTest, unittest.TestCase): encoding = "utf-7" @@ -2750,15 +2746,15 @@ self.assertRaisesRegex(UnicodeEncodeError, 'cp932', codecs.code_page_encode, 932, '\xff') self.assertRaisesRegex(UnicodeDecodeError, 'cp932', - codecs.code_page_decode, 932, b'\x81\x00') + codecs.code_page_decode, 932, b'\x81\x00', 'strict', True) self.assertRaisesRegex(UnicodeDecodeError, 'CP_UTF8', - codecs.code_page_decode, self.CP_UTF8, b'\xff') + codecs.code_page_decode, self.CP_UTF8, b'\xff', 'strict', True) def check_decode(self, cp, tests): for raw, errors, expected in tests: if expected is not None: try: - decoded = codecs.code_page_decode(cp, raw, errors) + decoded = codecs.code_page_decode(cp, raw, errors, True) except UnicodeDecodeError as err: self.fail('Unable to decode %a from "cp%s" with ' 'errors=%r: %s' % (raw, cp, errors, err)) @@ -2770,7 +2766,7 @@ self.assertLessEqual(decoded[1], len(raw)) else: self.assertRaises(UnicodeDecodeError, - codecs.code_page_decode, cp, raw, errors) + codecs.code_page_decode, cp, raw, errors, True) def check_encode(self, cp, tests): for text, errors, expected in tests: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ Library ------- +- Issue #20574: Implement incremental decoder for cp65001 code (Windows code + page 65001, Microsoft UTF-8). + - Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the initialization of the unquote_to_bytes() table of the urllib.parse module, to diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6817,28 +6817,6 @@ return PyBytes_AS_STRING(*obj); } -static int -is_dbcs_lead_byte(UINT code_page, const char *s, int offset) -{ - const char *curr = s + offset; - const char *prev; - - if (!IsDBCSLeadByteEx(code_page, *curr)) - return 0; - - prev = CharPrevExA(code_page, s, curr, 0); - if (prev == curr) - return 1; - /* FIXME: This code is limited to "true" double-byte encodings, - as it assumes an incomplete character consists of a single - byte. */ - if (curr - prev == 2) - return 1; - if (!IsDBCSLeadByteEx(code_page, *prev)) - return 1; - return 0; -} - static DWORD decode_code_page_flags(UINT code_page) { @@ -6913,7 +6891,7 @@ decode_code_page_errors(UINT code_page, PyObject **v, const char *in, const int size, - const char *errors) + const char *errors, int final) { const char *startin = in; const char *endin = in + size; @@ -6940,7 +6918,7 @@ if (encoding == NULL) return -1; - if (errors == NULL || strcmp(errors, "strict") == 0) { + if ((errors == NULL || strcmp(errors, "strict") == 0) && final) { /* The last error was ERROR_NO_UNICODE_TRANSLATION, then we raise a UnicodeDecodeError. */ make_decode_exception(&exc, encoding, in, size, 0, 0, reason); @@ -7003,6 +6981,10 @@ if (outsize <= 0) { Py_ssize_t startinpos, endinpos, outpos; + /* last character in partial decode? */ + if (in + insize >= endin && !final) + break; + startinpos = in - startin; endinpos = startinpos + 1; outpos = out - PyUnicode_AS_UNICODE(*v); @@ -7031,7 +7013,7 @@ assert(outsize <= PyUnicode_WSTR_LENGTH(*v)); if (unicode_resize(v, outsize) < 0) goto error; - ret = size; + ret = in - startin; error: Py_XDECREF(encoding_obj); @@ -7072,24 +7054,19 @@ done = 1; } - /* Skip trailing lead-byte unless 'final' is set */ - if (!final && is_dbcs_lead_byte(code_page, s, chunk_size - 1)) - --chunk_size; - if (chunk_size == 0 && done) { if (v != NULL) break; _Py_RETURN_UNICODE_EMPTY(); } - converted = decode_code_page_strict(code_page, &v, s, chunk_size); if (converted == -2) converted = decode_code_page_errors(code_page, &v, s, chunk_size, - errors); - assert(converted != 0); + errors, final); + assert(converted != 0 || done); if (converted < 0) { Py_XDECREF(v); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 17 23:17:44 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 17 Mar 2014 23:17:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320574=3A_Add_more?= =?utf-8?q?_tests_for_cp65001?= Message-ID: <3fnsMX30hwz7LjT@mail.python.org> http://hg.python.org/cpython/rev/85b87789f048 changeset: 89833:85b87789f048 user: Victor Stinner date: Mon Mar 17 23:16:02 2014 +0100 summary: Issue #20574: Add more tests for cp65001 files: Lib/test/test_codecs.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2879,5 +2879,11 @@ self.assertEqual(decoded, ('abc', 3)) + at unittest.skipUnless(sys.platform == 'win32', + 'cp65001 is specific to Windows') +class CP65001Test(UTF8Test, unittest.TestCase): + encoding = "cp65001" + + if __name__ == "__main__": unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 00:39:14 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 00:39:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320910=3A_Make_tes?= =?utf-8?q?ts_more_reliable=2C_less_dependent_on_time?= Message-ID: <3fnv9Z3gqxz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/d1c80a5b42ad changeset: 89834:d1c80a5b42ad user: Victor Stinner date: Tue Mar 18 00:39:04 2014 +0100 summary: Issue #20910: Make tests more reliable, less dependent on time * Tolerate 10 seconds instead of 3 seconds for slow test * Faster test, use sleep of 100 ms instead of 1 sec * Replace a number of iterations with an explicit deadline for the timeout files: Lib/test/fork_wait.py | 7 ++++++- Lib/test/test_docxmlrpc.py | 9 +++++---- Lib/test/test_fork1.py | 5 +++-- Lib/test/test_wait3.py | 3 ++- Lib/test/test_wait4.py | 5 +++-- 5 files changed, 19 insertions(+), 10 deletions(-) diff --git a/Lib/test/fork_wait.py b/Lib/test/fork_wait.py --- a/Lib/test/fork_wait.py +++ b/Lib/test/fork_wait.py @@ -48,7 +48,12 @@ for i in range(NUM_THREADS): _thread.start_new(self.f, (i,)) - time.sleep(LONGSLEEP) + # busy-loop to wait for threads + deadline = time.monotonic() + 10.0 + while len(self.alive) < NUM_THREADS: + time.sleep(0.1) + if time.monotonic() <= deadline: + break a = sorted(self.alive.keys()) self.assertEqual(a, list(range(NUM_THREADS))) diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -87,10 +87,11 @@ threading.Thread(target=server, args=(self.evt, 1)).start() # wait for port to be assigned - n = 1000 - while n > 0 and PORT is None: - time.sleep(0.001) - n -= 1 + deadline = time.monotonic() + 10.0 + while PORT is None: + time.sleep(0.010) + if time.monotonic() > deadline: + break self.client = http.client.HTTPConnection("localhost:%d" % PORT) diff --git a/Lib/test/test_fork1.py b/Lib/test/test_fork1.py --- a/Lib/test/test_fork1.py +++ b/Lib/test/test_fork1.py @@ -18,13 +18,14 @@ class ForkTest(ForkWait): def wait_impl(self, cpid): - for i in range(10): + deadline = time.monotonic() + 10.0 + while time.monotonic() <= deadline: # waitpid() shouldn't hang, but some of the buildbots seem to hang # in the forking tests. This is an attempt to fix the problem. spid, status = os.waitpid(cpid, os.WNOHANG) if spid == cpid: break - time.sleep(1.0) + time.sleep(0.1) self.assertEqual(spid, cpid) self.assertEqual(status, 0, "cause = %d, exit = %d" % (status&0xff, status>>8)) diff --git a/Lib/test/test_wait3.py b/Lib/test/test_wait3.py --- a/Lib/test/test_wait3.py +++ b/Lib/test/test_wait3.py @@ -18,7 +18,8 @@ # This many iterations can be required, since some previously run # tests (e.g. test_ctypes) could have spawned a lot of children # very quickly. - for i in range(30): + deadline = time.monotonic() + 10.0 + while time.monotonic() <= deadline: # wait3() shouldn't hang, but some of the buildbots seem to hang # in the forking tests. This is an attempt to fix the problem. spid, status, rusage = os.wait3(os.WNOHANG) diff --git a/Lib/test/test_wait4.py b/Lib/test/test_wait4.py --- a/Lib/test/test_wait4.py +++ b/Lib/test/test_wait4.py @@ -19,13 +19,14 @@ # Issue #11185: wait4 is broken on AIX and will always return 0 # with WNOHANG. option = 0 - for i in range(10): + deadline = time.monotonic() + 10.0 + while time.monotonic() <= deadline: # wait4() shouldn't hang, but some of the buildbots seem to hang # in the forking tests. This is an attempt to fix the problem. spid, status, rusage = os.wait4(cpid, option) if spid == cpid: break - time.sleep(1.0) + time.sleep(0.1) self.assertEqual(spid, cpid) self.assertEqual(status, 0, "cause = %d, exit = %d" % (status&0xff, status>>8)) self.assertTrue(rusage) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 01:27:01 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 01:27:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_a_stub_=22whatsnew_in_?= =?utf-8?q?3=2E5=22_document=2E?= Message-ID: <3fnwDj498Xz7LjW@mail.python.org> http://hg.python.org/cpython/rev/daa6bf71170f changeset: 89835:daa6bf71170f user: Victor Stinner date: Tue Mar 18 00:53:32 2014 +0100 summary: Add a stub "whatsnew in 3.5" document. files: Doc/whatsnew/3.5.rst | 186 +++++++++++++++++++++++++++++++ 1 files changed, 186 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,186 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. + +For full details, see the :source:`Misc/NEWS` file. + +.. note:: Prerelease users should be aware that this document is currently in + draft form. It will be updated substantially as Python 3.5 moves towards + release, so it's worth checking back even after reading earlier versions. + + +.. seealso:: + + .. :pep:`4XX` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.3. + Brevity is key. + +New syntax features: + +* None yet. + +New library modules: + +* None yet. + +New built-in features: + +* None yet. + +Implementation improvements: + +* None yet. + +Significantly Improved Library Modules: + +* None yet. + +Security improvements: + +* None yet. + +Please read on for a comprehensive list of user-facing changes. + + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + + + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* None yet. + + + +New Modules +=========== + +.. module name +.. ----------- + +* None yet. + + +Improved Modules +================ + +* None yet. + + +Optimizations +============= + +Major performance enhancements have been added: + +* None yet. + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* None yet. + + +Deprecated +========== + +Unsupported Operating Systems +----------------------------- + +* None yet. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* None yet. + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* None yet. + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +* Nothing yet. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 01:27:03 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 01:27:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319977=3A_When_the?= =?utf-8?b?IGBgTENfVFlQRWBgIGxvY2FsZSBpcyB0aGUgUE9TSVggbG9jYWxlIChgYENg?= =?utf-8?q?=60_locale=29=2C?= Message-ID: <3fnwDl0CRBz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/bc06f67234d0 changeset: 89836:bc06f67234d0 user: Victor Stinner date: Tue Mar 18 01:18:21 2014 +0100 summary: Issue #19977: When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the ``surrogateescape`` error handler, instead of the ``strict`` error handler. files: Doc/whatsnew/3.5.rst | 5 ++- Lib/test/test_sys.py | 44 ++++++++++++++++++++++++++++++++ Misc/NEWS | 4 ++ Python/pythonrun.c | 11 +++++++- 4 files changed, 62 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -79,7 +79,10 @@ Implementation improvements: -* None yet. +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the + ``surrogateescape`` error handler, instead of the ``strict`` error handler + (:issue:`19977`). Significantly Improved Library Modules: diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -615,6 +615,50 @@ expected = None self.check_fsencoding(fs_encoding, expected) + @unittest.skipIf(sys.platform == 'win32', + 'test specific to UNIX') + def test_c_locale_surrogateescape(self): + # Force the POSIX locale + env = os.environ.copy() + env["LC_ALL"] = "C" + code = '\n'.join(( + 'import codecs, sys', + 'def dump(name):', + ' std = getattr(sys, name)', + ' encoding = codecs.lookup(std.encoding).name', + ' print("%s: %s:%s" % (name, encoding, std.errors))', + 'dump("stdin")', + 'dump("stdout")', + 'dump("stderr")', + )) + p = subprocess.Popen([sys.executable, "-I", "-c", code], + stdout=subprocess.PIPE, env=env) + out = p.communicate()[0] + self.assertEqual(out, + b'stdin: ascii:surrogateescape\n' + b'stdout: ascii:surrogateescape\n' + b'stderr: ascii:backslashreplace\n') + + # replace the default error handler + env['PYTHONIOENCODING'] = ':strict' + p = subprocess.Popen([sys.executable, "-c", code], + stdout=subprocess.PIPE, env=env) + out = p.communicate()[0] + self.assertEqual(out, + b'stdin: ascii:strict\n' + b'stdout: ascii:strict\n' + b'stderr: ascii:backslashreplace\n') + + # force the encoding + env['PYTHONIOENCODING'] = 'iso8859-1' + p = subprocess.Popen([sys.executable, "-c", code], + stdout=subprocess.PIPE, env=env) + out = p.communicate()[0] + self.assertEqual(out, + b'stdin: iso8859-1:surrogateescape\n' + b'stdout: iso8859-1:surrogateescape\n' + b'stderr: iso8859-1:backslashreplace\n') + def test_implementation(self): # This test applies to all implementations equally. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,10 @@ Library ------- +- Issue #19977: When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the + ``surrogateescape`` error handler, instead of the ``strict`` error handler. + - Issue #20574: Implement incremental decoder for cp65001 code (Windows code page 65001, Microsoft UTF-8). diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1156,6 +1156,15 @@ encoding = _Py_StandardStreamEncoding; errors = _Py_StandardStreamErrors; if (!encoding || !errors) { + if (!errors) { + /* When the LC_CTYPE locale is the POSIX locale ("C locale"), + stdin and stdout use the surrogateescape error handler by + default, instead of the strict error handler. */ + char *loc = setlocale(LC_CTYPE, NULL); + if (loc != NULL && strcmp(loc, "C") == 0) + errors = "surrogateescape"; + } + pythonioencoding = Py_GETENV("PYTHONIOENCODING"); if (pythonioencoding) { char *err; @@ -1168,7 +1177,7 @@ if (err) { *err = '\0'; err++; - if (*err && !errors) { + if (*err && !_Py_StandardStreamErrors) { errors = err; } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 01:40:32 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 01:40:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320574=3A_Remove_d?= =?utf-8?q?uplicated_test_failing_on_Windows_XP?= Message-ID: <3fnwXJ51LWz7Ll7@mail.python.org> http://hg.python.org/cpython/rev/f6794a0fb2b3 changeset: 89837:f6794a0fb2b3 user: Victor Stinner date: Tue Mar 18 01:40:22 2014 +0100 summary: Issue #20574: Remove duplicated test failing on Windows XP files: Lib/test/test_codecs.py | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2879,11 +2879,5 @@ self.assertEqual(decoded, ('abc', 3)) - at unittest.skipUnless(sys.platform == 'win32', - 'cp65001 is specific to Windows') -class CP65001Test(UTF8Test, unittest.TestCase): - encoding = "cp65001" - - if __name__ == "__main__": unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 02:13:25 2014 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 18 Mar 2014 02:13:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Typo_fix?= Message-ID: <3fnxGF2LQTz7LjQ@mail.python.org> http://hg.python.org/peps/rev/d8bd02f83d16 changeset: 5417:d8bd02f83d16 user: Andrew Kuchling date: Mon Mar 17 21:13:16 2014 -0400 summary: Typo fix files: pep-0398.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -94,7 +94,7 @@ 3.3.5 schedule -------------- -Python 3.3.5 was be the last regular maintenance release before 3.3 entered +Python 3.3.5 was the last regular maintenance release before 3.3 entered security-fix only mode. - 3.3.5 candidate 1: February 22, 2014 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 18 02:32:21 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 02:32:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319977=2C_=2319036?= =?utf-8?q?=3A_Always_include_=3Clocale=2Eh=3E_in_pythonrun=2Ec?= Message-ID: <3fnxh53jFWzSQf@mail.python.org> http://hg.python.org/cpython/rev/3589980c98de changeset: 89838:3589980c98de user: Victor Stinner date: Tue Mar 18 02:06:38 2014 +0100 summary: Issue #19977, #19036: Always include in pythonrun.c to get LC_CTYPE constant on Windows. files: Python/pythonrun.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -15,6 +15,7 @@ #include "ast.h" #include "marshal.h" #include "osdefs.h" +#include #ifdef HAVE_SIGNAL_H #include @@ -25,7 +26,6 @@ #endif #ifdef HAVE_LANGINFO_H -#include #include #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 02:32:22 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 02:32:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319977=3A_Enable_t?= =?utf-8?q?est=5Fc=5Flocale=5Fsurrogateescape=28=29_on_Windows?= Message-ID: <3fnxh66q0mzSQf@mail.python.org> http://hg.python.org/cpython/rev/94d5025c70a3 changeset: 89839:94d5025c70a3 user: Victor Stinner date: Tue Mar 18 02:28:10 2014 +0100 summary: Issue #19977: Enable test_c_locale_surrogateescape() on Windows Only test the error handler. The encoding is not ASCII on Windows: it may the OEM or ANSI code page. files: Lib/test/test_sys.py | 55 ++++++++++++++++--------------- 1 files changed, 29 insertions(+), 26 deletions(-) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -615,49 +615,52 @@ expected = None self.check_fsencoding(fs_encoding, expected) - @unittest.skipIf(sys.platform == 'win32', - 'test specific to UNIX') - def test_c_locale_surrogateescape(self): + def c_locale_get_error_handler(self, isolated=False, encoding=None): # Force the POSIX locale env = os.environ.copy() env["LC_ALL"] = "C" code = '\n'.join(( - 'import codecs, sys', + 'import sys', 'def dump(name):', ' std = getattr(sys, name)', - ' encoding = codecs.lookup(std.encoding).name', - ' print("%s: %s:%s" % (name, encoding, std.errors))', + ' print("%s: %s" % (name, std.errors))', 'dump("stdin")', 'dump("stdout")', 'dump("stderr")', )) - p = subprocess.Popen([sys.executable, "-I", "-c", code], - stdout=subprocess.PIPE, env=env) - out = p.communicate()[0] + args = [sys.executable, "-c", code] + if isolated: + args.append("-I") + elif encoding: + env['PYTHONIOENCODING'] = encoding + p = subprocess.Popen(args, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + env=env, + universal_newlines=True) + stdout, stderr = p.communicate() + return stdout + + def test_c_locale_surrogateescape(self): + out = self.c_locale_get_error_handler(isolated=True) self.assertEqual(out, - b'stdin: ascii:surrogateescape\n' - b'stdout: ascii:surrogateescape\n' - b'stderr: ascii:backslashreplace\n') + 'stdin: surrogateescape\n' + 'stdout: surrogateescape\n' + 'stderr: backslashreplace\n') # replace the default error handler - env['PYTHONIOENCODING'] = ':strict' - p = subprocess.Popen([sys.executable, "-c", code], - stdout=subprocess.PIPE, env=env) - out = p.communicate()[0] + out = self.c_locale_get_error_handler(encoding=':strict') self.assertEqual(out, - b'stdin: ascii:strict\n' - b'stdout: ascii:strict\n' - b'stderr: ascii:backslashreplace\n') + 'stdin: strict\n' + 'stdout: strict\n' + 'stderr: backslashreplace\n') # force the encoding - env['PYTHONIOENCODING'] = 'iso8859-1' - p = subprocess.Popen([sys.executable, "-c", code], - stdout=subprocess.PIPE, env=env) - out = p.communicate()[0] + out = self.c_locale_get_error_handler(encoding='iso8859-1') self.assertEqual(out, - b'stdin: iso8859-1:surrogateescape\n' - b'stdout: iso8859-1:surrogateescape\n' - b'stderr: iso8859-1:backslashreplace\n') + 'stdin: surrogateescape\n' + 'stdout: surrogateescape\n' + 'stderr: backslashreplace\n') def test_implementation(self): # This test applies to all implementations equally. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 02:38:22 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 02:38:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319977=3A_Fix_test?= =?utf-8?q?=5Fcapi_when_LC=5FCTYPE_locale_is_POSIX?= Message-ID: <3fnxq20GGqz7LjW@mail.python.org> http://hg.python.org/cpython/rev/c9905e802042 changeset: 89840:c9905e802042 user: Victor Stinner date: Tue Mar 18 02:38:12 2014 +0100 summary: Issue #19977: Fix test_capi when LC_CTYPE locale is POSIX files: Lib/test/test_capi.py | 32 +++++++++++++++++------------- Modules/_testembed.c | 4 +- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -319,34 +319,38 @@ print() print(out) print(err) + expected_errors = sys.__stdout__.errors expected_stdin_encoding = sys.__stdin__.encoding expected_pipe_encoding = self._get_default_pipe_encoding() expected_output = os.linesep.join([ "--- Use defaults ---", "Expected encoding: default", "Expected errors: default", - "stdin: {0}:strict", - "stdout: {1}:strict", - "stderr: {1}:backslashreplace", + "stdin: {in_encoding}:{errors}", + "stdout: {out_encoding}:{errors}", + "stderr: {out_encoding}:backslashreplace", "--- Set errors only ---", "Expected encoding: default", - "Expected errors: surrogateescape", - "stdin: {0}:surrogateescape", - "stdout: {1}:surrogateescape", - "stderr: {1}:backslashreplace", + "Expected errors: ignore", + "stdin: {in_encoding}:ignore", + "stdout: {out_encoding}:ignore", + "stderr: {out_encoding}:backslashreplace", "--- Set encoding only ---", "Expected encoding: latin-1", "Expected errors: default", - "stdin: latin-1:strict", - "stdout: latin-1:strict", + "stdin: latin-1:{errors}", + "stdout: latin-1:{errors}", "stderr: latin-1:backslashreplace", "--- Set encoding and errors ---", "Expected encoding: latin-1", - "Expected errors: surrogateescape", - "stdin: latin-1:surrogateescape", - "stdout: latin-1:surrogateescape", - "stderr: latin-1:backslashreplace"]).format(expected_stdin_encoding, - expected_pipe_encoding) + "Expected errors: replace", + "stdin: latin-1:replace", + "stdout: latin-1:replace", + "stderr: latin-1:backslashreplace"]) + expected_output = expected_output.format( + in_encoding=expected_stdin_encoding, + out_encoding=expected_pipe_encoding, + errors=expected_errors) # This is useful if we ever trip over odd platform behaviour self.maxDiff = None self.assertEqual(out.strip(), expected_output) diff --git a/Modules/_testembed.c b/Modules/_testembed.c --- a/Modules/_testembed.c +++ b/Modules/_testembed.c @@ -109,11 +109,11 @@ printf("--- Use defaults ---\n"); check_stdio_details(NULL, NULL); printf("--- Set errors only ---\n"); - check_stdio_details(NULL, "surrogateescape"); + check_stdio_details(NULL, "ignore"); printf("--- Set encoding only ---\n"); check_stdio_details("latin-1", NULL); printf("--- Set encoding and errors ---\n"); - check_stdio_details("latin-1", "surrogateescape"); + check_stdio_details("latin-1", "replace"); /* Check calling after initialization fails */ Py_Initialize(); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 03:33:55 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 18 Mar 2014 03:33:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_updates_to_the_clang_docs?= =?utf-8?q?_from_Jeffrey_Walton_=28closes_=2320965=29?= Message-ID: <3fnz371m98z7LjW@mail.python.org> http://hg.python.org/devguide/rev/5746108c02c0 changeset: 679:5746108c02c0 user: Benjamin Peterson date: Mon Mar 17 21:33:25 2014 -0500 summary: updates to the clang docs from Jeffrey Walton (closes #20965) files: clang.rst | 142 ++++++++++++++++++++++++++++++++--------- 1 files changed, 110 insertions(+), 32 deletions(-) diff --git a/clang.rst b/clang.rst --- a/clang.rst +++ b/clang.rst @@ -7,11 +7,19 @@ downloading, building and installing the the latest Clang/LLVM combination (which is currently 3.4). +This document does not cover interpreting the findings. For a discussion of +interpreting results, see Marshall Clow's `Testing libc++ with +-fsanitize=undefined `_. The +blog posting is a detailed examinations of issues uncovered by Clang in +``libc++``. + What is Clang? ============== Clang is the C, C++ and Objective C front-end for the LLVM compiler. The -front-end provides access to LLVM's optimizer and code generator. +front-end provides access to LLVM's optimizer and code generator. The +sanitizers - or checkers - are hooks into the code generation phase to +instrument compiled code so suspicious behavior is flagged. What are Sanitizers? ==================== @@ -21,13 +29,25 @@ positives are kept to a minimum. There are a number of sanitizers available, but two that should be used on a -regular basis are the Address Sanitizer (or Asan) and the Undefined Behavior -Sanitizer (or UBsan). Asan is invoked with the compiler option -``-fsanitize=address``, and UBsan is invoked with ``-fsanitize=undefined``. +regular basis are the Address Sanitizer (or ASan) and the Undefined Behavior +Sanitizer (or UBSan). ASan is invoked with the compiler option +``-fsanitize=address``, and UBSan is invoked with ``-fsanitize=undefined``. The +flags are passed through ``CFLAGS`` and ``CXXFLAGS``, and sometimes through +``CC`` and ``CXX` (in addition to the compiler). A complete list of sanitizers can be found at `Controlling Code Generation `_. +.. note:: + + Because sanitizers operate at runtime on real program parameters, its + important to provide a complete set of positive and negative self tests. + +Clang and its sanitizers have strengths (and weaknesses). Its just one tool in +the war chest to uncovering bugs and improving code quality. Clang should be +used to compliment other methods, including Code Reviews, Valgrind, Coverity, +etc. + Clang/LLVM Setup ================ @@ -71,12 +91,10 @@ If you receive an error ``'LibraryDependencies.inc' file not found``, then ensure you are utilizing Python 2 and not Python 3. If you encounter the - error after switching to Python 2, then delete everything and start over. If - the issue still persists, then see - http://llvm.org/bugs/show_bug.cgi?id=19158. + error after switching to Python 2, then delete everything and start over. After ``make install`` executes, the compilers will be installed in -``/usr/local/bin`` and the various librarioes will be installed in +``/usr/local/bin`` and the various libraries will be installed in ``/usr/local/lib/clang/3.4/lib/linux/``: :: $ ls /usr/local/lib/clang/3.4/lib/linux/ @@ -86,8 +104,25 @@ libclang_rt.lsan-x86_64.a libclang_rt.ubsan_cxx-x86_64.a libclang_rt.msan-x86_64.a libclang_rt.ubsan-x86_64.a -You should never have to add the libraries to a project. Clang should handle it -for you. +On Mac OS X, the libraries are installed in +``/usr/local/lib/clang/3.3/lib/darwin/``: :: + + $ ls /usr/local/lib/clang/3.3/lib/darwin/ + libclang_rt.10.4.a libclang_rt.ios.a + libclang_rt.asan_osx.a libclang_rt.osx.a + libclang_rt.asan_osx_dynamic.dylib libclang_rt.profile_ios.a + libclang_rt.cc_kext.a libclang_rt.profile_osx.a + libclang_rt.cc_kext_ios5.a libclang_rt.ubsan_osx.a + libclang_rt.eprintf.a + +.. note:: + + You should never have to add the libraries to a project. Clang will handle + it for you. If you find you cannot pass the ``-fsanitize=XXX`` flag through + ``make``'s implicit variables (``CFLAGS``, ``CXXFLAGS``, ``CC``, + ``CXXFLAGS``, ``LDFLAGS``) during ``configure``, then you should modify the + makefile after configuring to ensure the flag is passed through the + compiler. The installer does not install all the components needed on occasion. For example, you might want to run a ``scan-build`` or examine the results with @@ -110,7 +145,7 @@ This portion of the document covers invoking Clang and LLVM with the options required so the sanitizers analyze Python with under its test suite. Two -checkers are used - Asan and UBsan. +checkers are used - ASan and UBSan. Because the sanitizers are runtime checkers, its best to have as many positive and negative self tests as possible. You can never have enough self tests. @@ -119,10 +154,10 @@ Clang will include the needed runtime libraries. However, you can't use ``CFLAGS`` and ``CXXFLAGS`` to pass the options through the compiler to the linker because the makefile rules for ``BUILDPYTHON``, ``_testembed`` and -``_freeze_importlib`` don't use the flags. +``_freeze_importlib`` don't use the implicit variables. As a workaround to the absence of flags to the linker, you can pass the -sanitizer options by way of the compilers - ``CC`` and ``CXX`. Passing the +sanitizer options by way of the compilers - ``CC`` and ``CXX``. Passing the flags though the compiler is used below, but passing them through ``LDFLAGS`` is also reported to work. @@ -132,31 +167,19 @@ To begin, export the variables of interest with the desired sanitizers. Its OK to specify both sanitizers: :: - # Asan + # ASan export CC="/usr/local/bin/clang -fsanitize=address" export CXX="/usr/local/bin/clang++ -fsanitize=address -fno-sanitize=vptr" -Or: +Or: :: - # UBsan + # UBSan export CC="/usr/local/bin/clang -fsanitize=undefined" export CXX="/usr/local/bin/clang++ -fsanitize=undefined -fno-sanitize=vptr" -The ``-fno-sanitize=vptr`` removes vtable checks that are part of UBsan from C++ -projects due to noise. - -Clang also accepts a blacklist of functions to ignore. There is no documentation -on the blacklist, but the check-in is available at -http://thread.gmane.org/gmane.comp.compilers.clang.scm/62388. You specify the -blacklist with ``-fsanitize-blacklist=XXX``. For example: :: - - -fsanitize-blacklist=my_blacklist.txt - -``my_blacklist.txt`` would then contain entries such as: :: - - fun:_Ios_Fmtflags - -Unfortunately, you won't know what to blacklist until you run the sanitizer. +The ``-fno-sanitize=vptr`` removes vtable checks that are part of UBSan from C++ +projects due to noise. Its not needed with Python, but you will likely need it +for other C++ projects. After exporting ``CC`` and ``CXX``, ``configure`` as normal: :: @@ -200,7 +223,10 @@ ``asan_symbolize.py`` to get a good trace. For example, from Issue 20953 during compile (formatting added for clarity): :: - $ /usr/local/bin/clang -fsanitize=address -Xlinker -export-dynamic + $ make test 2>&1 | asan_symbolize.py + ... + + /usr/local/bin/clang -fsanitize=address -Xlinker -export-dynamic -o python Modules/python.o libpython3.3m.a -ldl -lutil /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -lm ./python -E -S -m sysconfig --generate-posix-vars @@ -258,3 +284,55 @@ ==24064==ABORTING make: *** [pybuilddir.txt] Error 1 +.. note:: + + ``asan_symbolize.py`` is supposed to be installed during ``make install``. + If its not installed, then look in the Clang/LLVM build directory for it and + copy it to ``/usr/local/bin``. + +Blacklisting (Ignoring) Findings +-------------------------------- + +Clang allows you to alter the behavior of sanitizer tools for certain +source-level by providing a special blacklist file at compile-time. The +blacklist is needed because it reports every instance of an issue, even if the +issue is reported 10's of thousands of time in un-managed library code. + +You specify the blacklist with ``-fsanitize-blacklist=XXX``. For example: :: + + -fsanitize-blacklist=my_blacklist.txt + +``my_blacklist.txt`` would then contain entries such as the following. The entry +will ignore a bug in ``libc++``'s ``ios`` formatting functions: :: + + fun:_Ios_Fmtflags + +As an example with Pyhton 3.4.0, ``audioop.c`` will produce a number of +findings: :: + + ./Modules/audioop.c:422:11: runtime error: left shift of negative value -1 + ./Modules/audioop.c:446:19: runtime error: left shift of negative value -1 + ./Modules/audioop.c:476:19: runtime error: left shift of negative value -1 + ./Modules/audioop.c:504:16: runtime error: left shift of negative value -1 + ./Modules/audioop.c:533:22: runtime error: left shift of negative value -128 + ./Modules/audioop.c:775:19: runtime error: left shift of negative value -70 + ./Modules/audioop.c:831:19: runtime error: left shift of negative value -70 + ./Modules/audioop.c:881:19: runtime error: left shift of negative value -1 + ./Modules/audioop.c:920:22: runtime error: left shift of negative value -70 + ./Modules/audioop.c:967:23: runtime error: left shift of negative value -70 + ./Modules/audioop.c:968:23: runtime error: left shift of negative value -70 + ... + +One of the function of interest is ``audioop_getsample_impl`` (flagged at line +422), and the blacklist entry would include: :: + + fun:audioop_getsample_imp + +Or, you could ignore the entire file with: :: + + src:Modules/audioop.c + +Unfortunately, you won't know what to blacklist until you run the sanitizer. + +The documentation is available at `Sanitizer special case list +`_. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Tue Mar 18 03:34:16 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 18 Mar 2014 03:34:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_fix_typo?= Message-ID: <3fnz3X3FJLz7LjW@mail.python.org> http://hg.python.org/devguide/rev/ddc5ace26edd changeset: 680:ddc5ace26edd user: Benjamin Peterson date: Mon Mar 17 21:33:59 2014 -0500 summary: fix typo files: clang.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/clang.rst b/clang.rst --- a/clang.rst +++ b/clang.rst @@ -307,7 +307,7 @@ fun:_Ios_Fmtflags -As an example with Pyhton 3.4.0, ``audioop.c`` will produce a number of +As an example with Python 3.4.0, ``audioop.c`` will produce a number of findings: :: ./Modules/audioop.c:422:11: runtime error: left shift of negative value -1 -- Repository URL: http://hg.python.org/devguide From root at python.org Tue Mar 18 03:35:04 2014 From: root at python.org (Cron Daemon) Date: Tue, 18 Mar 2014 03:35:04 +0100 Subject: [Python-checkins] Cron /home/docs/build-devguide Message-ID: /home/docs/devguide/clang.rst:31: WARNING: Inline literal start-string without end-string. From python-checkins at python.org Tue Mar 18 03:36:08 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 18 Mar 2014 03:36:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_add_missing_tick?= Message-ID: <3fnz5h6YbPz7LjW@mail.python.org> http://hg.python.org/devguide/rev/85f290e474e2 changeset: 681:85f290e474e2 user: Benjamin Peterson date: Mon Mar 17 21:35:56 2014 -0500 summary: add missing tick files: clang.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/clang.rst b/clang.rst --- a/clang.rst +++ b/clang.rst @@ -33,7 +33,7 @@ Sanitizer (or UBSan). ASan is invoked with the compiler option ``-fsanitize=address``, and UBSan is invoked with ``-fsanitize=undefined``. The flags are passed through ``CFLAGS`` and ``CXXFLAGS``, and sometimes through -``CC`` and ``CXX` (in addition to the compiler). +``CC`` and ``CXX`` (in addition to the compiler). A complete list of sanitizers can be found at `Controlling Code Generation `_. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Tue Mar 18 07:48:01 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 18 Mar 2014 07:48:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_option_des?= =?utf-8?q?cription_that_is_a_warning_in_new_Sphinx_versions=2E?= Message-ID: <3fp4hK2pRBz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/f8dbec87dbfe changeset: 89841:f8dbec87dbfe branch: 3.4 parent: 89830:7093d5758954 user: Georg Brandl date: Tue Mar 18 07:44:07 2014 +0100 summary: Fix option description that is a warning in new Sphinx versions. files: Doc/library/compileall.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -20,7 +20,8 @@ .. program:: compileall -.. cmdoption:: [directory|file]... +.. cmdoption:: directory ... + file ... Positional arguments are files to compile or directories that contain source files, traversed recursively. If no argument is given, behave as if -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 07:48:02 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 18 Mar 2014 07:48:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3fp4hL4cLxz7LjT@mail.python.org> http://hg.python.org/cpython/rev/b31441753abc changeset: 89842:b31441753abc parent: 89840:c9905e802042 parent: 89841:f8dbec87dbfe user: Georg Brandl date: Tue Mar 18 07:47:35 2014 +0100 summary: merge with 3.4 files: Doc/library/compileall.rst | 3 ++- Doc/tools/sphinxext/susp-ignored.csv | 5 ----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -20,7 +20,8 @@ .. program:: compileall -.. cmdoption:: [directory|file]... +.. cmdoption:: directory ... + file ... Positional arguments are files to compile or directories that contain source files, traversed recursively. If no argument is given, behave as if diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -276,10 +276,5 @@ whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:platform,:platform: whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused" -whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -whatsnew/changelog,,:password,user:password -whatsnew/changelog,,:gz,w:gz -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 08:59:52 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 08:59:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Restore_removed_NEWS_items?= =?utf-8?q?_3=2E4=2E1rc1_into_3=2E5=2E0alpha1_sections?= Message-ID: <3fp6HD5sLjz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/b7f2b1bd49f3 changeset: 89843:b7f2b1bd49f3 user: Victor Stinner date: Tue Mar 18 08:59:30 2014 +0100 summary: Restore removed NEWS items 3.4.1rc1 into 3.5.0alpha1 sections files: Misc/NEWS | 60 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 60 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,14 @@ Core and Builtins ----------------- +- Issue #20929: Add a type cast to avoid shifting a negative number. + +- Issue #20731: Properly position in source code files even if they + are opened in text mode. Patch by Serhiy Storchaka. + +- Issue #20637: Key-sharing now also works for instance dictionaries of + subclasses. Patch by Peter Ingebretson. + Library ------- @@ -25,6 +33,58 @@ initialization of the unquote_to_bytes() table of the urllib.parse module, to not waste memory if these modules are not used. +- Issue #19157: Include the broadcast address in the usuable hosts for IPv6 + in ipaddress. + +- Issue #11599: When an external command (e.g. compiler) fails, distutils now + prints out the whole command line (instead of just the command name) if the + environment variable DISTUTILS_DEBUG is set. + +- Issue #4931: distutils should not produce unhelpful "error: None" messages + anymore. distutils.util.grok_environment_error is kept but doc-deprecated. + +- Issue #20875: Prevent possible gzip "'read' is not defined" NameError. + Patch by Claudiu Popa. + +- Issue #11558: ``email.message.Message.attach`` now returns a more + useful error message if ``attach`` is called on a message for which + ``is_multipart`` is False. + +- Issue #20283: RE pattern methods now accept the string keyword parameters + as documented. The pattern and source keyword parameters are left as + deprecated aliases. + +- Issue #20778: Fix modulefinder to work with bytecode-only modules. + +- Issue #20791: copy.copy() now doesn't make a copy when the input is + a bytes object. Initial patch by Peter Otten. + +- Issue #19748: On AIX, time.mktime() now raises an OverflowError for year + outsize range [1902; 2037]. + +Documentation +------------- + +- Issue #20765: Add missing documentation for PurePath.with_name() and + PurePath.with_suffix(). + +- Issue #19407: New package installation and distribution guides based on + the Python Packaging Authority tools. Existing guides have been retained + as legacy links from the distutils docs, as they still contain some + required reference material for tool developers that isn't recorded + anywhere else. + +Tests +----- + +- Issue #20743: Fix a reference leak in test_tcl. + +Tools/Demos +----------- + +- Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. + Patch by Arfrever Frehtes Taifersar Arahesis. + What's New in Python 3.4.0? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 09:01:29 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 18 Mar 2014 09:01:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Doc=3A_add_What=27s_New_in?= =?utf-8?q?_Python_3=2E5_to_whatsnew_index?= Message-ID: <3fp6K55vTyz7LlY@mail.python.org> http://hg.python.org/cpython/rev/1b096c601d84 changeset: 89844:1b096c601d84 user: Victor Stinner date: Tue Mar 18 09:01:21 2014 +0100 summary: Doc: add What's New in Python 3.5 to whatsnew index files: Doc/whatsnew/index.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Mar 18 09:57:30 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 18 Mar 2014 09:57:30 +0100 Subject: [Python-checkins] Daily reference leaks (c9905e802042): sum=0 Message-ID: results for c9905e802042 on branch "default" -------------------------------------------- test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogqDE01d', '-x'] From python-checkins at python.org Tue Mar 18 15:20:47 2014 From: python-checkins at python.org (zach.ware) Date: Tue, 18 Mar 2014 15:20:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwOTY2?= =?utf-8?q?=3A_Fix_Tkinter_Resources_link?= Message-ID: <3fpGkl0Sg4z7LjV@mail.python.org> http://hg.python.org/cpython/rev/be1e015a8405 changeset: 89845:be1e015a8405 branch: 2.7 parent: 89807:4495a9b11137 user: Zachary Ware date: Tue Mar 18 09:18:53 2014 -0500 summary: Issue #20966: Fix Tkinter Resources link files: Doc/library/tkinter.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -19,7 +19,7 @@ .. seealso:: - `Python Tkinter Resources `_ + `Python Tkinter Resources `_ The Python Tkinter Topic Guide provides a great deal of information on using Tk from Python and links to other sources of information on Tk. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 15:20:48 2014 From: python-checkins at python.org (zach.ware) Date: Tue, 18 Mar 2014 15:20:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTY2?= =?utf-8?q?=3A_Fix_Tkinter_Resources_link?= Message-ID: <3fpGkm41Kxz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/f28f63c5d30a changeset: 89846:f28f63c5d30a branch: 3.4 parent: 89841:f8dbec87dbfe user: Zachary Ware date: Tue Mar 18 09:19:18 2014 -0500 summary: Issue #20966: Fix Tkinter Resources link files: Doc/library/tkinter.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -15,7 +15,7 @@ .. seealso:: - `Python Tkinter Resources `_ + `Python Tkinter Resources `_ The Python Tkinter Topic Guide provides a great deal of information on using Tk from Python and links to other sources of information on Tk. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 15:20:53 2014 From: python-checkins at python.org (zach.ware) Date: Tue, 18 Mar 2014 15:20:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2320966=3A_Fix_Tkinter_Resources_link_=28merge_f?= =?utf-8?q?rom_3=2E4=29?= Message-ID: <3fpGks3wwFz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/f8b40d33e45d changeset: 89847:f8b40d33e45d parent: 89844:1b096c601d84 parent: 89846:f28f63c5d30a user: Zachary Ware date: Tue Mar 18 09:20:25 2014 -0500 summary: Closes #20966: Fix Tkinter Resources link (merge from 3.4) files: Doc/library/tkinter.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -15,7 +15,7 @@ .. seealso:: - `Python Tkinter Resources `_ + `Python Tkinter Resources `_ The Python Tkinter Topic Guide provides a great deal of information on using Tk from Python and links to other sources of information on Tk. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 18 19:07:59 2014 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 18 Mar 2014 19:07:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_New_version_of_pep_465_=28mat?= =?utf-8?q?rix_multiply_=40_operator=29=2E_Added_scan-ops=2Epy=2E?= Message-ID: <3fpMmv2t6Jz7Ljb@mail.python.org> http://hg.python.org/peps/rev/9d5b65530610 changeset: 5418:9d5b65530610 user: Guido van Rossum date: Tue Mar 18 11:07:32 2014 -0700 summary: New version of pep 465 (matrix multiply @ operator). Added scan-ops.py. files: pep-0465.txt | 323 ++++++++++++++++++++------------------ scan-ops.py | 132 +++++++++++++++ 2 files changed, 305 insertions(+), 150 deletions(-) diff --git a/pep-0465.txt b/pep-0465.txt --- a/pep-0465.txt +++ b/pep-0465.txt @@ -1,5 +1,5 @@ PEP: 465 -Title: Dedicated infix operators for matrix multiplication and matrix power +Title: A dedicated infix operator for matrix multiplication Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith @@ -13,30 +13,31 @@ Abstract ======== -This PEP proposes two new binary operators dedicated to matrix -multiplication and matrix power, spelled ``@`` and ``@@`` -respectively. (Mnemonic: ``@`` is ``*`` for mATrices.) +This PEP proposes a new binary operator to be used for matrix +multiplication, called ``@``. (Mnemonic: ``@`` is ``*`` for +mATrices.) Specification ============= -Two new binary operators are added to the Python language, together -with corresponding in-place versions: +A new binary operator is added to the Python language, together +with the corresponding in-place version: ======= ========================= =============================== Op Precedence/associativity Methods ======= ========================= =============================== -``@`` Same as ``*`` ``__matmul__``, ``__rmatmul__`` -``@@`` Same as ``**`` ``__matpow__``, ``__rmatpow__`` +``@`` *To be determined* ``__matmul__``, ``__rmatmul__`` ``@=`` n/a ``__imatmul__`` -``@@=`` n/a ``__imatpow__`` ======= ========================= =============================== No implementations of these methods are added to the builtin or standard library types. However, a number of projects have reached consensus on the recommended semantics for these operations; see -`Intended usage details`_ below. +`Intended usage details`_ below for details. + +For details on how this operator will be implemented in CPython, see +`Implementation details`_. Motivation @@ -90,9 +91,6 @@ and finally standardize on a single consensus duck type for all numerical array objects. -And, given the existence of ``@``, it makes more sense than not to -have ``@@``, ``@=``, and ``@@=``, so they are added as well. - Background: What's wrong with the status quo? --------------------------------------------- @@ -138,12 +136,13 @@ Matrix multiplication is more of a special case. It's only defined on 2d arrays (also known as "matrices"), and multiplication is the only -operation that has a meaningful "matrix" version -- "matrix addition" +operation that has an important "matrix" version -- "matrix addition" is the same as elementwise addition; there is no such thing as "matrix -bitwise-or" or "matrix floordiv"; "matrix division" can be defined but -is not very useful, etc. However, matrix multiplication is still used -very heavily across all numerical application areas; mathematically, -it's one of the most fundamental operations there is. +bitwise-or" or "matrix floordiv"; "matrix division" and "matrix +to-the-power-of" can be defined but are not very useful, etc. +However, matrix multiplication is still used very heavily across all +numerical application areas; mathematically, it's one of the most +fundamental operations there is. Because Python syntax currently allows for only a single multiplication operator ``*``, libraries providing array-like objects @@ -533,37 +532,22 @@ But isn't it weird to add an operator with no stdlib uses? ---------------------------------------------------------- -It's certainly unusual (though ``Ellipsis`` was also added without any -stdlib uses). But the important thing is whether a change will -benefit users, not where the software is being downloaded from. It's -clear from the above that ``@`` will be used, and used heavily. And -this PEP provides the critical piece that will allow the Python +It's certainly unusual (though extended slicing existed for some time +builtin types gained support for it, ``Ellipsis`` is still unused +within the stdlib, etc.). But the important thing is whether a change +will benefit users, not where the software is being downloaded from. +It's clear from the above that ``@`` will be used, and used heavily. +And this PEP provides the critical piece that will allow the Python numerical community to finally reach consensus on a standard duck type for all array-like objects, which is a necessary precondition to ever adding a numerical array type to the stdlib. -Matrix power and in-place operators ------------------------------------ - -The primary motivation for this PEP is ``@``; the other proposed -operators don't have nearly as much impact. The matrix power operator -``@@`` is useful and well-defined, but not really necessary. It is -still included, though, for consistency: if we have an ``@`` that is -analogous to ``*``, then it would be weird and surprising to *not* -have an ``@@`` that is analogous to ``**``. Similarly, the in-place -operators ``@=`` and ``@@=`` provide limited value -- it's more common -to write ``a = (b @ a)`` than it is to write ``a = (a @ b)``, and -in-place matrix operations still generally have to allocate -substantial temporary storage -- but they are included for -completeness and symmetry. - - Compatibility considerations ============================ Currently, the only legal use of the ``@`` token in Python code is at -statement beginning in decorators. The new operators are all infix; +statement beginning in decorators. The new operators are both infix; the one place they can never occur is at statement beginning. Therefore, no existing code will be broken by the addition of these operators, and there is no possible parsing ambiguity between @@ -583,7 +567,7 @@ This section is informative, rather than normative -- it documents the consensus of a number of libraries that provide array- or matrix-like -objects on how the ``@`` and ``@@`` operators will be implemented. +objects on how ``@`` will be implemented. This section uses the numpy terminology for describing arbitrary multidimensional arrays of data, because it is a superset of all other @@ -611,8 +595,8 @@ * 2d inputs are conventional matrices, and so the semantics are obvious: we apply conventional matrix multiplication. If we write - ``arr(2, 3)`` to represent an arbitrary 2x3 array, then ``arr(3, 4) - @ arr(4, 5)`` returns an array with shape (3, 5). + ``arr(2, 3)`` to represent an arbitrary 2x3 array, then ``arr(2, 3) + @ arr(3, 4)`` returns an array with shape (2, 4). * 1d vector inputs are promoted to 2d by prepending or appending a '1' to the shape, the operation is performed, and then the added @@ -705,36 +689,6 @@ elementwise ``*`` operator. Allowing scalar @ matrix would thus both require an unnecessary special case, and violate TOOWTDI. -The recommended semantics for ``@@`` are:: - - def __matpow__(self, n): - if not isinstance(n, numbers.Integral): - raise TypeError("@@ not implemented for fractional powers") - if n == 0: - return identity_matrix_with_shape(self.shape) - elif n < 0: - return inverse(self) @ (self @@ (n + 1)) - else: - return self @ (self @@ (n - 1)) - -(Of course we expect that much more efficient implementations will be -used in practice.) Notice that if given an appropriate definition of -``identity_matrix_with_shape``, then this definition will -automatically handle >2d arrays appropriately. Notice also that with -this definition, ``vector @@ 2`` gives the squared Euclidean length of -the vector, a commonly used value. Also, while it is rarely useful to -explicitly compute inverses or other negative powers in standard -immediate-mode dense matrix code, these computations are natural when -doing symbolic or deferred-mode computations (as in e.g. sympy, -theano, numba, numexpr); therefore, negative powers are fully -supported. Fractional powers, though, bring in variety of -`mathematical complications`_, so we leave it to individual projects -to decide whether they want to try to define some reasonable semantics -for fractional inputs. - -.. _`mathematical complications`: - https://en.wikipedia.org/wiki/Square_root_of_a_matrix - Adoption -------- @@ -743,12 +697,12 @@ types based on what API they currently use for elementwise and matrix multiplication. -**Projects which currently use * for *elementwise* multiplication, and -function/method calls for *matrix* multiplication:** +**Projects which currently use * for elementwise multiplication, and +function/method calls for matrix multiplication:** The developers of the following projects have expressed an intention -to implement ``@`` and ``@@`` on their array-like types using the -above semantics: +to implement ``@`` on their array-like types using the above +semantics: * numpy * pandas @@ -764,8 +718,8 @@ * pycuda * panda3d -**Projects which currently use * for *matrix* multiplication, and -function/method calls for *elementwise* multiplication:** +**Projects which currently use * for matrix multiplication, and +function/method calls for elementwise multiplication:** The following projects have expressed an intention, if this PEP is accepted, to migrate from their current API to the elementwise-``*``, @@ -784,8 +738,8 @@ * cvxopt -**Projects which currently use * for *matrix* multiplication, and -which do not implement elementwise multiplication at all:** +**Projects which currently use * for matrix multiplication, and which +don't really care about elementwise multiplication of matrices:** There are several projects which implement matrix types, but from a very different perspective than the numerical libraries discussed @@ -796,14 +750,13 @@ math point of view, there isn't much use for elementwise operations in the first place; as discussed in the Background section above, elementwise operations are motivated by the bag-of-numbers approach. -The different goals of these projects mean that they don't encounter -the basic problem that this PEP exists to address, making it mostly -irrelevant to them; while they appear superficially similar, they're -actually doing something quite different. They use ``*`` for matrix +So these projects don't encounter the basic problem that this PEP +exists to address, making it mostly irrelevant to them; while they +appear superficially similar to projects like numpy, they're actually +doing something quite different. They use ``*`` for matrix multiplication (and for group actions, and so forth), and if this PEP is accepted, their expressed intention is to continue doing so, while -perhaps adding ``@`` and ``@@`` on matrices as aliases for ``*`` and -``**``: +perhaps adding ``@`` as an alias. These projects include: * sympy * sage @@ -814,6 +767,23 @@ njs at pobox.com +Implementation details +====================== + +New functions ``operator.matmul`` and ``operator.__matmul__`` are +added to the standard library, with the usual semantics. + +A corresponding function ``PyObject* PyObject_MatrixMultiply(PyObject +*o1, PyObject o2)`` is added to the C API. + +A new AST node is added named ``MatMult``, along with a new token +``ATEQUAL`` and new bytecode opcodes ``BINARY_MATRIX_MULTIPLY`` and +``INPLACE_MATRIX_MULTIPLY``. + +Two new type slots are added; whether this is to ``PyNumberMethods`` +or a new ``PyMatrixMethods`` struct remains to be determined. + + Rationale for specification details =================================== @@ -848,16 +818,17 @@ What we need, though, is an operator that means "matrix multiplication, as opposed to scalar/elementwise multiplication". -There is no conventional symbol for these in mathematics or -programming, where these operations are usually distinguished by -context. (And U+2297 CIRCLED TIMES is actually used conventionally to -mean exactly the opposite: elementwise multiplication -- the "Hadamard -product" -- as opposed to matrix multiplication). ``@`` at least has -the virtue that it *looks* like a funny non-commutative operator; a -naive user who knows maths but not programming couldn't look at ``A * -B`` versus ``A ? B``, or ``A * B`` versus ``A ? B``, or ``A * B`` -versus ``A ? B`` and guess which one is the usual multiplication, and -which one is the special case. +There is no conventional symbol with this meaning in either +programming or mathematics, where these operations are usually +distinguished by context. (And U+2297 CIRCLED TIMES is actually used +conventionally to mean exactly the wrong things: elementwise +multiplication -- the "Hadamard product" -- or outer product, rather +than matrix/inner product like our operator). ``@`` at least has the +virtue that it *looks* like a funny non-commutative operator; a naive +user who knows maths but not programming couldn't look at ``A * B`` +versus ``A ? B``, or ``A * B`` versus ``A ? B``, or ``A * B`` versus +``A ? B`` and guess which one is the usual multiplication, and which +one is the special case. Finally, there is the option of using multi-character tokens. Some options: @@ -878,9 +849,9 @@ be too easy to confuse with ``*+``, which is just multiplication combined with the unary ``+`` operator. -* PEP 211 suggested ``~*`` and ``~**``. This has the downside that it - sort of suggests that there is a unary ``*`` operator that is being - combined with unary ``~``, but it could work. +* PEP 211 suggested ``~*``. This has the downside that it sort of + suggests that there is a unary ``*`` operator that is being combined + with unary ``~``, but it could work. * R uses ``%*%`` for matrix multiplication. In R this forms part of a general extensible infix system in which all tokens of the form @@ -888,12 +859,11 @@ token without stealing the system. * Some other plausible candidates that have been suggested: ``><`` (= - ascii drawing of the multiplication sign ?); the footnote operators - ``[*]`` and ``[**]`` or ``|*|`` and ``|**|`` (but when used in - context, the use of vertical grouping symbols tends to recreate the - nested parentheses visual clutter that was noted as one of the major - downsides of the function syntax we're trying to get away from); - ``^*`` and ``^^``. + ascii drawing of the multiplication sign ?); the footnote operator + ``[*]`` or ``|*|`` (but when used in context, the use of vertical + grouping symbols tends to recreate the nested parentheses visual + clutter that was noted as one of the major downsides of the function + syntax we're trying to get away from); ``^*`` and ``^^``. So, it doesn't matter much, but ``@`` seems as good or better than any of the alternatives: @@ -911,8 +881,12 @@ * The mATrices mnemonic is cute. -* The use of a single-character token reduces the line-noise effect, - and makes ``@@`` possible, which is a nice bonus. +* The swirly shape is reminiscent of the simultaneous sweeps over rows + and columns that define matrix multiplication + +* Its asymmetry is evocative of its non-commutative nature. + +* Whatever, we have to pick something. (Non)-Definitions for built-in types @@ -924,32 +898,41 @@ semantics for ``@`` are that it should raise an error on scalars. We do not -- for now -- define a ``__matmul__`` method on the standard -``memoryview`` or ``array.array`` objects, for several reasons. -First, there is currently no way to create multidimensional memoryview -objects using only the stdlib, and array objects cannot represent -multidimensional data at all, which makes ``__matmul__`` much less -useful. Second, providing a quality implementation of matrix -multiplication is highly non-trivial. Naive nested loop -implementations are very slow and providing one in CPython would just -create a trap for users. But the alternative -- providing a modern, -competitive matrix multiply -- would require that CPython link to a -BLAS library, which brings a set of new complications. In particular, -several popular BLAS libraries (including the one that ships by -default on OS X) currently break the use of ``multiprocessing`` -[#blas-fork]_. And finally, we'd have to add quite a bit beyond -``__matmul__`` before ``memoryview`` or ``array.array`` would be -useful for numeric work -- like elementwise versions of the other -arithmetic operators, just to start. Put together, these -considerations mean that the cost/benefit of adding ``__matmul__`` to -these types just isn't there, so for now we'll continue to delegate -these problems to numpy and friends, and defer a more systematic -solution to a future proposal. +``memoryview`` or ``array.array`` objects, for several reasons. Of +course this could be added if someone wants it, but these types would +require quite a bit of additional work beyond ``__matmul__`` before +they could be used for numeric work -- e.g., they have no way to do +addition or scalar multiplication either! -- and adding such +functionality is beyond the scope of this PEP. In addition, providing +a quality implementation of matrix multiplication is highly +non-trivial. Naive nested loop implementations are very slow and +shipping such an implementation in CPython would just create a trap +for users. But the alternative -- providing a modern, competitive +matrix multiply -- would require that CPython link to a BLAS library, +which brings a set of new complications. In particular, several +popular BLAS libraries (including the one that ships by default on +OS X) currently break the use of ``multiprocessing`` [#blas-fork]_. +Together, these considerations mean that the cost/benefit of adding +``__matmul__`` to these types just isn't there, so for now we'll +continue to delegate these problems to numpy and friends, and defer a +more systematic solution to a future proposal. There are also non-numeric Python builtins which define ``__mul__`` (``str``, ``list``, ...). We do not define ``__matmul__`` for these types either, because why would we even do that. +Non-definition of matrix power +------------------------------ + +Earlier versions of this PEP also proposed a matrix power operator, +``@@``, analogous to ``**``. But on further consideration, it was +decided that the utility of this was sufficiently unclear that it +would be better to leave it out for now, and only revisit the issue if +-- once we have more experience with ``@`` -- it turns out that ``@@`` +is truly missed. [#atat-discussion] + + Unresolved issues ----------------- @@ -1026,15 +1009,16 @@ general Python, and then overload it in numeric code:** This was the approach taken by PEP 211, which proposed defining ``@`` to be the equivalent of ``itertools.product``. The problem with this is that -when taken on its own terms, adding an infix operator for -``itertools.product`` is just silly. (During discussions of this PEP, -a similar suggestion was made to define ``@`` as a general purpose -function composition operator, and this suffers from the same problem; -``functools.compose`` isn't even useful enough to exist.) Matrix -multiplication has a uniquely strong rationale for inclusion as an -infix operator. There almost certainly don't exist any other binary -operations that will ever justify adding any other infix operators to -Python. +when taken on its own terms, it's pretty clear that +``itertools.product`` doesn't actually need a dedicated operator. It +hasn't even been deemed worth of a builtin. (During discussions of +this PEP, a similar suggestion was made to define ``@`` as a general +purpose function composition operator, and this suffers from the same +problem; ``functools.compose`` isn't even useful enough to exist.) +Matrix multiplication has a uniquely strong rationale for inclusion as +an infix operator. There almost certainly don't exist any other +binary operations that will ever justify adding any other infix +operators to Python. **Add a .dot method to array types so as to allow "pseudo-infix" A.dot(B) syntax:** This has been in numpy for some years, and in many @@ -1101,18 +1085,60 @@ evil hack like this that violates that system, creates bizarre error messages when accidentally misused, and whose underlying mechanisms can't be understood without deep knowledge of how object oriented -systems work. We've considered promoting this as a general solution, -and perhaps if the PEP is rejected we'll revisit this option, but so -far the numeric community has mostly elected to leave this one on the -shelf. +systems work. + +**Use a special "facade" type to support syntax like arr.M * arr:** +This is very similar to the previous proposal, in that the ``.M`` +attribute would basically return the same object as ``arr *dot` would, +and thus suffers the same objections about 'magicalness'. This +approach also has some non-obvious complexities: for example, while +``arr.M * arr`` must return an array, ``arr.M * arr.M`` and ``arr * +arr.M`` must return facade objects, or else ``arr.M * arr.M * arr`` +and ``arr * arr.M * arr`` will not work. But this means that facade +objects must be able to recognize both other array objects and other +facade objects (which creates additional complexity for writing +interoperating array types from different libraries who must now +recognize both each other's array types and their facade types). It +also creates pitfalls for users who may easily type ``arr * arr.M`` or +``arr.M * arr.M`` and expect to get back an array object; instead, +they will get a mysterious object that throws errors when they attempt +to use it. Basically with this approach users must be careful to +think of ``.M*`` as an indivisible unit that acts as an infix operator +-- and as infix-operator-like token strings go, at least ``*dot*`` +is prettier looking (look at its cute little ears!). + + +Discussions of this PEP +======================= + +Collected here for reference: + +* Github pull request containing much of the original discussion and + drafting: https://github.com/numpy/numpy/pull/4351 + +* sympy mailing list discussions of an early draft: + + * https://groups.google.com/forum/#!topic/sympy/22w9ONLa7qo + * https://groups.google.com/forum/#!topic/sympy/4tGlBGTggZY + +* sage-devel mailing list discussions of an early draft: + https://groups.google.com/forum/#!topic/sage-devel/YxEktGu8DeM + +* 13-Mar-2014 python-ideas thread: + https://mail.python.org/pipermail/python-ideas/2014-March/027053.html + +* numpy-discussion thread on whether to keep ``@@``: + http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069448.html + +* numpy-discussion thread on precedence/associativity of ``@``: + http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069444.html References ========== .. [#preprocessor] From a comment by GvR on a G+ post by GvR; the - comment itself does not seem to be directly linkable: - https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u + comment itself does not seem to be directly linkable: https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u .. [#infix-hack] http://code.activestate.com/recipes/384122-infix-operators/ http://www.sagemath.org/doc/reference/misc/sage/misc/decorators.html#sage.misc.decorators.infix_operator .. [#scipy-conf] http://conference.scipy.org/past.html @@ -1130,8 +1156,7 @@ average. Compare to eq. 2.139 in http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/tutorials/xegbohtmlnode17.html - Example code is adapted from - https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb7/rerpy/incremental_ls.py#L202 + Example code is adapted from https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb7/rerpy/incremental_ls.py#L202 .. [#pycon-tutorials] Out of the 36 tutorials scheduled for PyCon 2014 (https://us.pycon.org/2014/schedule/tutorials/), we guess that the @@ -1177,9 +1202,8 @@ from Python 3.2.3 to examine the tokens in all files ending ``.py`` underneath some directory. Only tokens which occur at least once in the source trees are included in the table. The counting script - will be available as an auxiliary file once this PEP is submitted; - until then, it can be found here: - https://gist.github.com/njsmith/9157645 + is available `in the PEP repository + `_. Matrix multiply counts were estimated by counting how often certain tokens which are used as matrix multiply function names occurred in @@ -1216,8 +1240,7 @@ .. [#broadcasting] http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html -.. [#matmul-other-langs] - http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html +.. [#matmul-other-langs] http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html .. [#github-details] Counts were produced by manually entering the string ``"import foo"`` or ``"from foo import"`` (with quotes) into diff --git a/scan-ops.py b/scan-ops.py new file mode 100644 --- /dev/null +++ b/scan-ops.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +# http://legacy.python.org/dev/peps/pep-0465/ +# https://gist.github.com/njsmith/9157645 + +# usage: +# python3 scan-ops.py stdlib_path sklearn_path nipy_path + +import sys +import os +import os.path +import tokenize +from collections import OrderedDict + +NON_SOURCE_TOKENS = [ + tokenize.COMMENT, tokenize.NL, tokenize.ENCODING, tokenize.NEWLINE, + tokenize.INDENT, tokenize.DEDENT, + ] + +SKIP_OPS = list("(),.:[]{}@;") + ["->", "..."] + +class TokenCounts(object): + def __init__(self, dot_names=[]): + self.counts = {} + self.sloc = 0 + self.dot_names = dot_names + + def count(self, path): + sloc_idxes = set() + for token in tokenize.tokenize(open(path, "rb").readline): + if token.type == tokenize.OP: + self.counts.setdefault(token.string, 0) + self.counts[token.string] += 1 + if token.string in self.dot_names: + self.counts.setdefault("dot", 0) + self.counts["dot"] += 1 + if token.type not in NON_SOURCE_TOKENS: + sloc_idxes.add(token.start[0]) + self.sloc += len(sloc_idxes) + + @classmethod + def combine(cls, objs): + combined = cls() + for obj in objs: + for op, count in obj.counts.items(): + combined.counts.setdefault(op, 0) + combined.counts[op] += count + combined.sloc += obj.sloc + return combined + +def count_tree(root, **kwargs): + c = TokenCounts(**kwargs) + for dirpath, _, filenames in os.walk(root): + for filename in filenames: + if filename.endswith(".py"): + path = os.path.join(dirpath, filename) + try: + c.count(path) + sys.stderr.write(".") + sys.stderr.flush() + except Exception as e: + sys.stderr.write("\nFailed to read %s: %s\n" % (path, e)) + return c + +# count_objs is OrderedDict (name -> TokenCounts) +def summarize(count_objs, out): + ops = {} + for count_obj in count_objs.values(): + for op in count_obj.counts: + ops[op] = [] + for count_obj in count_objs.values(): + for op, row in ops.items(): + count = count_obj.counts.get(op, 0) + row.append(count / count_obj.sloc) + titles = ["Op"] + list(count_objs) + # 4 chars is enough for ops and all numbers. + column_widths = [max(len(title), 4) for title in titles] + + rows = [] + for op, row in ops.items(): + #rows.append(["``" + op + "``"] + row) + rows.append([op] + row) + + rows.sort(key=lambda row: row[-1]) + rows.reverse() + + def write_row(entries): + out.write(" ".join(entries)) + out.write("\n") + + def lines(): + write_row("=" * w for w in column_widths) + + lines() + write_row(t.rjust(w) for w, t in zip(column_widths, titles)) + lines() + for row in rows: + op = row[0] + if op in SKIP_OPS: + continue + # numbers here are avg number of uses per sloc, which is + # inconveniently small. convert to uses/1e4 sloc + numbers = row[1:] + number_strs = [str(int(round(x * 10000))) for x in numbers] + formatted_row = [op] + number_strs + write_row(str(e).rjust(w) + for w, e in zip(column_widths, formatted_row)) + lines() + +def run_projects(names, dot_names, dirs, out): + assert len(names) == len(dot_names) == len(dirs) + count_objs = OrderedDict() + for name, dot_name, dir in zip(names, dot_names, dirs): + counts = count_tree(dir, dot_names=dot_name) + count_objs[name] = counts + out.write("%s: %s sloc\n" % (name, counts.sloc)) + count_objs["combined"] = TokenCounts.combine(count_objs.values()) + summarize(count_objs, out) + +if __name__ == "__main__": + run_projects(["stdlib", "scikit-learn", "nipy"], + [[], + # https://github.com/numpy/numpy/pull/4351#discussion_r9977913 + # sklearn fast_dot is used to fix up some optimizations that + # are missing from older numpy's, but in modern days is + # exactly the same, so it's fair to count. safe_sparse_dot + # has hacks to workaround some quirks in scipy.sparse + # matrices, but these quirks are also already fixed, so + # counting this calls is also fair. + ["dot", "fast_dot", "safe_sparse_dot"], + ["dot"]], + sys.argv[1:], + sys.stdout) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Mar 19 04:31:56 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 04:31:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Ignore_files_g?= =?utf-8?q?enerated_by_MSVC_PGO_builds=2E?= Message-ID: <3fpcHc0wNhz7LjW@mail.python.org> http://hg.python.org/cpython/rev/3df20e9ef7df changeset: 89848:3df20e9ef7df branch: 3.4 parent: 89846:f28f63c5d30a user: Zachary Ware date: Tue Mar 18 22:28:16 2014 -0500 summary: Ignore files generated by MSVC PGO builds. files: .hgignore | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -80,6 +80,8 @@ PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* +PCbuild/*-pgi +PCbuild/*-pgo PCbuild/amd64 PCbuild/ipch Tools/unicode/build/ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 04:31:57 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 04:31:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Ignore_files_generated_by_MSVC_PGO_builds_=28merge_from_?= =?utf-8?b?My40KS4=?= Message-ID: <3fpcHd2nt1z7LjW@mail.python.org> http://hg.python.org/cpython/rev/5d5523f7be91 changeset: 89849:5d5523f7be91 parent: 89847:f8b40d33e45d parent: 89848:3df20e9ef7df user: Zachary Ware date: Tue Mar 18 22:31:48 2014 -0500 summary: Ignore files generated by MSVC PGO builds (merge from 3.4). files: .hgignore | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -80,6 +80,8 @@ PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* +PCbuild/*-pgi +PCbuild/*-pgo PCbuild/amd64 PCbuild/ipch Tools/unicode/build/ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 04:40:11 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 04:40:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Clean_up_PCbui?= =?utf-8?q?ld/pcbuild=2Esln_a_bit=3A?= Message-ID: <3fpcT72QPRz7LjW@mail.python.org> http://hg.python.org/cpython/rev/88ff398a9ae0 changeset: 89850:88ff398a9ae0 branch: 3.4 parent: 89848:3df20e9ef7df user: Zachary Ware date: Tue Mar 18 22:34:52 2014 -0500 summary: Clean up PCbuild/pcbuild.sln a bit: - Remove configuration settings from removed _sha3.vcxproj - Don't try to build configurations of _testembed that don't exist (namely, PGInstrument and PGUpdate) files: PCbuild/pcbuild.sln | 46 ++++++++++---------------------- 1 files changed, 14 insertions(+), 32 deletions(-) diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -615,38 +615,6 @@ {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.PGUpdate|x64.ActiveCfg = Release|Win32 {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.Release|Win32.ActiveCfg = Release|Win32 {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.Release|x64.ActiveCfg = Release|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|Win32.ActiveCfg = Debug|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|Win32.Build.0 = Debug|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|x64.ActiveCfg = Debug|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|x64.Build.0 = Debug|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|x64.Build.0 = PGInstrument|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|x64.Build.0 = PGUpdate|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|Win32.ActiveCfg = Release|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|Win32.Build.0 = Release|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|x64.ActiveCfg = Release|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|x64.Build.0 = Release|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.ActiveCfg = Debug|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.Build.0 = Debug|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.ActiveCfg = Debug|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.Build.0 = Debug|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.Build.0 = PGInstrument|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.Build.0 = PGUpdate|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.ActiveCfg = Release|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.Build.0 = Release|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.ActiveCfg = Release|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.Build.0 = Release|x64 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|Win32.ActiveCfg = Debug|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|Win32.Build.0 = Debug|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|x64.ActiveCfg = Debug|x64 @@ -663,6 +631,20 @@ {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|Win32.Build.0 = Release|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|x64.ActiveCfg = Release|x64 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|x64.Build.0 = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.ActiveCfg = Debug|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.Build.0 = Debug|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.ActiveCfg = Debug|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.Build.0 = Debug|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.Build.0 = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.Build.0 = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.Build.0 = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 04:40:12 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 04:40:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_PCbuild/pcbuild=2Esln_cleanup_from_3=2E4?= Message-ID: <3fpcT84yntz7Lkl@mail.python.org> http://hg.python.org/cpython/rev/cfa7d7fe8250 changeset: 89851:cfa7d7fe8250 parent: 89849:5d5523f7be91 parent: 89850:88ff398a9ae0 user: Zachary Ware date: Tue Mar 18 22:40:23 2014 -0500 summary: Merge PCbuild/pcbuild.sln cleanup from 3.4 files: PCbuild/pcbuild.sln | 46 ++++++++++---------------------- 1 files changed, 14 insertions(+), 32 deletions(-) diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -615,38 +615,6 @@ {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.PGUpdate|x64.ActiveCfg = Release|Win32 {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.Release|Win32.ActiveCfg = Release|Win32 {19C0C13F-47CA-4432-AFF3-799A296A4DDC}.Release|x64.ActiveCfg = Release|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|Win32.ActiveCfg = Debug|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|Win32.Build.0 = Debug|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|x64.ActiveCfg = Debug|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Debug|x64.Build.0 = Debug|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGInstrument|x64.Build.0 = PGInstrument|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.PGUpdate|x64.Build.0 = PGUpdate|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|Win32.ActiveCfg = Release|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|Win32.Build.0 = Release|Win32 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|x64.ActiveCfg = Release|x64 - {254A0C05-6696-4B08-8CB2-EF7D533AEE01}.Release|x64.Build.0 = Release|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.ActiveCfg = Debug|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.Build.0 = Debug|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.ActiveCfg = Debug|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.Build.0 = Debug|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.Build.0 = PGInstrument|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.Build.0 = PGUpdate|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.ActiveCfg = Release|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.Build.0 = Release|Win32 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.ActiveCfg = Release|x64 - {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.Build.0 = Release|x64 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|Win32.ActiveCfg = Debug|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|Win32.Build.0 = Debug|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Debug|x64.ActiveCfg = Debug|x64 @@ -663,6 +631,20 @@ {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|Win32.Build.0 = Release|Win32 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|x64.ActiveCfg = Release|x64 {EB6E69DD-04BF-4543-9B92-49FAABCEAC2E}.Release|x64.Build.0 = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.ActiveCfg = Debug|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|Win32.Build.0 = Debug|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.ActiveCfg = Debug|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Debug|x64.Build.0 = Debug|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGInstrument|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|Win32.Build.0 = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.PGUpdate|x64.Build.0 = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.ActiveCfg = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.Build.0 = Release|Win32 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.ActiveCfg = Release|x64 + {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 05:08:48 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 05:08:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Avoid_compile_?= =?utf-8?q?warning_in_xxlimited_on_32-bit_Windows_non-Debug_builds=2E?= Message-ID: <3fpd680Hgrz7LjV@mail.python.org> http://hg.python.org/cpython/rev/c44258b4b7a4 changeset: 89852:c44258b4b7a4 branch: 3.4 parent: 89850:88ff398a9ae0 user: Zachary Ware date: Tue Mar 18 23:05:01 2014 -0500 summary: Avoid compile warning in xxlimited on 32-bit Windows non-Debug builds. files: PCbuild/xxlimited.vcxproj | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj --- a/PCbuild/xxlimited.vcxproj +++ b/PCbuild/xxlimited.vcxproj @@ -120,7 +120,7 @@ - NDEBUG;_WIN32;_WINDLL;Py_LIMITED_API + NDEBUG;_WIN32;_WINDLL;Py_LIMITED_API=0x03040000 wsock32.lib;%(AdditionalDependencies) @@ -140,7 +140,7 @@ - Py_LIMITED_API;%(PreprocessorDefinitions) + Py_LIMITED_API=0x03040000;%(PreprocessorDefinitions) wsock32.lib;%(AdditionalDependencies) @@ -161,7 +161,7 @@ - Py_LIMITED_API;%(PreprocessorDefinitions) + Py_LIMITED_API=0x03040000;%(PreprocessorDefinitions) wsock32.lib;%(AdditionalDependencies) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 05:08:49 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 05:08:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_xxlimited_compile_warning_fix_from_3=2E4?= Message-ID: <3fpd691s3Gz7LjV@mail.python.org> http://hg.python.org/cpython/rev/6f6f1f56eb2a changeset: 89853:6f6f1f56eb2a parent: 89851:cfa7d7fe8250 parent: 89852:c44258b4b7a4 user: Zachary Ware date: Tue Mar 18 23:09:00 2014 -0500 summary: Merge xxlimited compile warning fix from 3.4 files: PCbuild/xxlimited.vcxproj | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj --- a/PCbuild/xxlimited.vcxproj +++ b/PCbuild/xxlimited.vcxproj @@ -120,7 +120,7 @@ - NDEBUG;_WIN32;_WINDLL;Py_LIMITED_API + NDEBUG;_WIN32;_WINDLL;Py_LIMITED_API=0x03040000 wsock32.lib;%(AdditionalDependencies) @@ -140,7 +140,7 @@ - Py_LIMITED_API;%(PreprocessorDefinitions) + Py_LIMITED_API=0x03040000;%(PreprocessorDefinitions) wsock32.lib;%(AdditionalDependencies) @@ -161,7 +161,7 @@ - Py_LIMITED_API;%(PreprocessorDefinitions) + Py_LIMITED_API=0x03040000;%(PreprocessorDefinitions) wsock32.lib;%(AdditionalDependencies) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Mar 19 09:56:47 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 19 Mar 2014 09:56:47 +0100 Subject: [Python-checkins] Daily reference leaks (f8b40d33e45d): sum=8 Message-ID: results for f8b40d33e45d on branch "default" -------------------------------------------- test_asyncio leaked [4, 0, 0] memory blocks, sum=4 test_site leaked [0, 0, 2] references, sum=2 test_site leaked [0, 0, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogWaFGo5', '-x'] From python-checkins at python.org Wed Mar 19 11:29:49 2014 From: python-checkins at python.org (kristjan.jonsson) Date: Wed, 19 Mar 2014 11:29:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319009?= Message-ID: <3fpnYn2xd9z7Ljg@mail.python.org> http://hg.python.org/cpython/rev/49017c391564 changeset: 89854:49017c391564 user: Kristj?n Valur J?nsson date: Wed Mar 19 10:07:26 2014 +0000 summary: Issue #19009 Enhance HTTPResponse.readline() performance files: Lib/http/client.py | 176 ++++++++++---- Lib/test/test_httplib.py | 303 +++++++++++++++++++++++--- 2 files changed, 389 insertions(+), 90 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -271,7 +271,7 @@ return email.parser.Parser(_class=_class).parsestr(hstring) -class HTTPResponse(io.RawIOBase): +class HTTPResponse(io.BufferedIOBase): # See RFC 2616 sec 19.6 and RFC 1945 sec 6 for details. @@ -496,9 +496,10 @@ return b"" if amt is not None: - # Amount is given, so call base class version - # (which is implemented in terms of self.readinto) - return super(HTTPResponse, self).read(amt) + # Amount is given, implement using readinto + b = bytearray(amt) + n = self.readinto(b) + return memoryview(b)[:n].tobytes() else: # Amount is not given (unbounded read) so we must check self.length # and self.chunked @@ -578,71 +579,67 @@ if line in (b'\r\n', b'\n', b''): break + def _get_chunk_left(self): + # return self.chunk_left, reading a new chunk if necessary. + # chunk_left == 0: at the end of the current chunk, need to close it + # chunk_left == None: No current chunk, should read next. + # This function returns non-zero or None if the last chunk has + # been read. + chunk_left = self.chunk_left + if not chunk_left: # Can be 0 or None + if chunk_left is not None: + # We are at the end of chunk. dicard chunk end + self._safe_read(2) # toss the CRLF at the end of the chunk + try: + chunk_left = self._read_next_chunk_size() + except ValueError: + raise IncompleteRead(b'') + if chunk_left == 0: + # last chunk: 1*("0") [ chunk-extension ] CRLF + self._read_and_discard_trailer() + # we read everything; close the "file" + self._close_conn() + chunk_left = None + self.chunk_left = chunk_left + return chunk_left + def _readall_chunked(self): assert self.chunked != _UNKNOWN - chunk_left = self.chunk_left value = [] - while True: - if chunk_left is None: - try: - chunk_left = self._read_next_chunk_size() - if chunk_left == 0: - break - except ValueError: - raise IncompleteRead(b''.join(value)) - value.append(self._safe_read(chunk_left)) - - # we read the whole chunk, get another - self._safe_read(2) # toss the CRLF at the end of the chunk - chunk_left = None - - self._read_and_discard_trailer() - - # we read everything; close the "file" - self._close_conn() - - return b''.join(value) + try: + while True: + chunk_left = self._get_chunk_left() + if chunk_left is None: + break + value.append(self._safe_read(chunk_left)) + self.chunk_left = 0 + return b''.join(value) + except IncompleteRead: + raise IncompleteRead(b''.join(value)) def _readinto_chunked(self, b): assert self.chunked != _UNKNOWN - chunk_left = self.chunk_left - total_bytes = 0 mvb = memoryview(b) - while True: - if chunk_left is None: - try: - chunk_left = self._read_next_chunk_size() - if chunk_left == 0: - break - except ValueError: - raise IncompleteRead(bytes(b[0:total_bytes])) + try: + while True: + chunk_left = self._get_chunk_left() + if chunk_left is None: + return total_bytes - if len(mvb) < chunk_left: - n = self._safe_readinto(mvb) - self.chunk_left = chunk_left - n - return total_bytes + n - elif len(mvb) == chunk_left: - n = self._safe_readinto(mvb) - self._safe_read(2) # toss the CRLF at the end of the chunk - self.chunk_left = None - return total_bytes + n - else: - temp_mvb = mvb[0:chunk_left] + if len(mvb) <= chunk_left: + n = self._safe_readinto(mvb) + self.chunk_left = chunk_left - n + return total_bytes + n + + temp_mvb = mvb[:chunk_left] n = self._safe_readinto(temp_mvb) mvb = mvb[n:] total_bytes += n + self.chunk_left = 0 - # we read the whole chunk, get another - self._safe_read(2) # toss the CRLF at the end of the chunk - chunk_left = None - - self._read_and_discard_trailer() - - # we read everything; close the "file" - self._close_conn() - - return total_bytes + except IncompleteRead: + raise IncompleteRead(bytes(b[0:total_bytes])) def _safe_read(self, amt): """Read the number of bytes requested, compensating for partial reads. @@ -683,6 +680,73 @@ total_bytes += n return total_bytes + def read1(self, n=-1): + """Read with at most one underlying system call. If at least one + byte is buffered, return that instead. + """ + if self.fp is None or self._method == "HEAD": + return b"" + if self.chunked: + return self._read1_chunked(n) + try: + result = self.fp.read1(n) + except ValueError: + if n >= 0: + raise + # some implementations, like BufferedReader, don't support -1 + # Read an arbitrarily selected largeish chunk. + result = self.fp.read1(16*1024) + if not result and n: + self._close_conn() + return result + + def peek(self, n=-1): + # Having this enables IOBase.readline() to read more than one + # byte at a time + if self.fp is None or self._method == "HEAD": + return b"" + if self.chunked: + return self._peek_chunked(n) + return self.fp.peek(n) + + def readline(self, limit=-1): + if self.fp is None or self._method == "HEAD": + return b"" + if self.chunked: + # Fallback to IOBase readline which uses peek() and read() + return super().readline(limit) + result = self.fp.readline(limit) + if not result and limit: + self._close_conn() + return result + + def _read1_chunked(self, n): + # Strictly speaking, _get_chunk_left() may cause more than one read, + # but that is ok, since that is to satisfy the chunked protocol. + chunk_left = self._get_chunk_left() + if chunk_left is None or n == 0: + return b'' + if not (0 <= n <= chunk_left): + n = chunk_left # if n is negative or larger than chunk_left + read = self.fp.read1(n) + self.chunk_left -= len(read) + if not read: + raise IncompleteRead(b"") + return read + + def _peek_chunked(self, n): + # Strictly speaking, _get_chunk_left() may cause more than one read, + # but that is ok, since that is to satisfy the chunked protocol. + try: + chunk_left = self._get_chunk_left() + except IncompleteRead: + return b'' # peek doesn't worry about protocol + if chunk_left is None: + return b'' # eof + # peek is allowed to return more than requested. Just request the + # entire chunk, and truncate what we get. + return self.fp.peek(chunk_left)[:chunk_left] + def fileno(self): return self.fp.fileno() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -18,6 +18,26 @@ # Root cert file (CA) for svn.python.org's cert CACERT_svn_python_org = os.path.join(here, 'https_svn_python_org_root.pem') +# constants for testing chunked encoding +chunked_start = ( + 'HTTP/1.1 200 OK\r\n' + 'Transfer-Encoding: chunked\r\n\r\n' + 'a\r\n' + 'hello worl\r\n' + '3\r\n' + 'd! \r\n' + '8\r\n' + 'and now \r\n' + '22\r\n' + 'for something completely different\r\n' +) +chunked_expected = b'hello world! and now for something completely different' +chunk_extension = ";foo=bar" +last_chunk = "0\r\n" +last_chunk_extended = "0" + chunk_extension + "\r\n" +trailers = "X-Dummy: foo\r\nX-Dumm2: bar\r\n" +chunked_end = "\r\n" + HOST = support.HOST class FakeSocket: @@ -36,7 +56,10 @@ def makefile(self, mode, bufsize=None): if mode != 'r' and mode != 'rb': raise client.UnimplementedFileMode() - return self.fileclass(self.text) + # keep the file around so we can check how much was read from it + self.file = self.fileclass(self.text) + self.file.close = lambda:None #nerf close () + return self.file class EPipeSocket(FakeSocket): @@ -430,20 +453,8 @@ conn.request('POST', 'test', conn) def test_chunked(self): - chunked_start = ( - 'HTTP/1.1 200 OK\r\n' - 'Transfer-Encoding: chunked\r\n\r\n' - 'a\r\n' - 'hello worl\r\n' - '3\r\n' - 'd! \r\n' - '8\r\n' - 'and now \r\n' - '22\r\n' - 'for something completely different\r\n' - ) - expected = b'hello world! and now for something completely different' - sock = FakeSocket(chunked_start + '0\r\n') + expected = chunked_expected + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="GET") resp.begin() self.assertEqual(resp.read(), expected) @@ -451,7 +462,7 @@ # Various read sizes for n in range(1, 12): - sock = FakeSocket(chunked_start + '0\r\n') + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="GET") resp.begin() self.assertEqual(resp.read(n) + resp.read(n) + resp.read(), expected) @@ -474,23 +485,12 @@ resp.close() def test_readinto_chunked(self): - chunked_start = ( - 'HTTP/1.1 200 OK\r\n' - 'Transfer-Encoding: chunked\r\n\r\n' - 'a\r\n' - 'hello worl\r\n' - '3\r\n' - 'd! \r\n' - '8\r\n' - 'and now \r\n' - '22\r\n' - 'for something completely different\r\n' - ) - expected = b'hello world! and now for something completely different' + + expected = chunked_expected nexpected = len(expected) b = bytearray(128) - sock = FakeSocket(chunked_start + '0\r\n') + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="GET") resp.begin() n = resp.readinto(b) @@ -500,7 +500,7 @@ # Various read sizes for n in range(1, 12): - sock = FakeSocket(chunked_start + '0\r\n') + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="GET") resp.begin() m = memoryview(b) @@ -536,7 +536,7 @@ '1\r\n' 'd\r\n' ) - sock = FakeSocket(chunked_start + '0\r\n') + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="HEAD") resp.begin() self.assertEqual(resp.read(), b'') @@ -556,7 +556,7 @@ '1\r\n' 'd\r\n' ) - sock = FakeSocket(chunked_start + '0\r\n') + sock = FakeSocket(chunked_start + last_chunk + chunked_end) resp = client.HTTPResponse(sock, method="HEAD") resp.begin() b = bytearray(5) @@ -631,6 +631,7 @@ + '0' * 65536 + 'a\r\n' 'hello world\r\n' '0\r\n' + '\r\n' ) resp = client.HTTPResponse(FakeSocket(body)) resp.begin() @@ -670,6 +671,239 @@ conn.request('POST', '/', body) self.assertGreater(sock.sendall_calls, 1) + def test_chunked_extension(self): + extra = '3;foo=bar\r\n' + 'abc\r\n' + expected = chunked_expected + b'abc' + + sock = FakeSocket(chunked_start + extra + last_chunk_extended + chunked_end) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertEqual(resp.read(), expected) + resp.close() + + def test_chunked_missing_end(self): + """some servers may serve up a short chunked encoding stream""" + expected = chunked_expected + sock = FakeSocket(chunked_start + last_chunk) #no terminating crlf + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertEqual(resp.read(), expected) + resp.close() + + def test_chunked_trailers(self): + """See that trailers are read and ignored""" + expected = chunked_expected + sock = FakeSocket(chunked_start + last_chunk + trailers + chunked_end) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertEqual(resp.read(), expected) + # we should have reached the end of the file + self.assertEqual(sock.file.read(100), b"") #we read to the end + resp.close() + + def test_chunked_sync(self): + """Check that we don't read past the end of the chunked-encoding stream""" + expected = chunked_expected + extradata = "extradata" + sock = FakeSocket(chunked_start + last_chunk + trailers + chunked_end + extradata) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertEqual(resp.read(), expected) + # the file should now have our extradata ready to be read + self.assertEqual(sock.file.read(100), extradata.encode("ascii")) #we read to the end + resp.close() + + def test_content_length_sync(self): + """Check that we don't read past the end of the Content-Length stream""" + extradata = "extradata" + expected = b"Hello123\r\n" + sock = FakeSocket('HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\nHello123\r\n' + extradata) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + self.assertEqual(resp.read(), expected) + # the file should now have our extradata ready to be read + self.assertEqual(sock.file.read(100), extradata.encode("ascii")) #we read to the end + resp.close() + +class ExtendedReadTest(TestCase): + """ + Test peek(), read1(), readline() + """ + lines = ( + 'HTTP/1.1 200 OK\r\n' + '\r\n' + 'hello world!\n' + 'and now \n' + 'for something completely different\n' + 'foo' + ) + lines_expected = lines[lines.find('hello'):].encode("ascii") + lines_chunked = ( + 'HTTP/1.1 200 OK\r\n' + 'Transfer-Encoding: chunked\r\n\r\n' + 'a\r\n' + 'hello worl\r\n' + '3\r\n' + 'd!\n\r\n' + '9\r\n' + 'and now \n\r\n' + '23\r\n' + 'for something completely different\n\r\n' + '3\r\n' + 'foo\r\n' + '0\r\n' # terminating chunk + '\r\n' # end of trailers + ) + + def setUp(self): + sock = FakeSocket(self.lines) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + resp.fp = io.BufferedReader(resp.fp) + self.resp = resp + + + + def test_peek(self): + resp = self.resp + # patch up the buffered peek so that it returns not too much stuff + oldpeek = resp.fp.peek + def mypeek(n=-1): + p = oldpeek(n) + if n >= 0: + return p[:n] + return p[:10] + resp.fp.peek = mypeek + + all = [] + while True: + # try a short peek + p = resp.peek(3) + if p: + self.assertGreater(len(p), 0) + # then unbounded peek + p2 = resp.peek() + self.assertGreaterEqual(len(p2), len(p)) + self.assertTrue(p2.startswith(p)) + next = resp.read(len(p2)) + self.assertEqual(next, p2) + else: + next = resp.read() + self.assertFalse(next) + all.append(next) + if not next: + break + self.assertEqual(b"".join(all), self.lines_expected) + + def test_readline(self): + resp = self.resp + self._verify_readline(self.resp.readline, self.lines_expected) + + def _verify_readline(self, readline, expected): + all = [] + while True: + # short readlines + line = readline(5) + if line and line != b"foo": + if len(line) < 5: + self.assertTrue(line.endswith(b"\n")) + all.append(line) + if not line: + break + self.assertEqual(b"".join(all), expected) + + def test_read1(self): + resp = self.resp + def r(): + res = resp.read1(4) + self.assertLessEqual(len(res), 4) + return res + readliner = Readliner(r) + self._verify_readline(readliner.readline, self.lines_expected) + + def test_read1_unbounded(self): + resp = self.resp + all = [] + while True: + data = resp.read1() + if not data: + break + all.append(data) + self.assertEqual(b"".join(all), self.lines_expected) + + def test_read1_bounded(self): + resp = self.resp + all = [] + while True: + data = resp.read1(10) + if not data: + break + self.assertLessEqual(len(data), 10) + all.append(data) + self.assertEqual(b"".join(all), self.lines_expected) + + def test_read1_0(self): + self.assertEqual(self.resp.read1(0), b"") + + def test_peek_0(self): + p = self.resp.peek(0) + self.assertLessEqual(0, len(p)) + +class ExtendedReadTestChunked(ExtendedReadTest): + """ + Test peek(), read1(), readline() in chunked mode + """ + lines = ( + 'HTTP/1.1 200 OK\r\n' + 'Transfer-Encoding: chunked\r\n\r\n' + 'a\r\n' + 'hello worl\r\n' + '3\r\n' + 'd!\n\r\n' + '9\r\n' + 'and now \n\r\n' + '23\r\n' + 'for something completely different\n\r\n' + '3\r\n' + 'foo\r\n' + '0\r\n' # terminating chunk + '\r\n' # end of trailers + ) + + +class Readliner: + """ + a simple readline class that uses an arbitrary read function and buffering + """ + def __init__(self, readfunc): + self.readfunc = readfunc + self.remainder = b"" + + def readline(self, limit): + data = [] + datalen = 0 + read = self.remainder + try: + while True: + idx = read.find(b'\n') + if idx != -1: + break + if datalen + len(read) >= limit: + idx = limit - datalen - 1 + # read more data + data.append(read) + read = self.readfunc() + if not read: + idx = 0 #eof condition + break + idx += 1 + data.append(read[:idx]) + self.remainder = read[idx:] + return b"".join(data) + except: + self.remainder = b"".join(data) + raise + class OfflineTest(TestCase): def test_responses(self): self.assertEqual(client.responses[client.NOT_FOUND], "Not Found") @@ -973,7 +1207,8 @@ def test_main(verbose=None): support.run_unittest(HeaderTests, OfflineTest, BasicTest, TimeoutTest, HTTPSTest, RequestBodyTest, SourceAddressTest, - HTTPResponseTest) + HTTPResponseTest, ExtendedReadTest, + ExtendedReadTestChunked) if __name__ == '__main__': test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 16:39:09 2014 From: python-checkins at python.org (ethan.furman) Date: Wed, 19 Mar 2014 16:39:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue19995=3A_passing_a_no?= =?utf-8?q?n-int_to_=25o=2C_=25c=2C_=25x=2C_or_=25X_now_raises_an_exceptio?= =?utf-8?q?n?= Message-ID: <3fpwQj2hVlz7Lmp@mail.python.org> http://hg.python.org/cpython/rev/9120196b3114 changeset: 89855:9120196b3114 user: Ethan Furman date: Wed Mar 19 08:38:52 2014 -0700 summary: Issue19995: passing a non-int to %o, %c, %x, or %X now raises an exception files: Lib/test/string_tests.py | 3 +-- Lib/test/test_format.py | 10 ---------- Lib/test/test_unicode.py | 10 +++++----- Misc/NEWS | 2 ++ Objects/unicodeobject.c | 28 ++-------------------------- 5 files changed, 10 insertions(+), 43 deletions(-) diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -1178,8 +1178,7 @@ self.checkraises(TypeError, 'abc', '__mod__') self.checkraises(TypeError, '%(foo)s', '__mod__', 42) self.checkraises(TypeError, '%s%s', '__mod__', (42,)) - with self.assertWarns(DeprecationWarning): - self.checkraises(TypeError, '%c', '__mod__', (None,)) + self.checkraises(TypeError, '%c', '__mod__', (None,)) self.checkraises(ValueError, '%(foo', '__mod__', {}) self.checkraises(TypeError, '%(foo)s %(bar)s', '__mod__', ('foo', 42)) self.checkraises(TypeError, '%d', '__mod__', "42") # not numeric diff --git a/Lib/test/test_format.py b/Lib/test/test_format.py --- a/Lib/test/test_format.py +++ b/Lib/test/test_format.py @@ -142,8 +142,6 @@ testformat("%#+027.23X", big, "+0X0001234567890ABCDEF12345") # same, except no 0 flag testformat("%#+27.23X", big, " +0X001234567890ABCDEF12345") - with self.assertWarns(DeprecationWarning): - testformat("%x", float(big), "123456_______________", 6) big = 0o12345670123456701234567012345670 # 32 octal digits testformat("%o", big, "12345670123456701234567012345670") testformat("%o", -big, "-12345670123456701234567012345670") @@ -183,8 +181,6 @@ testformat("%034.33o", big, "0012345670123456701234567012345670") # base marker shouldn't change that testformat("%0#34.33o", big, "0o012345670123456701234567012345670") - with self.assertWarns(DeprecationWarning): - testformat("%o", float(big), "123456__________________________", 6) # Some small ints, in both Python int and flavors). testformat("%d", 42, "42") testformat("%d", -42, "-42") @@ -195,8 +191,6 @@ testformat("%#x", 1, "0x1") testformat("%#X", 1, "0X1") testformat("%#X", 1, "0X1") - with self.assertWarns(DeprecationWarning): - testformat("%#x", 1.0, "0x1") testformat("%#o", 1, "0o1") testformat("%#o", 1, "0o1") testformat("%#o", 0, "0o0") @@ -213,14 +207,10 @@ testformat("%x", -0x42, "-42") testformat("%x", 0x42, "42") testformat("%x", -0x42, "-42") - with self.assertWarns(DeprecationWarning): - testformat("%x", float(0x42), "42") testformat("%o", 0o42, "42") testformat("%o", -0o42, "-42") testformat("%o", 0o42, "42") testformat("%o", -0o42, "-42") - with self.assertWarns(DeprecationWarning): - testformat("%o", float(0o42), "42") testformat("%r", "\u0378", "'\\u0378'") # non printable testformat("%a", "\u0378", "'\\u0378'") # non printable testformat("%r", "\u0374", "'\u0374'") # printable diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1149,11 +1149,11 @@ self.assertEqual('%X' % letter_m, '6D') self.assertEqual('%o' % letter_m, '155') self.assertEqual('%c' % letter_m, 'm') - self.assertWarns(DeprecationWarning, '%x'.__mod__, pi), - self.assertWarns(DeprecationWarning, '%x'.__mod__, 3.14), - self.assertWarns(DeprecationWarning, '%X'.__mod__, 2.11), - self.assertWarns(DeprecationWarning, '%o'.__mod__, 1.79), - self.assertWarns(DeprecationWarning, '%c'.__mod__, pi), + self.assertRaises(TypeError, '%x'.__mod__, pi), + self.assertRaises(TypeError, '%x'.__mod__, 3.14), + self.assertRaises(TypeError, '%X'.__mod__, 2.11), + self.assertRaises(TypeError, '%o'.__mod__, 1.79), + self.assertRaises(TypeError, '%c'.__mod__, pi), def test_formatting_with_enum(self): # issue18780 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,8 @@ - Issue #20637: Key-sharing now also works for instance dictionaries of subclasses. Patch by Peter Ingebretson. +- Issue #19995: %c, %o, %x, and %X now raise TypeError on non-integer input. + Library ------- diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13987,23 +13987,11 @@ goto wrongtype; /* make sure number is a type of integer */ - /* if not, issue deprecation warning for now */ if (!PyLong_Check(v)) { if (type == 'o' || type == 'x' || type == 'X') { iobj = PyNumber_Index(v); if (iobj == NULL) { - PyErr_Clear(); - if (PyErr_WarnEx(PyExc_DeprecationWarning, - "automatic int conversions have been deprecated", - 1)) { - return -1; - } - iobj = PyNumber_Long(v); - if (iobj == NULL ) { - if (PyErr_ExceptionMatches(PyExc_TypeError)) - goto wrongtype; - return -1; - } + return -1; } } else { @@ -14085,22 +14073,10 @@ PyObject *iobj; long x; /* make sure number is a type of integer */ - /* if not, issue deprecation warning for now */ if (!PyLong_Check(v)) { iobj = PyNumber_Index(v); if (iobj == NULL) { - PyErr_Clear(); - if (PyErr_WarnEx(PyExc_DeprecationWarning, - "automatic int conversions have been deprecated", - 1)) { - return -1; - } - iobj = PyNumber_Long(v); - if (iobj == NULL ) { - if (PyErr_ExceptionMatches(PyExc_TypeError)) - goto onError; - return -1; - } + goto onError; } v = iobj; Py_DECREF(iobj); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 17:34:22 2014 From: python-checkins at python.org (victor.stinner) Date: Wed, 19 Mar 2014 17:34:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogU2tpcCB0ZXN0X3Vy?= =?utf-8?q?llib2=2Etest=5Fissue16464=28=29_is_the_ssl_module_is_missing?= Message-ID: <3fpxfQ4cLLz7LkK@mail.python.org> http://hg.python.org/cpython/rev/68335b8afb1f changeset: 89856:68335b8afb1f branch: 3.4 parent: 89852:c44258b4b7a4 user: Victor Stinner date: Wed Mar 19 17:31:20 2014 +0100 summary: Skip test_urllib2.test_issue16464() is the ssl module is missing files: Lib/test/test_urllib2.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -6,6 +6,10 @@ import socket import array import sys +try: + import ssl +except ImportError: + ssl = None import urllib.request # The proxy bypass method imported below has logic specific to the OSX @@ -1438,6 +1442,8 @@ @unittest.skipUnless(support.is_resource_enabled('network'), 'test requires network access') + @unittest.skipIf(ssl is None, + 'test requires the ssl module') def test_issue16464(self): opener = urllib.request.build_opener() request = urllib.request.Request("http://www.python.org/~jeremy/") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 17:34:24 2014 From: python-checkins at python.org (victor.stinner) Date: Wed, 19 Mar 2014 17:34:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Skip_test=5Furllib2=2Etest=5Fissue1646?= =?utf-8?q?4=28=29_is_the_ssl_module_is_missing?= Message-ID: <3fpxfS0WQVz7Lm5@mail.python.org> http://hg.python.org/cpython/rev/ad0c75b7bd7d changeset: 89857:ad0c75b7bd7d parent: 89855:9120196b3114 parent: 89856:68335b8afb1f user: Victor Stinner date: Wed Mar 19 17:34:12 2014 +0100 summary: (Merge 3.4) Skip test_urllib2.test_issue16464() is the ssl module is missing files: Lib/test/test_urllib2.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -6,6 +6,10 @@ import socket import array import sys +try: + import ssl +except ImportError: + ssl = None import urllib.request # The proxy bypass method imported below has logic specific to the OSX @@ -1438,6 +1442,8 @@ @unittest.skipUnless(support.is_resource_enabled('network'), 'test requires network access') + @unittest.skipIf(ssl is None, + 'test requires the ssl module') def test_issue16464(self): opener = urllib.request.build_opener() request = urllib.request.Request("http://www.python.org/~jeremy/") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 20:48:12 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 20:48:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Use_the_correc?= =?utf-8?q?t_VS_edition_names_in_PCbuild/readme=2Etxt?= Message-ID: <3fq1y440LJz7LlV@mail.python.org> http://hg.python.org/cpython/rev/4f6ac7d65ef1 changeset: 89858:4f6ac7d65ef1 branch: 3.4 parent: 89856:68335b8afb1f user: Zachary Ware date: Wed Mar 19 14:46:25 2014 -0500 summary: Use the correct VS edition names in PCbuild/readme.txt files: PCbuild/readme.txt | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -6,16 +6,19 @@ bit platforms. Using this directory requires an installation of Microsoft Visual C++ 2010 (MSVC 10.0) of any edition. The specific requirements are as follows: + Visual C++ 2010 Express Edition Required for building 32-bit Debug and Release configuration builds. This edition does not support "solution folders", which pcbuild.sln uses; this will not prevent building. -Visual Studio 2010 Standard Edition +Visual Studio 2010 Professional Edition Required for building 64-bit Debug and Release configuration builds -Visual Studio 2010 Professional Edition +Visual Studio 2010 Premium Edition Required for building Release configuration builds that make use of - Profile Guided Optimization (PGO), on either platform. The official - Python releases are built with Professional Edition using PGO. + Profile Guided Optimization (PGO), on either platform. + +The official Python releases are built with PGO using Visual Studio 2010 +Ultimate Edition. All you need to do to build is open the solution "pcbuild.sln" in Visual Studio, select the desired combination of configuration and platform, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 20:48:13 2014 From: python-checkins at python.org (zach.ware) Date: Wed, 19 Mar 2014 20:48:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <3fq1y568rnz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/7f0910efc5e7 changeset: 89859:7f0910efc5e7 parent: 89857:ad0c75b7bd7d parent: 89858:4f6ac7d65ef1 user: Zachary Ware date: Wed Mar 19 14:47:49 2014 -0500 summary: Merge with 3.4 files: PCbuild/readme.txt | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -6,16 +6,19 @@ bit platforms. Using this directory requires an installation of Microsoft Visual C++ 2010 (MSVC 10.0) of any edition. The specific requirements are as follows: + Visual C++ 2010 Express Edition Required for building 32-bit Debug and Release configuration builds. This edition does not support "solution folders", which pcbuild.sln uses; this will not prevent building. -Visual Studio 2010 Standard Edition +Visual Studio 2010 Professional Edition Required for building 64-bit Debug and Release configuration builds -Visual Studio 2010 Professional Edition +Visual Studio 2010 Premium Edition Required for building Release configuration builds that make use of - Profile Guided Optimization (PGO), on either platform. The official - Python releases are built with Professional Edition using PGO. + Profile Guided Optimization (PGO), on either platform. + +The official Python releases are built with PGO using Visual Studio 2010 +Ultimate Edition. All you need to do to build is open the solution "pcbuild.sln" in Visual Studio, select the desired combination of configuration and platform, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 21:25:58 2014 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Mar 2014 21:25:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2313437=3A_link_to_the_so?= =?utf-8?q?urce_code_for_a_few_more_modules?= Message-ID: <3fq2nf3brvz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/bc107f5faedc changeset: 89860:bc107f5faedc user: Andrew Kuchling date: Wed Mar 19 16:23:01 2014 -0400 summary: #13437: link to the source code for a few more modules files: Doc/library/code.rst | 2 +- Doc/library/codecs.rst | 2 +- Doc/library/configparser.rst | 2 ++ Doc/library/csv.rst | 1 + Doc/library/datetime.rst | 2 ++ Doc/library/decimal.rst | 3 ++- Doc/library/difflib.rst | 2 ++ 7 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Doc/library/code.rst b/Doc/library/code.rst --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,7 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -165,4 +166,3 @@ newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -7,6 +7,7 @@ .. sectionauthor:: Marc-Andr? Lemburg .. sectionauthor:: Martin v. L?wis +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -1418,4 +1419,3 @@ BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,6 +11,8 @@ .. sectionauthor:: Christopher G. Petrilli .. sectionauthor:: ?ukasz Langa +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -5,6 +5,7 @@ :synopsis: Write and read tabular data to and from delimited files. .. sectionauthor:: Skip Montanaro +**Source code:** :source:`Lib/csv.py` .. index:: single: csv diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters .. sectionauthor:: A.M. Kuchling +**Source code:** :source:`Lib/datetime.py` + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah .. sectionauthor:: Raymond D. Hettinger +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -2092,4 +2094,3 @@ >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters .. Markup by Fred L. Drake, Jr. +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 19 21:44:26 2014 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Mar 2014 21:44:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2314332=3A_provide_a_bett?= =?utf-8?q?er_explanation_of_junk_in_difflib_docs?= Message-ID: <3fq3By2Jzpz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/0a69b1e8b7fe changeset: 89861:0a69b1e8b7fe user: Andrew Kuchling date: Wed Mar 19 16:43:06 2014 -0400 summary: #14332: provide a better explanation of junk in difflib docs Initial patch by Alba Magallanes. files: Doc/library/difflib.rst | 14 +++++++++++--- Lib/difflib.py | 26 +++++++++++++------------- 2 files changed, 24 insertions(+), 16 deletions(-) diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -27,7 +27,9 @@ little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -210,7 +212,7 @@ Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -224,7 +226,7 @@ *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -624,6 +626,12 @@ length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: diff --git a/Lib/difflib.py b/Lib/difflib.py --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -853,10 +853,9 @@ and return true iff the string is junk. The module-level function `IS_LINE_JUNK` may be used to filter out lines without visible characters, except for at most one splat ('#'). It is recommended - to leave linejunk None; as of Python 2.3, the underlying - SequenceMatcher class has grown an adaptive notion of "noise" lines - that's better than any static definition the author has ever been - able to craft. + to leave linejunk None; the underlying SequenceMatcher class has + an adaptive notion of "noise" lines that's better than any static + definition the author has ever been able to craft. - `charjunk`: A function that should accept a string of length 1. The module-level function `IS_CHARACTER_JUNK` may be used to filter out @@ -1299,17 +1298,18 @@ Compare `a` and `b` (lists of strings); return a `Differ`-style delta. Optional keyword parameters `linejunk` and `charjunk` are for filter - functions (or None): + functions, or can be None: - - linejunk: A function that should accept a single string argument, and + - linejunk: A function that should accept a single string argument and return true iff the string is junk. The default is None, and is - recommended; as of Python 2.3, an adaptive notion of "noise" lines is - used that does a good job on its own. + recommended; the underlying SequenceMatcher class has an adaptive + notion of "noise" lines. - - charjunk: A function that should accept a string of length 1. The - default is module-level function IS_CHARACTER_JUNK, which filters out - whitespace characters (a blank or tab; note: bad idea to include newline - in this!). + - charjunk: A function that accepts a character (string of length + 1), and returns true iff the character is junk. The default is + the module-level function IS_CHARACTER_JUNK, which filters out + whitespace characters (a blank or tab; note: it's a bad idea to + include newline in this!). Tools/scripts/ndiff.py is a command-line front-end to this function. @@ -1680,7 +1680,7 @@ tabsize -- tab stop spacing, defaults to 8. wrapcolumn -- column number where lines are broken and wrapped, defaults to None where lines are not wrapped. - linejunk,charjunk -- keyword arguments passed into ndiff() (used to by + linejunk,charjunk -- keyword arguments passed into ndiff() (used by HtmlDiff() to generate the side by side HTML differences). See ndiff() documentation for argument default values and descriptions. """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 02:53:27 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 02:53:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_add_Nehal_Huss?= =?utf-8?q?ain?= Message-ID: <3fqB3W4W38z7LjW@mail.python.org> http://hg.python.org/cpython/rev/e27d3d9379cd changeset: 89862:e27d3d9379cd branch: 3.4 parent: 89858:4f6ac7d65ef1 user: Benjamin Peterson date: Wed Mar 19 20:52:17 2014 -0500 summary: add Nehal Hussain files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -574,6 +574,7 @@ Jim Hugunin Greg Humphreys Eric Huss +Nehal Hussain Taihyun Hwang Jeremy Hylton Ludwig H?hne -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 02:53:28 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 02:53:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fqB3X6Mx2z7LjW@mail.python.org> http://hg.python.org/cpython/rev/2978643766b9 changeset: 89863:2978643766b9 parent: 89861:0a69b1e8b7fe parent: 89862:e27d3d9379cd user: Benjamin Peterson date: Wed Mar 19 20:52:25 2014 -0500 summary: merge 3.4 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -574,6 +574,7 @@ Jim Hugunin Greg Humphreys Eric Huss +Nehal Hussain Taihyun Hwang Jeremy Hylton Ludwig H?hne -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 02:53:30 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 02:53:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_Nehal_Huss?= =?utf-8?q?ain?= Message-ID: <3fqB3Z0bq6z7Lkj@mail.python.org> http://hg.python.org/cpython/rev/bb48e705d92b changeset: 89864:bb48e705d92b branch: 2.7 parent: 89845:be1e015a8405 user: Benjamin Peterson date: Wed Mar 19 20:52:17 2014 -0500 summary: add Nehal Hussain files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -470,6 +470,7 @@ Jim Hugunin Greg Humphreys Eric Huss +Nehal Hussain Jeremy Hylton Ludwig H?hne Gerhard H?ring -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 02:55:43 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 02:55:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogc3luYyB3aXRoIDMu?= =?utf-8?q?4_branch?= Message-ID: <3fqB67639Vz7LjW@mail.python.org> http://hg.python.org/cpython/rev/7860a42bcf98 changeset: 89865:7860a42bcf98 branch: 2.7 user: Benjamin Peterson date: Wed Mar 19 20:54:59 2014 -0500 summary: sync with 3.4 branch files: Misc/ACKS | 383 ++++++++++++++++++++++++++++++++++++----- 1 files changed, 334 insertions(+), 49 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1,4 +1,3 @@ - Acknowledgements ---------------- @@ -14,19 +13,26 @@ Aahz Michael Abbott +Rajiv Abraham David Abrahams +Marc Abramowitz +Ron Adam +Ali Afshar Jim Ahlstrom Farhan Ahmad +Matthew Ahrens Nir Aides Yaniv Aknin Jyrki Alakuijala Steve Alexander Fred Allen +Ray Allen Billy G. Allie Kevin Altis Joe Amenta A. Amoroso Mark Anacker +Shashwat Anand Anders Andersen John Anderson Pehr Anderson @@ -35,20 +41,25 @@ Ross Andrus Juancarlo A?ez Chris Angelico +J?r?my Anger Ankur Ankan +Jon Anglin Heidi Annexstad +Ramchandra Apte ?ric Araujo +Alicia Arlen Jeffrey Armstrong Jason Asbahr David Ascher Chris AtLee +Aymeric Augustin John Aycock -Jan-Hein B?hrman Donovan Baarda Arne Babenhauserheide Attila Babo +Matt Bachmann +Marcin Bachry Alfonso Baciero -Marcin Bachry Dwayne Bailey Stig Bakken Greg Ball @@ -56,53 +67,66 @@ Jeff Balogh Manuel Balsera Matt Bandy +Dmi Baranov Michael J. Barber Daniel Barclay +Nicolas Bareil Chris Barker Anton Barkovsky Nick Barnes Quentin Barnes +David Barnett Matthew Barnett Richard Barran Cesar Eduardo Barros Des Barry Ulf Bartelt Don Bashford +Pior Bastida Nick Bastin Ned Batchelder Jeff Bauer -Mike Bayer Michael R Bax Anthony Baxter +Mike Bayer Samuel L. Bayer Donald Beaudry David Beazley +Neal Becker Robin Becker -Neal Becker Torsten Becker Bill Bedford +Stefan Behnel Reimer Behrends Ben Bell Thomas Bellman Alexander ?????? Belopolsky +Eli Bendersky David Benjamin Oscar Benjamin Andrew Bennetts Andy Bensky Bennett Benson +Ezra Berch Michel Van den Bergh +Julian Berman +Brice Berna +Olivier Bernard Eric Beser Steven Bethard Stephen Bevan Ron Bickers Natalia B. Bidart +Adrian von Bidder David Binger Dominic Binks Philippe Biondi Michael Birtwell Stuart Bishop Roy Bixler +Daniel Black Jonathan Black +Renaud Blanch Mike Bland Martin Bless Pablo Bleyer @@ -115,9 +139,13 @@ David Bolen Wouter Bolsterlee Gawain Bolton +Forest Bond Gregory Bond +Matias Bordese +Jonas Borgstr?m Jurjen Bos Peter Bosch +Dan Boswell Eric Bouck Thierry Bousch Sebastian Boving @@ -128,22 +156,32 @@ Georg Brandl Christopher Brannon Terrence Brannon +Germ?n M. Bravo +Sven Brauch Erik Bray Brian Brazil +Demian Brecht Dave Brennan Tom Bridgman +Anthony Briggs Keith Briggs +Tobias Brink Richard Brodie Michael Broghton Daniel Brotsky Jean Brouwers Gary S. Brown +Titus Brown Oleg Broytmann Dave Brueck Francisco Mart?n Brugu? Ian Bruntlett +Floris Bruynooghe +Matt Bryant Stan Bubrouski Erik de Bueger +Jan-Hein B?hrman +Lars Buitinck Dick Bulterman Bill Bumgarner Jimmy Burgett @@ -154,19 +192,22 @@ Tarn Weisner Burton Lee Busby Ralph Butler +Nicolas Cadou Jp Calderone Arnaud Calmettes Daniel Calvelo Tony Campbell Brett Cannon Mike Carlton +Pierre Carrier Terry Carroll -Damien Cassou +Edward Catmur Lorenzo M. Catucci Donn Cave Charles Cazabon Jes?s Cea Avi?n Per Cederqvist +Matej Cepl Carl Cerecke Octavian Cerna Dave Chambers @@ -174,13 +215,16 @@ John Chandler Hye-Shik Chang Jeffrey Chang +Godefroid Chapelle +Brad Chapman +Greg Chapman Mitch Chapman -Greg Chapman -Brad Chapman Yogesh Chaudhari David Chaum Nicolas Chauvat +Jerry Chen Michael Chermside +Ingrid Cheung Albert Chin-A-Young Adal Chiriliuc Matt Chisholm @@ -191,21 +235,30 @@ David Cinege Craig Citro Gilles Civario +Chris Clark +Laurie Clark-Michalek Mike Clarkson Andrew Clegg Brad Clements +Robbie Clemons Steve Clift +Herv? Coatanhay Nick Coghlan Josh Cogliati Dave Cole +Terrence Cole Benjamin Collar Jeffery Collins Robert Collins Paul Colomiets +Christophe Combelles Geremy Condra +Denver Coneybeare +Phil Connell Juan Jos? Conti Matt Conway David M. Cooke +Jason R. Coombs Garrett Cooper Greg Copeland Aldo Cortesi @@ -227,8 +280,10 @@ Joaquin Cuenca Abela John Cugini Tom Culliton +Antonio Cuni Brian Curtin Lisandro Dalcin +Darren Dale Andrew Dalke Lars Damerow Evan Dandrea @@ -237,35 +292,45 @@ Ben Darnell Kushal Das Jonathan Dasteel +Pierre-Yves David A. Jesse Jiryu Davis +Merlijn van Deen John DeGood Ned Deily Vincent Delft Arnaud Delobelle +Konrad Delong Erik Demaine John Dennis L. Peter Deutsch Roger Dev Philippe Devalkeneer Raghuram Devarakonda +Caleb Deveraux Catherine Devlin Scott Dial Toby Dickenson Mark Dickinson Jack Diederich Daniel Diniz +Humberto Diogenes Yves Dionne Daniel Dittmar +Josip Djolonga Jaromir Dolecek Ismail Donmez Robert Donohue Marcos Donolo Dima Dorfman +Yves Dorfsman Cesar Douady Dean Draayer Fred L. Drake, Jr. +Derk Drukker John DuBois Paul Dubois +Jacques Ducasse +Andrei Dorian Duma Graham Dumpleton Quinn Dunkan Robin Dunn @@ -279,36 +344,52 @@ Maxim Dzumanenko Walter D?rwald Hans Eckardt +Rodolpho Eckhardt +Ulrich Eckhardt David Edelsohn +John Edmonds Grant Edwards John Ehresman +Tal Einat Eric Eisner Andrew Eland +Julien ?lie Lance Ellinghaus +Daniel Ellis +Phil Elson David Ely Jeff Epler Tom Epperly +G?kcen Eraslan Stoffel Erasmus J?rgen A. Erhard Michael Ernst Ben Escoto Andy Eskilsson +Andr? Espaze Stefan Esser +Nicolas Estibals +Carey Evans Stephen D Evans -Carey Evans Tim Everett Paul Everitt David Everly Daniel Evers +Winston Ewert Greg Ewing Martijn Faassen Clovis Fabricio Andreas Faerber Bill Fancher +Michael Farrell Troy J. Farrell Mark Favas +Boris Feld +Thomas Fenzl Niels Ferguson Sebastian Fernandez +Florian Festi +John Feuerstein Carl Feynman Vincent Fiack Tomer Filiba @@ -325,6 +406,7 @@ Amaury Forgeot d'Arc Doug Fort John Fouhy +Andrew Francis Stefan Franke Martin Franklin Kent Frazier @@ -332,10 +414,12 @@ Robin Friedrich Bradley Froehle Ivan Frohne +Matthias Fuchs Jim Fulton Tadayoshi Funaba Gyro Funch Peter Funk +Ethan Furman Geoff Furnish Ulisses Furquim Hagen F?rstenau @@ -347,6 +431,7 @@ Yitzchak Gale Matthew Gallagher Quentin Gallet-Gilles +Riccardo Attilio Galli Raymund Galvin Nitin Ganatra Fred Gansevles @@ -368,9 +453,15 @@ Johannes Gijsbers Michael Gilfix Julian Gindi +Yannick Gingras +Matt Giuca Wim Glenn +Michael Goderbauer +Jeroen Van Goey Christoph Gohlke Tim Golden +Guilherme Gon?alves +Tiago Gon?alves Chris Gonnerman Shelley Gooch David Goodger @@ -378,16 +469,18 @@ Nathaniel Gray Eddy De Greef Grant Griffin +Andrea Griffini Duncan Grisby Fabian Groffen -John S. Gruber +Eric Groo Dag Gruneau Filip Gruszczy?ski Thomas Guettler -Ajitesh Gupta +Anuj Gupta Michael Guravage Lars Gust?bel Thomas G?ttler +Jonas H. Barry Haddow Philipp Hagemeister Paul ten Hagen @@ -395,9 +488,11 @@ Peter Haight V?clav Haisman Zbigniew Halas +Walker Hale IV Bob Halley Jesse Hallio Jun Hamano +Alexandre Hamelin Anders Hammarquist Mark Hammond Harald Hanche-Olsen @@ -408,6 +503,8 @@ Lynda Hardman Derek Harland Jason Harper +Brian Harring +Jonathan Hartley Travis B. Hartwell Larry Hastings Tim Hatch @@ -416,12 +513,14 @@ Rycharde Hawkes Ben Hayden Jochen Hayek +Tim Heaney Henrik Heimbuerger Christian Heimes Thomas Heller Malte Helmert Lance Finn Helsten Jonathan Hendry +Michael Henry James Henstridge Kasun Herath Chris Herborth @@ -434,6 +533,7 @@ Magnus L. Hetland Raymond Hettinger Kevan Heydon +Kelsey Hightower Jason Hildebrand Richie Hindle Konrad Hinsen @@ -452,6 +552,7 @@ Gerrit Holl Shane Holloway Rune Holm +Thomas Holmes Philip Homburg Naofumi Honda Jeffrey Honig @@ -460,86 +561,111 @@ Brian Hooper Randall Hopper Nadav Horesh +Alon Horev Jan Hosang +Alan Hourihane Ken Howard Brad Howes Mike Hoy Chih-Hao Huang +Christian Hudon Lawrence Hudson Michael Hudson Jim Hugunin Greg Humphreys Eric Huss Nehal Hussain +Taihyun Hwang Jeremy Hylton Ludwig H?hne Gerhard H?ring Fredrik H??rd Catalin Iacob Mihai Ibanescu +Ali Ikinci +Aaron Iles Lars Immisch Bobby Impollonia Meador Inge +Peter Ingebretson Tony Ingraldi John Interrante Bob Ippolito Roger Irwin Atsuo Ishimoto +Adam Jackson +Ben Jackson Paul Jackson -Ben Jackson Manuel Jacob David Jacobs Kevin Jacobs Kjetil Jacobsen +Bertrand Janin Geert Jansen Jack Jansen Bill Janssen Thomas Jarosch +Juhana Jauhiainen Rajagopalasarma Jayakrishnan +Zbigniew J?drzejewski-Szmek +Julien Jehannet Drew Jenkins Flemming Kj?r Jensen Philip H. Jensen Philip Jenvey +MunSic Jeong Chris Jerdonek -Jiba +Jim Jewett Pedro Diaz Jimenez Orjan Johansen Fredrik Johansson Gregory K. Johnson Kent Johnson +Michael Johnson Simon Johnston +Matt Joiner +Thomas Jollans Nicolas Joly +Brian K. Jones Evan Jones Jeremy Jones Richard Jones Irmen de Jong Lucas de Jonge +Kristj?n Valur J?nsson +Jens B. Jorgensen John Jorgensen -Jens B. Jorgensen Sijin Joseph Andreas Jung Tattoo Mabonzo K. Sarah K. +Sunny K Bohuslav Kabrda +Alexey Kachayev Bob Kahn Kurt B. Kaiser Tamito Kajiyama +Jan Kaliszewski Peter van Kampen +Rafe Kaplan Jacob Kaplan-Moss -Piotr Kasprzyk +Janne Karila +Per ?yvind Karlsen Anton Kasyanov Lou Kates Hiroaki Kawai Sebastien Keim Ryan Kelly +Dan Kenigsberg +Randall Kern Robert Kern -Randall Kern Jim Kerr Magnus Kessler Lawrence Kesteloot -Rafe Kettler Vivek Khera +Dhiru Kholia Mads Kiilerich +Jason Killen Jan Kim Taek Joo Kim W. Trevor King @@ -549,6 +675,7 @@ Kamil Kisiel Akira Kitada Ron Klatchko +Reid Kleckner Bastian Kleineidam Bob Kline Matthias Klose @@ -558,12 +685,17 @@ Lenny Kneler Pat Knight Jeff Knupp +Kubilay Kocak Greg Kochanski Damon Kohler Marko Kohtala Vajrasky Kok Guido Kollerie +Jacek Konieczny +???? ????????? +Arkady Koplyarov Peter A. Koren +Vlad Korolev Joseph Koshy Daniel Kozan Jerzy Kozera @@ -575,34 +707,45 @@ Michael Kremer Fabian Kreutz C?dric Krier +Pedro Kroger Hannu Krosing Andrej Krpic Ivan Krsti? Steven Kryskalla Andrew Kuchling -Ralf W. Grosse-Kunstleve Dave Kuhlman +Jon Kuhn +Toshio Kuratomi Vladimir Kushnir Erno Kuusela -Kirill Kuzminykh (?????? ?????????) Ross Lagerwall Cameron Laird +David Lam Thomas Lamb +Valerie Lambert +Jean-Baptiste "Jiba" Lamy +Ronan Lamy Torsten Landschoff ?ukasz Langa Tino Lange +Glenn Langford Andrew Langmead Detlef Lannert Soren Larsen +Amos Latteier Piers Lauder Ben Laurie Simon Law +Julia Lawall Chris Lawrence Brian Leair +Mathieu Leduc-Hamel +Christopher Lee +Inyeol Lee +James Lee John J. Lee -Inyeol Lee Thomas Lee -Christopher Lee +Tennessee Leeuwenburg Luc Lefebvre Pierre Paul Lefebvre Glyph Lefkowitz @@ -617,14 +760,20 @@ Mateusz Lenik John Lenton Kostyantyn Leschenko +Benno Leslie Christopher Tur Lesniewski-Laas +Alain Leufroy Mark Levinson +Mark Levitt William Lewis +Akira Li Xuanji Li Robert van Liere Ross Light Shawn Ligocki Martin Ligr +Gediminas Liktaras +Grant Limberg Christopher Lindblad Ulf A. Lindgren Bj?rn Lindqvist @@ -635,47 +784,63 @@ Mirko Liss Nick Lockwood Stephanie Lockwood +Hugo Lopes Tavares Anne Lord Tom Loredo Justin Love +Ned Jackson Lovely Jason Lowe Tony Lownds Ray Loyzaga +Kang-Hao (Kenny) Lu Lukas Lueg Loren Luke Fredrik Lundh Zhongyue Luo Mark Lutz +Taras Lyapun Jim Lynch Mikael Lyngvig Martin von L?wis -Till Maas +Guillermo L?pez-Anglada Jeff MacDonald John Machin Andrew I MacIntyre Tim MacKenzie Nick Maclaren +Don MacMillen +Tomasz Ma?kowiak Steve Majewski +Marek Majkowski Grzegorz Makarewicz David Malcolm +Greg Malcolm +William Mallard Ken Manheimer Vladimir Marangozov +Colin Marc Vincent Marchetti David Marek Doug Marien Sven Marnach Alex Martelli Anthony Martin +Owen Martin Westley Mart?nez S?bastien Martini +Sidney San Mart?n Roger Masse Nick Mathewson +Simon Mathieu Laura Matson Graham Matthews Martin Matusiak Dieter Maurer Daniel May +Madison May +Lucas Maystre Arnaud Mazin +Matt McClure Rebecca McCreary Kirk McDonald Chris McDonough @@ -683,10 +848,11 @@ Alan McIntyre Jessica McKellar Michael McLay +Brendan McLoughlin Mark Mc Mahon Gordon McMillan +Andrew McNamara Caolan McNamara -Andrew McNamara Jeff McNeil Craig McPheeters Lambert Meertens @@ -699,39 +865,55 @@ Carl Meyer Mike Meyer Piotr Meyer +Alexis M?taireau Steven Miale +Trent Mick Jason Michalski -Trent Mick +Franck Michea Tom Middleton +Thomas Miedema Stan Mihai +Stefan Mihaila Aristotelis Mikropoulos Paolo Milani +Chad Miller Damien Miller -Chad Miller Jason V. Miller Jay T. Miller +Katie Miller Roman Milner +Julien Miotte Andrii V. Mishkovskyi +Dom Mitchell Dustin J. Mitchell -Dom Mitchell +Zubin Mithra Florian Mladitsch Doug Moen -Jaakko Moisio The Dragon De Monsyne +Bastien Montagne Skip Montanaro +Peter Moody Paul Moore Ross Moore +Ben Morgan Derek Morr James A Morrison +Martin Morrison +Derek McTavish Mounce Alessandro Moura Pablo Mouzo +Mher Movsisyan Ruslan Mstoi +Valentina Mukhamedzhanova +Michael Mulich +Sape Mullender Sjoerd Mullender -Sape Mullender Michael Muller Neil Muller +Louis Munro R. David Murray Matti M?ki +J?rg M?ller Dale Nagata John Nagle Takahiro Nakayama @@ -739,17 +921,18 @@ Charles-Fran?ois Natali Vilmos Nebehaj Fredrik Nehr +Tony Nelson Trent Nelson -Tony Nelson Chad Netzer Max Neunh?ffer George Neville-Neil Hieu Nguyen Johannes Nicolai Samuel Nicolary +Jonathan Niehof Gustavo Niemeyer Oscar Nierstrasz -Hrvoje Niksic +Hrvoje Nik?i? Gregory Nofi Jesse Noller Bill Noon @@ -757,13 +940,19 @@ Tim Northover Joe Norton Neal Norwitz +Mikhail Novikov Michal Nowikowski Steffen Daode Nurpmeso Nigel O'Brian +John O'Connor Kevin O'Connor Tim O'Malley Zooko O'Whielacronx +Aaron Oakley +James Oakley Elena Oat +Jon Oberheide +Milan Oberkirch Pascal Oberndoerfer Jeffrey Ollie Adam Olsen @@ -776,6 +965,7 @@ Michele Orr? Oleg Oshmyan Denis S. Otkidach +Peter Otten Michael Otteneder R. M. Oudkerk Russel Owen @@ -787,6 +977,7 @@ Todd R. Palmer Juan David Ib??ez Palomar Jan Palus +Mathias Panzenb?ck M. Papillon Peter Parente Alexandre Parenteau @@ -794,57 +985,82 @@ William Park Heikki Partanen Harri Pasanen +Ga?l Pasgrimaud +Ashish Nitin Patil Randy Pausch Samuele Pedroni +Justin Peel Marcel van der Peijl Berker Peksag Andreas Pelme Steven Pemberton Bo Peng Santiago Peres?n +George Peristerakis +Mathieu Perreault Mark Perrego Trevor Perrin Gabriel de Perthuis Tim Peters Benjamin Peterson +Joe Peterson Chris Petrilli +Roumen Petrov Bjorn Pettersen Justin D. Pettit +Esa Peuha +Ronny Pfannschmidt Geoff Philbrick Gavrie Philipson Adrian Phillips Christopher J. Phoenix Neale Pickett -Matti Picus Jim St. Pierre Dan Pierson Martijn Pieters +Anand B. Pillai Fran?ois Pinard Tom Pinckney Zach Pincus +Zero Piraeus Michael Piotrowski Antoine Pitrou Jean-Fran?ois Pi?ronne Oleg Plakhotnyuk +Remi Pointel +Ariel Poliak Guilherme Polo +Illia Polosukhin Michael Pomraning +Martin Pool Iustin Pop +Claudiu Popa John Popplewell +Guillaume Pratte Amrit Prem Paul Prescod Donovan Preston +Paul Price +Iuliia Proskurnia +Jyrki Pulliainen Steve Purcell +Eduardo P?rez Fernando P?rez -Eduardo P?rez +Pierre Quentel Brian Quinlan +Kevin Jing Qiu Anders Qvist Thomas Rachel +Ram Rachum +J?r?me Radix Burton Radons Jeff Ramnani Brodie Rao +Senko Rasic Antti Rasinen Nikolaus Rath Sridhar Ratnakumar +Ysj Ray Eric S. Raymond Edward K. Ream Chris Rebert @@ -854,37 +1070,47 @@ Gareth Rees Steve Reeves Lennart Regebro +John Regehr +Federico Reghenzani Ofir Reichenberg Sean Reifschneider Michael P. Reilly Bernhard Reiter Steven Reiz Roeland Rengelink +Antoine Reversat +Fl?vio Ribeiro +Francesco Ricciardi Tim Rice -Francesco Ricciardi Jan Pieter Riegel Armin Rigo +Arc Riley Nicholas Riley Jean-Claude Rimbault Vlad Riscutia Wes Rishel -Dan Riti +Daniel Riti Juan M. Bello Rivas Davide Rizzo Anthony Roach Carl Robben Mark Roberts +Andy Robinson Jim Robinson -Andy Robinson +Mark Roddy Kevin Rodgers Sean Rodman Giampaolo Rodola +Elson Rodriguez Adi Roiban +Luis Rojas Mike Romberg Armin Ronacher Case Roole Timothy Roscoe +Erik Rose Jim Roskind +Brian Rosner Guido van Rossum Just van Rossum Hugo van Rossum @@ -896,22 +1122,28 @@ Clinton Roy Paul Rubin Sam Ruby +Demur Rumed Audun S. Runde Eran Rundstein Rauli Ruohonen Jeff Rush Sam Rushing Mark Russell +Rusty Russell Nick Russo Chris Ryland Constantina S. +Patrick Sabin S?bastien Sabl? Suman Saha Hajime Saitou George Sakkis Rich Salz Kevin Samborn +Adrian Sampson +James Sanders Ilya Sandler +Rafael Santos Simon Sapin Mark Sapiro Ty Sarna @@ -920,39 +1152,46 @@ Ben Sayer sbt Marco Scataglini +Andrew Schaaf Michael Scharf +Andreas Schawo Neil Schemenauer David Scherer +Wolfgang Scherer Hynek Schlawack +Bob Schmertz Gregor Schmid Ralf Schmitt Michael Schneider Peter Schneider-Kamp Arvin Schnell Scott Schram +Robin Schreiber Chad J. Schroeder Christian Schubert Sam Schulenburg Stefan Schwarzer Dietmar Schwertberger Federico Schwindt +Barry Scott Steven Scott -Barry Scott Nick Seidenman -?iga Seilnach +?iga Seilnacht Yury Selivanov Fred Sells Jiwon Seo +I?igo Serna Joakim Sernbrant -Roger Serwy +Roger D. Serwy Jerry Seutter Pete Sevander Denis Severson Ian Seyer -Aman Shah +Daniel Shahaf Ha Shao Mark Shannon Richard Shapiro +Varun Sharma Vlad Shcherbina Justin Sheehy Charlie Shepherd @@ -962,18 +1201,18 @@ Michael Shiplett John W. Shipman Joel Shprentz -Itamar Shtull-Trauring Yue Shuaijie Terrel Shumway Eric Siegerman Paul Sijben +SilentGhost Tim Silk Michael Simcich Ionel Simionescu Kirill Simonov Nathan Paul Simons Guilherme Sim?es -Kyle Simpson +Adam Simpkins Ravi Sinha Janne Sinkkonen Ng Pheng Siong @@ -981,16 +1220,21 @@ J. Sipprell Kragen Sitaker Michael Sloan +Nick Sloan +V?clav ?milauer +Christopher Smith Eric V. Smith -Christopher Smith Gregory P. Smith +Mark Smith Roy Smith Ryan Smith-Roberts Rafal Smotrzyk +Eric Snow Dirk Soede Paul Sokolovsky Evgeny Sologubov Cody Somerville +Edoardo Spadolini Clay Spence Stefan Sperling Nicholas Spies @@ -1002,10 +1246,13 @@ Tage Stabell-Kulo Quentin Stafford-Fraser Frank Stajano +Joel Stanley Anthony Starks Oliver Steele Greg Stein +Baruch Sterin Chris Stern +Alex Stewart Victor Stinner Richard Stoakley Peter Stoehr @@ -1013,28 +1260,34 @@ Michael Stone Serhiy Storchaka Ken Stox -Patrick Strawderman Dan Stromberg +Donald Stufft Daniel Stutzbach Andreas St?hrk +Colin Su +Pal Subbiah Nathan Sullivan Mark Summerfield Reuben Sumner +Marek ?uppa Hisao Suzuki +Kalle Svensson Andrew Svetlov -Kalle Svensson Paul Swartz Thenault Sylvain P?ter Szab? Amir Szekely Arfrever Frehtes Taifersar Arahesis +Hideaki Takahashi +Indra Talip +Neil Tallim Geoff Talvola +Musashi Tamura William Tanksley Christian Tanzer -Stefano Taschini Steven Taschuk +Amy Taylor Monty Taylor -Amy Taylor Anatoly Techtonik Mikhail Terekhov Victor Terr?n @@ -1045,28 +1298,40 @@ Nicolas M. Thi?ry James Thomas Robin Thomas +Brian Thorne Stephen Thorne +Jeremy Thurgood Eric Tiedemann +July Tikhonov Tracy Tims Oren Tirosh Jason Tishler Christian Tismer Jim Tittsler Frank J. Tobin +Bennett Todd R Lindsay Todd -Bennett Todd Eugene Toder +Erik Tollerud +Stephen Tonkin Matias Torchinsky Sandro Tosi Richard Townsend +David Townshend +Nathan Trapuzzano Laurence Tratt +Alberto Trevino +Matthias Troffaes Tom Tromey John Tromp +Diane Trout Jason Trowbridge Brent Tubbs Anthony Tuininga +Erno Tukia David Turner Stephen Turner +Itamar Turner-Trauring Theodore Turocy Bill Tutt Fraser Tweedale @@ -1074,19 +1339,23 @@ Eren T?rkay Lionel Ulmer Roger Upole +Daniel Urban Michael Urman Hector Urtubia Ville Vainio Andi Vajda Case Van Horsen Kyle VanderBeek +Andrew Vant Atul Varma Dmitry Vasiliev Sebastian Ortiz Vasquez Alexandre Vassalotti +Nadeem Vawda Frank Vercruesse Mike Verdone Jaap Vermeulen +Nikita Vetoshkin Al Vezza Jacques A. Vidrine John Viega @@ -1094,22 +1363,28 @@ Kannan Vijayan Kurt Vile Norman Vine +Pauli Virtanen Frank Visser Johannes Vogel Alex Volkov Martijn Vries +Sjoerd de Vries Niki W. Waibel Wojtek Walczak Charles Waldman Richard Walker Larry Wall Kevin Walzer +Rodrigo Steinmuller Wanderley Ke Wang Greg Ward +Tom Wardill Zachary Ware +Jonas Wagner Barry Warsaw Steve Waterbury Bob Watson +David Watson Aaron Watters Henrik Weber Corran Webster @@ -1129,12 +1404,14 @@ Felix Wiemann Gerry Wiener Frank Wierzbicki +Santoso Wijaya Bryce "Zooko" Wilcox-O'Hearn Timothy Wild +Jakub Wilk +Gerald S. Williams Jason Williams John Williams Sue Williams -Gerald S. Williams Steven Willis Frank Willison Geoff Wilson @@ -1150,25 +1427,30 @@ John Wiseman Chris Withers Stefan Witzel +Irek Wlizlo David Wolever Klaus-Juergen Wolf Dan Wolfe Richard Wolff Adam Woodbeck Steven Work +Gordon Worley Darren Worrall -Gordon Worley Thomas Wouters +Daniel Wozniak Heiko Wundram Doug Wyatt Robert Xiao Florent Xicluna Hirokazu Yamamoto Ka-Ping Yee +Jason Yeo +EungJun Yi Bob Yodlowski Danny Yoo Rory Yorke George Yoshida +Kazuhiro Yoshida Masazumi Yoshikawa Arnaud Ysmal Bernard Yue @@ -1179,7 +1461,10 @@ Yury V. Zaytsev Siebren van der Zee Nickolai Zeldovich +Yuxiao Zeng Uwe Zessin Cheng Zhang +Kai Zhu Tarek Ziad? +Gennadiy Zlobin Peter ?strand -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 08:55:49 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 08:55:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTc4?= =?utf-8?q?=3A_Remove_last_part_of_OS/2_support_in_distutils?= Message-ID: <3fqL5d3QBtz7LjT@mail.python.org> http://hg.python.org/cpython/rev/6718d00b872b changeset: 89866:6718d00b872b branch: 3.4 parent: 89862:e27d3d9379cd user: Victor Stinner date: Thu Mar 20 08:50:33 2014 +0100 summary: Issue #20978: Remove last part of OS/2 support in distutils files: Lib/distutils/spawn.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -36,8 +36,6 @@ _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': _spawn_nt(cmd, search_path, dry_run=dry_run) - elif os.name == 'os2': - _spawn_os2(cmd, search_path, dry_run=dry_run) else: raise DistutilsPlatformError( "don't know how to spawn programs on platform '%s'" % os.name) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 08:55:50 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 08:55:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTc4?= =?utf-8?q?=3A_pyflakes=3A_fix_undefined_names?= Message-ID: <3fqL5f517Vz7LjM@mail.python.org> http://hg.python.org/cpython/rev/8de5d0809dd8 changeset: 89867:8de5d0809dd8 branch: 3.4 user: Victor Stinner date: Thu Mar 20 08:50:52 2014 +0100 summary: Issue #20978: pyflakes: fix undefined names files: Lib/ftplib.py | 1 - Lib/multiprocessing/connection.py | 2 +- Lib/tempfile.py | 2 +- 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -320,7 +320,6 @@ raise err else: raise OSError("getaddrinfo returns an empty list") - raise OSError(msg) sock.listen(1) port = sock.getsockname()[1] # Get proper port host = self.sock.getsockname()[0] # Get proper host diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -29,7 +29,7 @@ try: import _winapi - from _winapi import WAIT_OBJECT_0, WAIT_TIMEOUT, INFINITE + from _winapi import WAIT_OBJECT_0, WAIT_ABANDONED_0, WAIT_TIMEOUT, INFINITE except ImportError: if sys.platform == 'win32': raise diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -72,7 +72,7 @@ # file doesn't exist. def _stat(fn): fd = _os.open(fn, _os.O_RDONLY) - os.close(fd) + _os.close(fd) def _exists(fn): try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 08:55:51 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 08:55:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2320978=3A_pyflakes=3A_fix_unde?= =?utf-8?q?fined_names=3B_remove_last_part_of?= Message-ID: <3fqL5g6qn5z7Ljt@mail.python.org> http://hg.python.org/cpython/rev/52404c14be07 changeset: 89868:52404c14be07 parent: 89863:2978643766b9 parent: 89867:8de5d0809dd8 user: Victor Stinner date: Thu Mar 20 08:51:24 2014 +0100 summary: (Merge 3.4) Issue #20978: pyflakes: fix undefined names; remove last part of OS/2 support in distutils files: Lib/distutils/spawn.py | 2 -- Lib/ftplib.py | 1 - Lib/multiprocessing/connection.py | 2 +- Lib/tempfile.py | 2 +- 4 files changed, 2 insertions(+), 5 deletions(-) diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -36,8 +36,6 @@ _spawn_posix(cmd, search_path, dry_run=dry_run) elif os.name == 'nt': _spawn_nt(cmd, search_path, dry_run=dry_run) - elif os.name == 'os2': - _spawn_os2(cmd, search_path, dry_run=dry_run) else: raise DistutilsPlatformError( "don't know how to spawn programs on platform '%s'" % os.name) diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -320,7 +320,6 @@ raise err else: raise OSError("getaddrinfo returns an empty list") - raise OSError(msg) sock.listen(1) port = sock.getsockname()[1] # Get proper port host = self.sock.getsockname()[0] # Get proper host diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -29,7 +29,7 @@ try: import _winapi - from _winapi import WAIT_OBJECT_0, WAIT_TIMEOUT, INFINITE + from _winapi import WAIT_OBJECT_0, WAIT_ABANDONED_0, WAIT_TIMEOUT, INFINITE except ImportError: if sys.platform == 'win32': raise diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -72,7 +72,7 @@ # file doesn't exist. def _stat(fn): fd = _os.open(fn, _os.O_RDONLY) - os.close(fd) + _os.close(fd) def _exists(fn): try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 09:28:44 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 09:28:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTc2?= =?utf-8?q?=3A_pyflakes=3A_Remove_unused_imports?= Message-ID: <3fqLqc6sXmz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/f6f691ff27b9 changeset: 89869:f6f691ff27b9 branch: 3.4 parent: 89867:8de5d0809dd8 user: Victor Stinner date: Thu Mar 20 09:16:38 2014 +0100 summary: Issue #20976: pyflakes: Remove unused imports files: Lib/_strptime.py | 2 +- Lib/asynchat.py | 1 - Lib/compileall.py | 1 - Lib/ctypes/util.py | 2 +- Lib/decimal.py | 1 - Lib/difflib.py | 1 - Lib/email/_header_value_parser.py | 2 +- Lib/email/generator.py | 4 ---- Lib/email/header.py | 2 -- Lib/email/mime/text.py | 1 - Lib/email/parser.py | 2 -- Lib/email/quoprimime.py | 1 - Lib/email/utils.py | 3 --- Lib/functools.py | 2 +- Lib/http/client.py | 1 - Lib/http/server.py | 2 -- Lib/logging/config.py | 2 +- Lib/logging/handlers.py | 3 +-- Lib/mailbox.py | 1 - Lib/multiprocessing/connection.py | 1 - Lib/multiprocessing/heap.py | 2 -- Lib/multiprocessing/pool.py | 2 +- Lib/multiprocessing/popen_spawn_posix.py | 3 --- Lib/multiprocessing/reduction.py | 1 - Lib/multiprocessing/semaphore_tracker.py | 2 -- Lib/multiprocessing/synchronize.py | 2 -- Lib/multiprocessing/util.py | 2 -- Lib/nntplib.py | 1 - Lib/pickle.py | 2 +- Lib/platform.py | 1 - Lib/pydoc.py | 2 +- Lib/runpy.py | 1 - Lib/socketserver.py | 1 - Lib/sre_compile.py | 2 +- Lib/sre_parse.py | 2 -- Lib/ssl.py | 3 --- Lib/subprocess.py | 2 -- Lib/tarfile.py | 2 -- Lib/threading.py | 1 - Lib/tkinter/__init__.py | 2 -- Lib/tkinter/tix.py | 2 +- Lib/venv/__init__.py | 1 - Lib/xmlrpc/client.py | 1 - 43 files changed, 11 insertions(+), 64 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -14,7 +14,7 @@ import locale import calendar from re import compile as re_compile -from re import IGNORECASE, ASCII +from re import IGNORECASE from re import escape as re_escape from datetime import (date as datetime_date, timedelta as datetime_timedelta, diff --git a/Lib/asynchat.py b/Lib/asynchat.py --- a/Lib/asynchat.py +++ b/Lib/asynchat.py @@ -45,7 +45,6 @@ method) up to the terminator, and then control will be returned to you - by calling your self.found_terminator() method. """ -import socket import asyncore from collections import deque diff --git a/Lib/compileall.py b/Lib/compileall.py --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -12,7 +12,6 @@ """ import os import sys -import errno import importlib.util import py_compile import struct diff --git a/Lib/ctypes/util.py b/Lib/ctypes/util.py --- a/Lib/ctypes/util.py +++ b/Lib/ctypes/util.py @@ -85,7 +85,7 @@ elif os.name == "posix": # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump - import re, tempfile, errno + import re, tempfile def _findLib_gcc(name): expr = r'[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name) diff --git a/Lib/decimal.py b/Lib/decimal.py --- a/Lib/decimal.py +++ b/Lib/decimal.py @@ -142,7 +142,6 @@ # See http://speleotrove.com/decimal/ __libmpdec_version__ = "2.4.0" # compatible libmpdec version -import copy as _copy import math as _math import numbers as _numbers import sys diff --git a/Lib/difflib.py b/Lib/difflib.py --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -30,7 +30,6 @@ 'Differ','IS_CHARACTER_JUNK', 'IS_LINE_JUNK', 'context_diff', 'unified_diff', 'HtmlDiff', 'Match'] -import warnings import heapq from collections import namedtuple as _namedtuple diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -70,7 +70,7 @@ import re import urllib # For urllib.parse.unquote from string import hexdigits -from collections import namedtuple, OrderedDict +from collections import OrderedDict from email import _encoded_words as _ew from email import errors from email import utils diff --git a/Lib/email/generator.py b/Lib/email/generator.py --- a/Lib/email/generator.py +++ b/Lib/email/generator.py @@ -10,14 +10,10 @@ import sys import time import random -import warnings from copy import deepcopy from io import StringIO, BytesIO -from email._policybase import compat32 -from email.header import Header from email.utils import _has_surrogates -import email.charset as _charset UNDERSCORE = '_' NL = '\n' # XXX: no longer used by the code below. diff --git a/Lib/email/header.py b/Lib/email/header.py --- a/Lib/email/header.py +++ b/Lib/email/header.py @@ -100,7 +100,6 @@ words.append((encoded, encoding, charset)) # Now loop over words and remove words that consist of whitespace # between two encoded strings. - import sys droplist = [] for n, w in enumerate(words): if n>1 and w[1] and words[n-2][1] and words[n-1][0].isspace(): @@ -362,7 +361,6 @@ for string, charset in self._chunks: if hasspace is not None: hasspace = string and self._nonctext(string[0]) - import sys if lastcs not in (None, 'us-ascii'): if not hasspace or charset not in (None, 'us-ascii'): formatter.add_transition() diff --git a/Lib/email/mime/text.py b/Lib/email/mime/text.py --- a/Lib/email/mime/text.py +++ b/Lib/email/mime/text.py @@ -6,7 +6,6 @@ __all__ = ['MIMEText'] -from email.encoders import encode_7or8bit from email.mime.nonmultipart import MIMENonMultipart diff --git a/Lib/email/parser.py b/Lib/email/parser.py --- a/Lib/email/parser.py +++ b/Lib/email/parser.py @@ -7,11 +7,9 @@ __all__ = ['Parser', 'HeaderParser', 'BytesParser', 'BytesHeaderParser', 'FeedParser', 'BytesFeedParser'] -import warnings from io import StringIO, TextIOWrapper from email.feedparser import FeedParser, BytesFeedParser -from email.message import Message from email._policybase import compat32 diff --git a/Lib/email/quoprimime.py b/Lib/email/quoprimime.py --- a/Lib/email/quoprimime.py +++ b/Lib/email/quoprimime.py @@ -40,7 +40,6 @@ ] import re -import io from string import ascii_letters, digits, hexdigits diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -25,13 +25,10 @@ import os import re import time -import base64 import random import socket import datetime import urllib.parse -import warnings -from io import StringIO from email._parseaddr import quote from email._parseaddr import AddressList as _AddressList diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -19,7 +19,7 @@ pass from abc import get_cache_token from collections import namedtuple -from types import MappingProxyType, MethodType +from types import MappingProxyType from weakref import WeakKeyDictionary try: from _thread import RLock diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -73,7 +73,6 @@ import socket import collections from urllib.parse import urlsplit -import warnings __all__ = ["HTTPResponse", "HTTPConnection", "HTTPException", "NotConnected", "UnknownProtocol", diff --git a/Lib/http/server.py b/Lib/http/server.py --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -85,8 +85,6 @@ __all__ = ["HTTPServer", "BaseHTTPRequestHandler"] import html -import email.message -import email.parser import http.client import io import mimetypes diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -24,7 +24,7 @@ To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, socket, struct, traceback, re +import sys, logging, logging.handlers, struct, traceback, re import io try: diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -23,8 +23,7 @@ To use, simply 'import logging.handlers' and log away! """ -import errno, logging, socket, os, pickle, struct, time, re -from codecs import BOM_UTF8 +import logging, socket, os, pickle, struct, time, re from stat import ST_DEV, ST_INO, ST_MTIME import queue try: diff --git a/Lib/mailbox.py b/Lib/mailbox.py --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -6,7 +6,6 @@ # or returning from a flush() method. See functions _sync_flush() and # _sync_close(). -import sys import os import time import calendar diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -14,7 +14,6 @@ import sys import socket import struct -import errno import time import tempfile import itertools diff --git a/Lib/multiprocessing/heap.py b/Lib/multiprocessing/heap.py --- a/Lib/multiprocessing/heap.py +++ b/Lib/multiprocessing/heap.py @@ -8,13 +8,11 @@ # import bisect -import itertools import mmap import os import sys import tempfile import threading -import _multiprocessing from . import context from . import reduction diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -24,7 +24,7 @@ # If threading is available then ThreadPool should be provided. Therefore # we avoid top-level imports which are liable to fail on some systems. from . import util -from . import get_context, cpu_count, TimeoutError +from . import get_context, TimeoutError # # Constants representing the state of a pool diff --git a/Lib/multiprocessing/popen_spawn_posix.py b/Lib/multiprocessing/popen_spawn_posix.py --- a/Lib/multiprocessing/popen_spawn_posix.py +++ b/Lib/multiprocessing/popen_spawn_posix.py @@ -1,4 +1,3 @@ -import fcntl import io import os @@ -8,8 +7,6 @@ from . import spawn from . import util -from . import current_process - __all__ = ['Popen'] diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -16,7 +16,6 @@ import sys from . import context -from . import util __all__ = ['send_handle', 'recv_handle', 'ForkingPickler', 'register', 'dump'] diff --git a/Lib/multiprocessing/semaphore_tracker.py b/Lib/multiprocessing/semaphore_tracker.py --- a/Lib/multiprocessing/semaphore_tracker.py +++ b/Lib/multiprocessing/semaphore_tracker.py @@ -11,7 +11,6 @@ # python" would probably leave unlinked semaphores. # -import errno import os import signal import sys @@ -21,7 +20,6 @@ from . import spawn from . import util -from . import current_process __all__ = ['ensure_running', 'register', 'unregister'] diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -11,10 +11,8 @@ 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Condition', 'Event' ] -import os import threading import sys -import itertools import tempfile import _multiprocessing diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -7,8 +7,6 @@ # Licensed to PSF under a Contributor Agreement. # -import sys -import functools import os import itertools import weakref diff --git a/Lib/nntplib.py b/Lib/nntplib.py --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -1085,7 +1085,6 @@ # Test retrieval when run as a script. if __name__ == '__main__': import argparse - from email.utils import parsedate parser = argparse.ArgumentParser(description="""\ nntplib built-in demo - display the latest articles in a newsgroup""") diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -23,7 +23,7 @@ """ -from types import FunctionType, ModuleType +from types import FunctionType from copyreg import dispatch_table from copyreg import _extension_registry, _inverted_registry, _extension_cache from itertools import islice diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -503,7 +503,6 @@ # Import the needed APIs try: - import win32api from win32api import RegQueryValueEx, RegOpenKeyEx, \ RegCloseKey, GetVersionEx from win32con import HKEY_LOCAL_MACHINE, VER_PLATFORM_WIN32_NT, \ diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -66,7 +66,7 @@ import warnings from collections import deque from reprlib import Repr -from traceback import extract_tb, format_exception_only +from traceback import format_exception_only # --------------------------------------------------------- common routines diff --git a/Lib/runpy.py b/Lib/runpy.py --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -10,7 +10,6 @@ # to implement PEP 338 (Executing Modules as Scripts) -import os import sys import importlib.machinery # importlib first so we can test #15386 via -m import importlib.util diff --git a/Lib/socketserver.py b/Lib/socketserver.py --- a/Lib/socketserver.py +++ b/Lib/socketserver.py @@ -131,7 +131,6 @@ import socket import select -import sys import os import errno try: diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py --- a/Lib/sre_compile.py +++ b/Lib/sre_compile.py @@ -10,7 +10,7 @@ """Internal support module for sre""" -import _sre, sys +import _sre import sre_parse from sre_constants import * from _sre import MAXREPEAT diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -12,8 +12,6 @@ # XXX: show string offset and offending character for all errors -import sys - from sre_constants import * from _sre import MAXREPEAT diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -147,12 +147,9 @@ if sys.platform == "win32": from _ssl import enum_certificates, enum_crls -from socket import getnameinfo as _getnameinfo -from socket import SHUT_RDWR as _SHUT_RDWR from socket import socket, AF_INET, SOCK_STREAM, create_connection from socket import SOL_SOCKET, SO_TYPE import base64 # for DER-to-PEM translation -import traceback import errno diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -350,8 +350,6 @@ import io import os import time -import traceback -import gc import signal import builtins import warnings diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -65,8 +65,6 @@ # from tarfile import * __all__ = ["TarFile", "TarInfo", "is_tarfile", "TarError"] -from builtins import open as _open # Since 'open' is TarFile.open - #--------------------------------------------------------- # tar constants #--------------------------------------------------------- diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -3,7 +3,6 @@ import sys as _sys import _thread -from time import sleep as _sleep try: from time import monotonic as _time except ImportError: diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -35,8 +35,6 @@ # Attempt to configure Tcl/Tk without requiring PATH from tkinter import _fix -import warnings - import _tkinter # If this fails your Python may not be configured for Tk TclError = _tkinter.TclError from tkinter.constants import * diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -27,7 +27,7 @@ # from tkinter import * -from tkinter import _flatten, _cnfmerge, _default_root +from tkinter import _cnfmerge, _default_root # WARNING - TkVersion is a limited precision floating point number if TkVersion < 3.999: diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -32,7 +32,6 @@ import shutil import subprocess import sys -import sysconfig import types logger = logging.getLogger(__name__) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -134,7 +134,6 @@ import http.client import urllib.parse from xml.parsers import expat -import socket import errno from io import BytesIO try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 09:28:46 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 09:28:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2320976=3A_pyflakes=3A_Remove_u?= =?utf-8?q?nused_imports?= Message-ID: <3fqLqf4ssQz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/714002a5c1b7 changeset: 89870:714002a5c1b7 parent: 89868:52404c14be07 parent: 89869:f6f691ff27b9 user: Victor Stinner date: Thu Mar 20 09:22:39 2014 +0100 summary: (Merge 3.4) Issue #20976: pyflakes: Remove unused imports files: Lib/_strptime.py | 2 +- Lib/asynchat.py | 1 - Lib/compileall.py | 1 - Lib/ctypes/util.py | 2 +- Lib/decimal.py | 1 - Lib/difflib.py | 1 - Lib/email/_header_value_parser.py | 2 +- Lib/email/generator.py | 4 ---- Lib/email/header.py | 2 -- Lib/email/mime/text.py | 1 - Lib/email/parser.py | 2 -- Lib/email/quoprimime.py | 1 - Lib/email/utils.py | 3 --- Lib/functools.py | 2 +- Lib/http/client.py | 1 - Lib/http/server.py | 2 -- Lib/logging/config.py | 2 +- Lib/logging/handlers.py | 3 +-- Lib/mailbox.py | 1 - Lib/multiprocessing/connection.py | 1 - Lib/multiprocessing/heap.py | 2 -- Lib/multiprocessing/pool.py | 2 +- Lib/multiprocessing/popen_spawn_posix.py | 3 --- Lib/multiprocessing/reduction.py | 1 - Lib/multiprocessing/semaphore_tracker.py | 2 -- Lib/multiprocessing/synchronize.py | 2 -- Lib/multiprocessing/util.py | 2 -- Lib/nntplib.py | 1 - Lib/pickle.py | 2 +- Lib/platform.py | 1 - Lib/pydoc.py | 2 +- Lib/runpy.py | 1 - Lib/socketserver.py | 1 - Lib/sre_compile.py | 2 +- Lib/sre_parse.py | 2 -- Lib/ssl.py | 3 --- Lib/subprocess.py | 2 -- Lib/tarfile.py | 2 -- Lib/threading.py | 1 - Lib/tkinter/__init__.py | 2 -- Lib/tkinter/tix.py | 2 +- Lib/venv/__init__.py | 1 - Lib/xmlrpc/client.py | 1 - 43 files changed, 11 insertions(+), 64 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -14,7 +14,7 @@ import locale import calendar from re import compile as re_compile -from re import IGNORECASE, ASCII +from re import IGNORECASE from re import escape as re_escape from datetime import (date as datetime_date, timedelta as datetime_timedelta, diff --git a/Lib/asynchat.py b/Lib/asynchat.py --- a/Lib/asynchat.py +++ b/Lib/asynchat.py @@ -45,7 +45,6 @@ method) up to the terminator, and then control will be returned to you - by calling your self.found_terminator() method. """ -import socket import asyncore from collections import deque diff --git a/Lib/compileall.py b/Lib/compileall.py --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -12,7 +12,6 @@ """ import os import sys -import errno import importlib.util import py_compile import struct diff --git a/Lib/ctypes/util.py b/Lib/ctypes/util.py --- a/Lib/ctypes/util.py +++ b/Lib/ctypes/util.py @@ -85,7 +85,7 @@ elif os.name == "posix": # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump - import re, tempfile, errno + import re, tempfile def _findLib_gcc(name): expr = r'[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name) diff --git a/Lib/decimal.py b/Lib/decimal.py --- a/Lib/decimal.py +++ b/Lib/decimal.py @@ -142,7 +142,6 @@ # See http://speleotrove.com/decimal/ __libmpdec_version__ = "2.4.0" # compatible libmpdec version -import copy as _copy import math as _math import numbers as _numbers import sys diff --git a/Lib/difflib.py b/Lib/difflib.py --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -30,7 +30,6 @@ 'Differ','IS_CHARACTER_JUNK', 'IS_LINE_JUNK', 'context_diff', 'unified_diff', 'HtmlDiff', 'Match'] -import warnings import heapq from collections import namedtuple as _namedtuple diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -70,7 +70,7 @@ import re import urllib # For urllib.parse.unquote from string import hexdigits -from collections import namedtuple, OrderedDict +from collections import OrderedDict from email import _encoded_words as _ew from email import errors from email import utils diff --git a/Lib/email/generator.py b/Lib/email/generator.py --- a/Lib/email/generator.py +++ b/Lib/email/generator.py @@ -10,14 +10,10 @@ import sys import time import random -import warnings from copy import deepcopy from io import StringIO, BytesIO -from email._policybase import compat32 -from email.header import Header from email.utils import _has_surrogates -import email.charset as _charset UNDERSCORE = '_' NL = '\n' # XXX: no longer used by the code below. diff --git a/Lib/email/header.py b/Lib/email/header.py --- a/Lib/email/header.py +++ b/Lib/email/header.py @@ -100,7 +100,6 @@ words.append((encoded, encoding, charset)) # Now loop over words and remove words that consist of whitespace # between two encoded strings. - import sys droplist = [] for n, w in enumerate(words): if n>1 and w[1] and words[n-2][1] and words[n-1][0].isspace(): @@ -362,7 +361,6 @@ for string, charset in self._chunks: if hasspace is not None: hasspace = string and self._nonctext(string[0]) - import sys if lastcs not in (None, 'us-ascii'): if not hasspace or charset not in (None, 'us-ascii'): formatter.add_transition() diff --git a/Lib/email/mime/text.py b/Lib/email/mime/text.py --- a/Lib/email/mime/text.py +++ b/Lib/email/mime/text.py @@ -6,7 +6,6 @@ __all__ = ['MIMEText'] -from email.encoders import encode_7or8bit from email.mime.nonmultipart import MIMENonMultipart diff --git a/Lib/email/parser.py b/Lib/email/parser.py --- a/Lib/email/parser.py +++ b/Lib/email/parser.py @@ -7,11 +7,9 @@ __all__ = ['Parser', 'HeaderParser', 'BytesParser', 'BytesHeaderParser', 'FeedParser', 'BytesFeedParser'] -import warnings from io import StringIO, TextIOWrapper from email.feedparser import FeedParser, BytesFeedParser -from email.message import Message from email._policybase import compat32 diff --git a/Lib/email/quoprimime.py b/Lib/email/quoprimime.py --- a/Lib/email/quoprimime.py +++ b/Lib/email/quoprimime.py @@ -40,7 +40,6 @@ ] import re -import io from string import ascii_letters, digits, hexdigits diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -25,13 +25,10 @@ import os import re import time -import base64 import random import socket import datetime import urllib.parse -import warnings -from io import StringIO from email._parseaddr import quote from email._parseaddr import AddressList as _AddressList diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -19,7 +19,7 @@ pass from abc import get_cache_token from collections import namedtuple -from types import MappingProxyType, MethodType +from types import MappingProxyType from weakref import WeakKeyDictionary try: from _thread import RLock diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -73,7 +73,6 @@ import socket import collections from urllib.parse import urlsplit -import warnings __all__ = ["HTTPResponse", "HTTPConnection", "HTTPException", "NotConnected", "UnknownProtocol", diff --git a/Lib/http/server.py b/Lib/http/server.py --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -85,8 +85,6 @@ __all__ = ["HTTPServer", "BaseHTTPRequestHandler"] import html -import email.message -import email.parser import http.client import io import mimetypes diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -24,7 +24,7 @@ To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, socket, struct, traceback, re +import sys, logging, logging.handlers, struct, traceback, re import io try: diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -23,8 +23,7 @@ To use, simply 'import logging.handlers' and log away! """ -import errno, logging, socket, os, pickle, struct, time, re -from codecs import BOM_UTF8 +import logging, socket, os, pickle, struct, time, re from stat import ST_DEV, ST_INO, ST_MTIME import queue try: diff --git a/Lib/mailbox.py b/Lib/mailbox.py --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -6,7 +6,6 @@ # or returning from a flush() method. See functions _sync_flush() and # _sync_close(). -import sys import os import time import calendar diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -14,7 +14,6 @@ import sys import socket import struct -import errno import time import tempfile import itertools diff --git a/Lib/multiprocessing/heap.py b/Lib/multiprocessing/heap.py --- a/Lib/multiprocessing/heap.py +++ b/Lib/multiprocessing/heap.py @@ -8,13 +8,11 @@ # import bisect -import itertools import mmap import os import sys import tempfile import threading -import _multiprocessing from . import context from . import reduction diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -24,7 +24,7 @@ # If threading is available then ThreadPool should be provided. Therefore # we avoid top-level imports which are liable to fail on some systems. from . import util -from . import get_context, cpu_count, TimeoutError +from . import get_context, TimeoutError # # Constants representing the state of a pool diff --git a/Lib/multiprocessing/popen_spawn_posix.py b/Lib/multiprocessing/popen_spawn_posix.py --- a/Lib/multiprocessing/popen_spawn_posix.py +++ b/Lib/multiprocessing/popen_spawn_posix.py @@ -1,4 +1,3 @@ -import fcntl import io import os @@ -8,8 +7,6 @@ from . import spawn from . import util -from . import current_process - __all__ = ['Popen'] diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -16,7 +16,6 @@ import sys from . import context -from . import util __all__ = ['send_handle', 'recv_handle', 'ForkingPickler', 'register', 'dump'] diff --git a/Lib/multiprocessing/semaphore_tracker.py b/Lib/multiprocessing/semaphore_tracker.py --- a/Lib/multiprocessing/semaphore_tracker.py +++ b/Lib/multiprocessing/semaphore_tracker.py @@ -11,7 +11,6 @@ # python" would probably leave unlinked semaphores. # -import errno import os import signal import sys @@ -21,7 +20,6 @@ from . import spawn from . import util -from . import current_process __all__ = ['ensure_running', 'register', 'unregister'] diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -11,10 +11,8 @@ 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Condition', 'Event' ] -import os import threading import sys -import itertools import tempfile import _multiprocessing diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -7,8 +7,6 @@ # Licensed to PSF under a Contributor Agreement. # -import sys -import functools import os import itertools import weakref diff --git a/Lib/nntplib.py b/Lib/nntplib.py --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -1085,7 +1085,6 @@ # Test retrieval when run as a script. if __name__ == '__main__': import argparse - from email.utils import parsedate parser = argparse.ArgumentParser(description="""\ nntplib built-in demo - display the latest articles in a newsgroup""") diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -23,7 +23,7 @@ """ -from types import FunctionType, ModuleType +from types import FunctionType from copyreg import dispatch_table from copyreg import _extension_registry, _inverted_registry, _extension_cache from itertools import islice diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -503,7 +503,6 @@ # Import the needed APIs try: - import win32api from win32api import RegQueryValueEx, RegOpenKeyEx, \ RegCloseKey, GetVersionEx from win32con import HKEY_LOCAL_MACHINE, VER_PLATFORM_WIN32_NT, \ diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -66,7 +66,7 @@ import warnings from collections import deque from reprlib import Repr -from traceback import extract_tb, format_exception_only +from traceback import format_exception_only # --------------------------------------------------------- common routines diff --git a/Lib/runpy.py b/Lib/runpy.py --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -10,7 +10,6 @@ # to implement PEP 338 (Executing Modules as Scripts) -import os import sys import importlib.machinery # importlib first so we can test #15386 via -m import importlib.util diff --git a/Lib/socketserver.py b/Lib/socketserver.py --- a/Lib/socketserver.py +++ b/Lib/socketserver.py @@ -131,7 +131,6 @@ import socket import select -import sys import os import errno try: diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py --- a/Lib/sre_compile.py +++ b/Lib/sre_compile.py @@ -10,7 +10,7 @@ """Internal support module for sre""" -import _sre, sys +import _sre import sre_parse from sre_constants import * from _sre import MAXREPEAT diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -12,8 +12,6 @@ # XXX: show string offset and offending character for all errors -import sys - from sre_constants import * from _sre import MAXREPEAT diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -147,12 +147,9 @@ if sys.platform == "win32": from _ssl import enum_certificates, enum_crls -from socket import getnameinfo as _getnameinfo -from socket import SHUT_RDWR as _SHUT_RDWR from socket import socket, AF_INET, SOCK_STREAM, create_connection from socket import SOL_SOCKET, SO_TYPE import base64 # for DER-to-PEM translation -import traceback import errno diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -350,8 +350,6 @@ import io import os import time -import traceback -import gc import signal import builtins import warnings diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -65,8 +65,6 @@ # from tarfile import * __all__ = ["TarFile", "TarInfo", "is_tarfile", "TarError"] -from builtins import open as _open # Since 'open' is TarFile.open - #--------------------------------------------------------- # tar constants #--------------------------------------------------------- diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -3,7 +3,6 @@ import sys as _sys import _thread -from time import sleep as _sleep try: from time import monotonic as _time except ImportError: diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -35,8 +35,6 @@ # Attempt to configure Tcl/Tk without requiring PATH from tkinter import _fix -import warnings - import _tkinter # If this fails your Python may not be configured for Tk TclError = _tkinter.TclError from tkinter.constants import * diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -27,7 +27,7 @@ # from tkinter import * -from tkinter import _flatten, _cnfmerge, _default_root +from tkinter import _cnfmerge, _default_root # WARNING - TkVersion is a limited precision floating point number if TkVersion < 3.999: diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -32,7 +32,6 @@ import shutil import subprocess import sys -import sysconfig import types logger = logging.getLogger(__name__) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -134,7 +134,6 @@ import http.client import urllib.parse from xml.parsers import expat -import socket import errno from io import BytesIO try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 09:28:47 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 09:28:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXN1c2UgIzEyMzI4?= =?utf-8?q?=2C_=2320978=3A_Add_=5Fwinapi=2EWAIT=5FABANDONED=5F0_symbol=2C_?= =?utf-8?q?needed_by?= Message-ID: <3fqLqg6lvJz7Lkn@mail.python.org> http://hg.python.org/cpython/rev/bdad874195d6 changeset: 89871:bdad874195d6 branch: 3.4 parent: 89869:f6f691ff27b9 user: Victor Stinner date: Thu Mar 20 09:26:55 2014 +0100 summary: Isuse #12328, #20978: Add _winapi.WAIT_ABANDONED_0 symbol, needed by multiprocessing.connection files: Modules/_winapi.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Modules/_winapi.c b/Modules/_winapi.c --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1343,6 +1343,7 @@ WINAPI_CONSTANT(F_DWORD, STILL_ACTIVE); WINAPI_CONSTANT(F_DWORD, SW_HIDE); WINAPI_CONSTANT(F_DWORD, WAIT_OBJECT_0); + WINAPI_CONSTANT(F_DWORD, WAIT_ABANDONED_0); WINAPI_CONSTANT(F_DWORD, WAIT_TIMEOUT); WINAPI_CONSTANT("i", NULL); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 09:28:49 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 20 Mar 2014 09:28:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogKE1lcmdlIDMuNCkgSXNzdWUgIzEyMzI4LCAjMjA5Nzg6IEFkZCBfd2lu?= =?utf-8?q?api=2EWAIT=5FABANDONED=5F0_symbol=2C_needed?= Message-ID: <3fqLqj1mlrz7LlW@mail.python.org> http://hg.python.org/cpython/rev/2e4692a762d5 changeset: 89872:2e4692a762d5 parent: 89870:714002a5c1b7 parent: 89871:bdad874195d6 user: Victor Stinner date: Thu Mar 20 09:27:11 2014 +0100 summary: (Merge 3.4) Issue #12328, #20978: Add _winapi.WAIT_ABANDONED_0 symbol, needed by multiprocessing.connection files: Modules/_winapi.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Modules/_winapi.c b/Modules/_winapi.c --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1343,6 +1343,7 @@ WINAPI_CONSTANT(F_DWORD, STILL_ACTIVE); WINAPI_CONSTANT(F_DWORD, SW_HIDE); WINAPI_CONSTANT(F_DWORD, WAIT_OBJECT_0); + WINAPI_CONSTANT(F_DWORD, WAIT_ABANDONED_0); WINAPI_CONSTANT(F_DWORD, WAIT_TIMEOUT); WINAPI_CONSTANT("i", NULL); -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Mar 20 09:30:50 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 20 Mar 2014 09:30:50 +0100 Subject: [Python-checkins] Daily reference leaks (2978643766b9): sum=4 Message-ID: results for 2978643766b9 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 4] memory blocks, sum=4 test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog3jzUbF', '-x'] From python-checkins at python.org Thu Mar 20 13:44:02 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 13:44:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzEwMTQx?= =?utf-8?q?=3A_updated_new_usages_of_AF=5FCAN_to_be_in_=23ifdef_AF=5FCAN_r?= =?utf-8?q?ather_than?= Message-ID: <3fqSVB4zM2z7Ljh@mail.python.org> http://hg.python.org/cpython/rev/df427bf067d7 changeset: 89873:df427bf067d7 branch: 3.4 parent: 89871:bdad874195d6 user: Vinay Sajip date: Thu Mar 20 12:42:42 2014 +0000 summary: Issue #10141: updated new usages of AF_CAN to be in #ifdef AF_CAN rather than #ifdef HAVE_LINUX_CAN_H to allow compilation on older Linuxes. files: Modules/socketmodule.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1165,7 +1165,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { struct sockaddr_can *a = (struct sockaddr_can *)addr; @@ -1589,7 +1589,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: switch (s->sock_proto) { case CAN_RAW: @@ -1796,7 +1796,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { *len_ret = sizeof (struct sockaddr_can); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 13:44:03 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 13:44:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merged_from_3=2E4=2E?= Message-ID: <3fqSVC6lcWz7Ljc@mail.python.org> http://hg.python.org/cpython/rev/76255e78c309 changeset: 89874:76255e78c309 parent: 89872:2e4692a762d5 parent: 89873:df427bf067d7 user: Vinay Sajip date: Thu Mar 20 12:43:50 2014 +0000 summary: Merged from 3.4. files: Modules/socketmodule.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1165,7 +1165,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { struct sockaddr_can *a = (struct sockaddr_can *)addr; @@ -1589,7 +1589,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: switch (s->sock_proto) { case CAN_RAW: @@ -1796,7 +1796,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { *len_ret = sizeof (struct sockaddr_can); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:05:38 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 14:05:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNTU4?= =?utf-8?q?=3A_Improved_implementation_of_error_handling=2E?= Message-ID: <3fqSz66bDCz7LkX@mail.python.org> http://hg.python.org/cpython/rev/46fe23e000a1 changeset: 89875:46fe23e000a1 branch: 3.4 parent: 89873:df427bf067d7 user: Vinay Sajip date: Thu Mar 20 13:03:17 2014 +0000 summary: Issue #20558: Improved implementation of error handling. files: Lib/logging/config.py | 23 +++++++++++------------ 1 files changed, 11 insertions(+), 12 deletions(-) diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -19,13 +19,19 @@ is based on PEP 282 and comments thereto in comp.lang.python, and influenced by Apache's log4j system. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, struct, traceback, re +import errno import io +import logging +import logging.handlers +import re +import struct +import sys +import traceback try: import _thread as thread @@ -38,10 +44,7 @@ DEFAULT_LOGGING_CONFIG_PORT = 9030 -if sys.platform == "win32": - RESET_ERROR = 10054 #WSAECONNRESET -else: - RESET_ERROR = 104 #ECONNRESET +RESET_ERROR = errno.ECONNRESET # # The following code implements a socket listener for on-the-fly @@ -867,12 +870,8 @@ if self.server.ready: self.server.ready.set() except OSError as e: - if not isinstance(e.args, tuple): + if e.errno != RESET_ERROR: raise - else: - errcode = e.args[0] - if errcode != RESET_ERROR: - raise class ConfigSocketReceiver(ThreadingTCPServer): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:06:20 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 14:06:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2320558=3A_Improved_implementation_of_error_hand?= =?utf-8?q?ling=2E?= Message-ID: <3fqSzw12f5z7Ljt@mail.python.org> http://hg.python.org/cpython/rev/9c27c9844be2 changeset: 89876:9c27c9844be2 parent: 89874:76255e78c309 parent: 89875:46fe23e000a1 user: Vinay Sajip date: Thu Mar 20 13:06:07 2014 +0000 summary: Closes #20558: Improved implementation of error handling. files: Lib/logging/config.py | 23 +++++++++++------------ 1 files changed, 11 insertions(+), 12 deletions(-) diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -19,13 +19,19 @@ is based on PEP 282 and comments thereto in comp.lang.python, and influenced by Apache's log4j system. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, struct, traceback, re +import errno import io +import logging +import logging.handlers +import re +import struct +import sys +import traceback try: import _thread as thread @@ -38,10 +44,7 @@ DEFAULT_LOGGING_CONFIG_PORT = 9030 -if sys.platform == "win32": - RESET_ERROR = 10054 #WSAECONNRESET -else: - RESET_ERROR = 104 #ECONNRESET +RESET_ERROR = errno.ECONNRESET # # The following code implements a socket listener for on-the-fly @@ -867,12 +870,8 @@ if self.server.ready: self.server.ready.set() except OSError as e: - if not isinstance(e.args, tuple): + if e.errno != RESET_ERROR: raise - else: - errcode = e.args[0] - if errcode != RESET_ERROR: - raise class ConfigSocketReceiver(ThreadingTCPServer): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:15:36 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 14:15:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDQ0?= =?utf-8?q?=3A_Reduced_code_duplication=2E?= Message-ID: <3fqTBc1ft7z7LjY@mail.python.org> http://hg.python.org/cpython/rev/f27f594a96c3 changeset: 89877:f27f594a96c3 branch: 3.4 parent: 89875:46fe23e000a1 user: Vinay Sajip date: Thu Mar 20 13:14:39 2014 +0000 summary: Issue #20444: Reduced code duplication. files: Lib/logging/config.py | 84 ++++++++++++------------------ 1 files changed, 34 insertions(+), 50 deletions(-) diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -277,6 +277,30 @@ return True +class ConvertingMixin(object): + """For ConvertingXXX's, this mixin class provides common functions""" + + def convert_with_key(self, key, value, replace=True): + result = self.configurator.convert(value) + #If the converted value is different, save for next time + if value is not result: + if replace: + self[key] = result + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + result.key = key + return result + + def convert(self, value): + result = self.configurator.convert(value) + if value is not result: + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + return result + + # The ConvertingXXX classes are wrappers around standard Python containers, # and they serve to convert any suitable values in the container. The # conversion converts base dicts, lists and tuples to their wrapped @@ -286,77 +310,37 @@ # Each wrapper should have a configurator attribute holding the actual # configurator to use for conversion. -class ConvertingDict(dict): +class ConvertingDict(dict, ConvertingMixin): """A converting dictionary wrapper.""" def __getitem__(self, key): value = dict.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def get(self, key, default=None): value = dict.get(self, key, default) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, key, default=None): value = dict.pop(self, key, default) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value, replace=False) -class ConvertingList(list): +class ConvertingList(list, ConvertingMixin): """A converting list wrapper.""" def __getitem__(self, key): value = list.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, idx=-1): value = list.pop(self, idx) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - return result + return self.convert(value) -class ConvertingTuple(tuple): +class ConvertingTuple(tuple, ConvertingMixin): """A converting tuple wrapper.""" def __getitem__(self, key): value = tuple.__getitem__(self, key) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + # Can't replace a tuple entry. + return self.convert_with_key(key, value, replace=False) class BaseConfigurator(object): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:15:37 2014 From: python-checkins at python.org (vinay.sajip) Date: Thu, 20 Mar 2014 14:15:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2320444=3A_Merged_fix_from_3=2E4=2E?= Message-ID: <3fqTBd5t42z7Ljk@mail.python.org> http://hg.python.org/cpython/rev/135aa1c4349d changeset: 89878:135aa1c4349d parent: 89876:9c27c9844be2 parent: 89877:f27f594a96c3 user: Vinay Sajip date: Thu Mar 20 13:15:23 2014 +0000 summary: Closes #20444: Merged fix from 3.4. files: Lib/logging/config.py | 84 ++++++++++++------------------ 1 files changed, 34 insertions(+), 50 deletions(-) diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -277,6 +277,30 @@ return True +class ConvertingMixin(object): + """For ConvertingXXX's, this mixin class provides common functions""" + + def convert_with_key(self, key, value, replace=True): + result = self.configurator.convert(value) + #If the converted value is different, save for next time + if value is not result: + if replace: + self[key] = result + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + result.key = key + return result + + def convert(self, value): + result = self.configurator.convert(value) + if value is not result: + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + return result + + # The ConvertingXXX classes are wrappers around standard Python containers, # and they serve to convert any suitable values in the container. The # conversion converts base dicts, lists and tuples to their wrapped @@ -286,77 +310,37 @@ # Each wrapper should have a configurator attribute holding the actual # configurator to use for conversion. -class ConvertingDict(dict): +class ConvertingDict(dict, ConvertingMixin): """A converting dictionary wrapper.""" def __getitem__(self, key): value = dict.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def get(self, key, default=None): value = dict.get(self, key, default) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, key, default=None): value = dict.pop(self, key, default) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value, replace=False) -class ConvertingList(list): +class ConvertingList(list, ConvertingMixin): """A converting list wrapper.""" def __getitem__(self, key): value = list.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, idx=-1): value = list.pop(self, idx) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - return result + return self.convert(value) -class ConvertingTuple(tuple): +class ConvertingTuple(tuple, ConvertingMixin): """A converting tuple wrapper.""" def __getitem__(self, key): value = tuple.__getitem__(self, key) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + # Can't replace a tuple entry. + return self.convert_with_key(key, value, replace=False) class BaseConfigurator(object): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:32:00 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 14:32:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Issue_=2320993=3A_Fix_lin?= =?utf-8?q?k_to_PEPs?= Message-ID: <3fqTYX5bLVz7Lkd@mail.python.org> http://hg.python.org/devguide/rev/0d9f428f28ce changeset: 682:0d9f428f28ce user: Zachary Ware date: Thu Mar 20 08:31:49 2014 -0500 summary: Issue #20993: Fix link to PEPs files: index.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -218,7 +218,7 @@ .. _Buildbot status: http://python.org/dev/buildbot/ .. _Firefox search engine plug-in: http://www.python.org/dev/searchplugin/ .. _Misc directory: http://hg.python.org/cpython/file/default/Misc -.. _PEPs: http://www.python.org/dev/peps +.. _PEPs: http://www.python.org/dev/peps/ .. _python.org maintenance: http://python.org/dev/pydotorg/ .. _Python: http://www.python.org/ .. _Python Mentors: http://pythonmentors.com/ -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Thu Mar 20 14:44:43 2014 From: python-checkins at python.org (raymond.hettinger) Date: Thu, 20 Mar 2014 14:44:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Clean-up_docst?= =?utf-8?q?ring?= Message-ID: <3fqTrC4Zh5z7LjR@mail.python.org> http://hg.python.org/cpython/rev/3cdb5ceb30d1 changeset: 89879:3cdb5ceb30d1 branch: 3.4 parent: 89877:f27f594a96c3 user: Raymond Hettinger date: Thu Mar 20 06:42:31 2014 -0700 summary: Clean-up docstring files: Lib/collections/__init__.py | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -274,9 +274,7 @@ return OrderedDict(zip(self._fields, self)) def _asdict(self): - '''Return a new OrderedDict which maps field names to their values. - This method is obsolete. Use vars(nt) or nt.__dict__ instead. - ''' + 'Return a new OrderedDict which maps field names to their values.' return self.__dict__ def __getnewargs__(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 14:44:44 2014 From: python-checkins at python.org (raymond.hettinger) Date: Thu, 20 Mar 2014 14:44:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3fqTrD71t0z7Ljp@mail.python.org> http://hg.python.org/cpython/rev/2470f5d4e269 changeset: 89880:2470f5d4e269 parent: 89878:135aa1c4349d parent: 89879:3cdb5ceb30d1 user: Raymond Hettinger date: Thu Mar 20 06:44:33 2014 -0700 summary: merge files: Lib/collections/__init__.py | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -274,9 +274,7 @@ return OrderedDict(zip(self._fields, self)) def _asdict(self): - '''Return a new OrderedDict which maps field names to their values. - This method is obsolete. Use vars(nt) or nt.__dict__ instead. - ''' + 'Return a new OrderedDict which maps field names to their values.' return self.__dict__ def __getnewargs__(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:03 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_parameter_?= =?utf-8?q?name_in_docs_for_os=2Emakedirs_and_os=2Eremovedirs=2E?= Message-ID: <3fqZ9b0svpz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/8d53d3f61b0c changeset: 89881:8d53d3f61b0c branch: 3.4 parent: 89879:3cdb5ceb30d1 user: Zachary Ware date: Thu Mar 20 09:42:01 2014 -0500 summary: Fix parameter name in docs for os.makedirs and os.removedirs. Pointed out by Colin Davis on docs at . files: Doc/library/os.rst | 4 ++-- Lib/os.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1607,7 +1607,7 @@ The *dir_fd* argument. -.. function:: makedirs(path, mode=0o777, exist_ok=False) +.. function:: makedirs(name, mode=0o777, exist_ok=False) .. index:: single: directory; creating @@ -1763,7 +1763,7 @@ The *dir_fd* argument. -.. function:: removedirs(path) +.. function:: removedirs(name) .. index:: single: directory; deleting diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -216,7 +216,7 @@ # (Inspired by Eric Raymond; the doc strings are mostly his) def makedirs(name, mode=0o777, exist_ok=False): - """makedirs(path [, mode=0o777][, exist_ok=False]) + """makedirs(name [, mode=0o777][, exist_ok=False]) Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not @@ -260,7 +260,7 @@ raise def removedirs(name): - """removedirs(path) + """removedirs(name) Super-rmdir; remove a leaf directory and all empty intermediate ones. Works like rmdir except that, if the leaf directory is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:04 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_missing_pa?= =?utf-8?q?renthesis=2E__Found_by_cocoatomo_on_docs=40=2E?= Message-ID: <3fqZ9c3Bcvz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/0a656d469284 changeset: 89882:0a656d469284 branch: 3.4 user: Zachary Ware date: Thu Mar 20 09:46:09 2014 -0500 summary: Add missing parenthesis. Found by cocoatomo on docs at . files: Doc/distutils/apiref.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -993,7 +993,7 @@ Files in *src* that begin with :file:`.nfs` are skipped (more information on these files is available in answer D2 of the `NFS FAQ page - `_. + `_). .. versionchanged:: 3.3.1 NFS files are ignored. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:05 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_spelling_i?= =?utf-8?q?n_enum_docs=2E?= Message-ID: <3fqZ9d69QJz7Lk9@mail.python.org> http://hg.python.org/cpython/rev/9d0d0de3161d changeset: 89883:9d0d0de3161d branch: 3.4 user: Zachary Ware date: Thu Mar 20 10:01:48 2014 -0500 summary: Fix spelling in enum docs. "equivalant" was caught by Tobias K?s on docs@, "seperated" and "chartruese" were discovered by a spell-checker. files: Doc/library/enum.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -442,7 +442,7 @@ :value: What the new Enum class will record as its name. -:names: The Enum members. This can be a whitespace or comma seperated string +:names: The Enum members. This can be a whitespace or comma separated string (values will start at 1):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -453,7 +453,7 @@ or a mapping:: - {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + {'chartreuse': 7, 'sea_green': 11, 'rosemary': 42} :module: name of module where new Enum class can be found. @@ -543,7 +543,7 @@ add methods and don't specify another data type such as :class:`int` or :class:`str`. 3. When another data type is mixed in, the :attr:`value` attribute is *not the - same* as the enum member itself, although it is equivalant and will compare + same* as the enum member itself, although it is equivalent and will compare equal. 4. %-style formatting: `%s` and `%r` call :class:`Enum`'s :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:07 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_typos_in_D?= =?utf-8?q?oc/faq/extending=2E__Found_by_cocoatomo_on_docs=40=2E?= Message-ID: <3fqZ9g181Zz7LkB@mail.python.org> http://hg.python.org/cpython/rev/3afa9116859c changeset: 89884:3afa9116859c branch: 3.4 user: Zachary Ware date: Thu Mar 20 10:16:09 2014 -0500 summary: Fix typos in Doc/faq/extending. Found by cocoatomo on docs at . files: Doc/faq/extending.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -95,8 +95,8 @@ There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls -like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well -as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et. +like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc. as well +as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et al.) and mappings in the PyMapping APIs. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:08 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogVXNlICJ4IiBpbnN0?= =?utf-8?q?ead_of_=22i=22_for_s=2Eindex_and_s=2Ecount_entries_in_sequence_?= =?utf-8?q?ops_table=2E?= Message-ID: <3fqZ9h32F4z7Ljg@mail.python.org> http://hg.python.org/cpython/rev/babb9479b79f changeset: 89885:babb9479b79f branch: 2.7 parent: 89865:7860a42bcf98 user: Zachary Ware date: Thu Mar 20 11:00:29 2014 -0500 summary: Use "x" instead of "i" for s.index and s.count entries in sequence ops table. Suggested by Bruce Esrig on docs at . files: Doc/library/stdtypes.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -754,11 +754,11 @@ +------------------+--------------------------------+----------+ | ``max(s)`` | largest item of *s* | | +------------------+--------------------------------+----------+ -| ``s.index(i)`` | index of the first occurrence | | -| | of *i* in *s* | | +| ``s.index(x)`` | index of the first occurrence | | +| | of *x* in *s* | | +------------------+--------------------------------+----------+ -| ``s.count(i)`` | total number of occurrences of | | -| | *i* in *s* | | +| ``s.count(x)`` | total number of occurrences of | | +| | *x* in *s* | | +------------------+--------------------------------+----------+ Sequence types also support comparisons. In particular, tuples and lists -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:00:09 2014 From: python-checkins at python.org (zach.ware) Date: Thu, 20 Mar 2014 18:00:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_several_minor_doc_fixes_from_3=2E4?= Message-ID: <3fqZ9j6ZtCz7LkY@mail.python.org> http://hg.python.org/cpython/rev/a7bf938bc1e3 changeset: 89886:a7bf938bc1e3 parent: 89880:2470f5d4e269 parent: 89884:3afa9116859c user: Zachary Ware date: Thu Mar 20 11:28:16 2014 -0500 summary: Merge several minor doc fixes from 3.4 files: Doc/distutils/apiref.rst | 2 +- Doc/faq/extending.rst | 4 ++-- Doc/library/enum.rst | 6 +++--- Doc/library/os.rst | 4 ++-- Lib/os.py | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -993,7 +993,7 @@ Files in *src* that begin with :file:`.nfs` are skipped (more information on these files is available in answer D2 of the `NFS FAQ page - `_. + `_). .. versionchanged:: 3.3.1 NFS files are ignored. diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -95,8 +95,8 @@ There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls -like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well -as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et. +like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc. as well +as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et al.) and mappings in the PyMapping APIs. diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -442,7 +442,7 @@ :value: What the new Enum class will record as its name. -:names: The Enum members. This can be a whitespace or comma seperated string +:names: The Enum members. This can be a whitespace or comma separated string (values will start at 1):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -453,7 +453,7 @@ or a mapping:: - {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + {'chartreuse': 7, 'sea_green': 11, 'rosemary': 42} :module: name of module where new Enum class can be found. @@ -543,7 +543,7 @@ add methods and don't specify another data type such as :class:`int` or :class:`str`. 3. When another data type is mixed in, the :attr:`value` attribute is *not the - same* as the enum member itself, although it is equivalant and will compare + same* as the enum member itself, although it is equivalent and will compare equal. 4. %-style formatting: `%s` and `%r` call :class:`Enum`'s :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1607,7 +1607,7 @@ The *dir_fd* argument. -.. function:: makedirs(path, mode=0o777, exist_ok=False) +.. function:: makedirs(name, mode=0o777, exist_ok=False) .. index:: single: directory; creating @@ -1763,7 +1763,7 @@ The *dir_fd* argument. -.. function:: removedirs(path) +.. function:: removedirs(name) .. index:: single: directory; deleting diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -216,7 +216,7 @@ # (Inspired by Eric Raymond; the doc strings are mostly his) def makedirs(name, mode=0o777, exist_ok=False): - """makedirs(path [, mode=0o777][, exist_ok=False]) + """makedirs(name [, mode=0o777][, exist_ok=False]) Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not @@ -260,7 +260,7 @@ raise def removedirs(name): - """removedirs(path) + """removedirs(name) Super-rmdir; remove a leaf directory and all empty intermediate ones. Works like rmdir except that, if the leaf directory is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:40:25 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 18:40:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_improve_start_?= =?utf-8?q?default_for_relpath?= Message-ID: <3fqb494tBXz7LlL@mail.python.org> http://hg.python.org/cpython/rev/9e0626766ac0 changeset: 89887:9e0626766ac0 branch: 3.4 parent: 89884:3afa9116859c user: Benjamin Peterson date: Thu Mar 20 12:39:53 2014 -0500 summary: improve start default for relpath files: Doc/library/os.path.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -242,7 +242,7 @@ links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path, start=None) +.. function:: relpath(path, start=os.curdir) Return a relative filepath to *path* either from the current directory or from an optional *start* directory. This is a path computation: the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 18:40:26 2014 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 20 Mar 2014 18:40:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fqb4B6GjLz7LkV@mail.python.org> http://hg.python.org/cpython/rev/2346d5290b96 changeset: 89888:2346d5290b96 parent: 89886:a7bf938bc1e3 parent: 89887:9e0626766ac0 user: Benjamin Peterson date: Thu Mar 20 12:40:03 2014 -0500 summary: merge 3.4 files: Doc/library/os.path.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -242,7 +242,7 @@ links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path, start=None) +.. function:: relpath(path, start=os.curdir) Return a relative filepath to *path* either from the current directory or from an optional *start* directory. This is a path computation: the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 21:11:33 2014 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 20 Mar 2014 21:11:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2320744=3A_don=27t_try_ru?= =?utf-8?q?nning_an_external_=27zip=27_in_shutil=2Emake=5Farchive=28=29?= Message-ID: <3fqfQY0djbz7LlW@mail.python.org> http://hg.python.org/cpython/rev/681e20f8b717 changeset: 89889:681e20f8b717 user: Andrew Kuchling date: Thu Mar 20 16:11:16 2014 -0400 summary: #20744: don't try running an external 'zip' in shutil.make_archive() Instead we'll just use the stdlib zipfile module. Patch by Derek Chiang files: Lib/shutil.py | 55 ++++++++++---------------------------- 1 files changed, 15 insertions(+), 40 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -630,23 +630,6 @@ return archive_name -def _call_external_zip(base_dir, zip_filename, verbose=False, dry_run=False): - # XXX see if we want to keep an external call here - if verbose: - zipoptions = "-r" - else: - zipoptions = "-rq" - from distutils.errors import DistutilsExecError - from distutils.spawn import spawn - try: - spawn(["zip", zipoptions, zip_filename, base_dir], dry_run=dry_run) - except DistutilsExecError: - # XXX really should distinguish between "couldn't find - # external 'zip' command" and "zip failed". - raise ExecError("unable to create zip file '%s': " - "could neither import the 'zipfile' module nor " - "find a standalone zip utility") % zip_filename - def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None): """Create a zip file from all the files under 'base_dir'. @@ -656,6 +639,8 @@ available, raises ExecError. Returns the name of the output zip file. """ + import zipfile + zip_filename = base_name + ".zip" archive_dir = os.path.dirname(base_name) @@ -665,30 +650,20 @@ if not dry_run: os.makedirs(archive_dir) - # If zipfile module is not available, try spawning an external 'zip' - # command. - try: - import zipfile - except ImportError: - zipfile = None + if logger is not None: + logger.info("creating '%s' and adding '%s' to it", + zip_filename, base_dir) - if zipfile is None: - _call_external_zip(base_dir, zip_filename, verbose, dry_run) - else: - if logger is not None: - logger.info("creating '%s' and adding '%s' to it", - zip_filename, base_dir) - - if not dry_run: - with zipfile.ZipFile(zip_filename, "w", - compression=zipfile.ZIP_DEFLATED) as zf: - for dirpath, dirnames, filenames in os.walk(base_dir): - for name in filenames: - path = os.path.normpath(os.path.join(dirpath, name)) - if os.path.isfile(path): - zf.write(path, path) - if logger is not None: - logger.info("adding '%s'", path) + if not dry_run: + with zipfile.ZipFile(zip_filename, "w", + compression=zipfile.ZIP_DEFLATED) as zf: + for dirpath, dirnames, filenames in os.walk(base_dir): + for name in filenames: + path = os.path.normpath(os.path.join(dirpath, name)) + if os.path.isfile(path): + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) return zip_filename -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 21:54:40 2014 From: python-checkins at python.org (giampaolo.rodola) Date: Thu, 20 Mar 2014 21:54:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_issue_18931=3A_selecto?= =?utf-8?q?rs_module_now_supports_/dev/poll_on_Solaris=2E?= Message-ID: <3fqgNJ6kfwz7Ll5@mail.python.org> http://hg.python.org/cpython/rev/0a51a516bc70 changeset: 89890:0a51a516bc70 parent: 89872:2e4692a762d5 user: Giampaolo Rodola' date: Thu Mar 20 21:43:41 2014 +0100 summary: Fix issue 18931: selectors module now supports /dev/poll on Solaris. files: Doc/library/selectors.rst | 11 ++++ Lib/selectors.py | 62 +++++++++++++++++++++++++- Lib/test/test_selectors.py | 10 +++- Misc/NEWS | 3 + 4 files changed, 84 insertions(+), 2 deletions(-) diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -45,6 +45,7 @@ +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector @@ -207,6 +208,16 @@ This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + +.. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -441,6 +441,64 @@ super().close() +if hasattr(select, 'devpoll'): + + class DevpollSelector(_BaseSelectorImpl): + """Solaris /dev/poll selector.""" + + def __init__(self): + super().__init__() + self._devpoll = select.devpoll() + + def fileno(self): + return self._devpoll.fileno() + + def register(self, fileobj, events, data=None): + key = super().register(fileobj, events, data) + poll_events = 0 + if events & EVENT_READ: + poll_events |= select.POLLIN + if events & EVENT_WRITE: + poll_events |= select.POLLOUT + self._devpoll.register(key.fd, poll_events) + return key + + def unregister(self, fileobj): + key = super().unregister(fileobj) + self._devpoll.unregister(key.fd) + return key + + def select(self, timeout=None): + if timeout is None: + timeout = None + elif timeout <= 0: + timeout = 0 + else: + # devpoll() has a resolution of 1 millisecond, round away from + # zero to wait *at least* timeout seconds. + timeout = math.ceil(timeout * 1e3) + ready = [] + try: + fd_event_list = self._devpoll.poll(timeout) + except InterruptedError: + return ready + for fd, event in fd_event_list: + events = 0 + if event & ~select.POLLIN: + events |= EVENT_WRITE + if event & ~select.POLLOUT: + events |= EVENT_READ + + key = self._key_from_fd(fd) + if key: + ready.append((key, events & key.events)) + return ready + + def close(self): + self._devpoll.close() + super().close() + + if hasattr(select, 'kqueue'): class KqueueSelector(_BaseSelectorImpl): @@ -513,12 +571,14 @@ super().close() -# Choose the best implementation: roughly, epoll|kqueue > poll > select. +# Choose the best implementation: roughly, epoll|kqueue|devpoll > poll > select. # select() also can't accept a FD > FD_SETSIZE (usually around 1024) if 'KqueueSelector' in globals(): DefaultSelector = KqueueSelector elif 'EpollSelector' in globals(): DefaultSelector = EpollSelector +elif 'DevpollSelector' in globals(): + DefaultSelector = DevpollSelector elif 'PollSelector' in globals(): DefaultSelector = PollSelector else: diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -441,10 +441,18 @@ SELECTOR = getattr(selectors, 'KqueueSelector', None) + at unittest.skipUnless(hasattr(selectors, 'DevpollSelector'), + "Test needs selectors.DevpollSelector") +class DevpollSelectorTestCase(BaseSelectorTestCase, ScalableSelectorMixIn): + + SELECTOR = getattr(selectors, 'DevpollSelector', None) + + + def test_main(): tests = [DefaultSelectorTestCase, SelectSelectorTestCase, PollSelectorTestCase, EpollSelectorTestCase, - KqueueSelectorTestCase] + KqueueSelectorTestCase, DevpollSelectorTestCase] support.run_unittest(*tests) support.reap_children() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue 18931: selectors module now supports /dev/poll on Solaris. + Patch by Giampaolo Rodola'. + - Issue #19977: When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the ``surrogateescape`` error handler, instead of the ``strict`` error handler. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 20 21:54:42 2014 From: python-checkins at python.org (giampaolo.rodola) Date: Thu, 20 Mar 2014 21:54:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge_heads?= Message-ID: <3fqgNL2Xwnz7Llx@mail.python.org> http://hg.python.org/cpython/rev/cbb18e801505 changeset: 89891:cbb18e801505 parent: 89890:0a51a516bc70 parent: 89889:681e20f8b717 user: Giampaolo Rodola' date: Thu Mar 20 21:44:53 2014 +0100 summary: merge heads files: Doc/distutils/apiref.rst | 2 +- Doc/faq/extending.rst | 4 +- Doc/library/enum.rst | 6 +- Doc/library/os.path.rst | 2 +- Doc/library/os.rst | 4 +- Lib/collections/__init__.py | 4 +- Lib/logging/config.py | 107 ++++++++++------------- Lib/os.py | 4 +- Lib/shutil.py | 55 +++-------- Modules/socketmodule.c | 6 +- 10 files changed, 75 insertions(+), 119 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -993,7 +993,7 @@ Files in *src* that begin with :file:`.nfs` are skipped (more information on these files is available in answer D2 of the `NFS FAQ page - `_. + `_). .. versionchanged:: 3.3.1 NFS files are ignored. diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -95,8 +95,8 @@ There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls -like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well -as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et. +like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc. as well +as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et al.) and mappings in the PyMapping APIs. diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -442,7 +442,7 @@ :value: What the new Enum class will record as its name. -:names: The Enum members. This can be a whitespace or comma seperated string +:names: The Enum members. This can be a whitespace or comma separated string (values will start at 1):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -453,7 +453,7 @@ or a mapping:: - {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + {'chartreuse': 7, 'sea_green': 11, 'rosemary': 42} :module: name of module where new Enum class can be found. @@ -543,7 +543,7 @@ add methods and don't specify another data type such as :class:`int` or :class:`str`. 3. When another data type is mixed in, the :attr:`value` attribute is *not the - same* as the enum member itself, although it is equivalant and will compare + same* as the enum member itself, although it is equivalent and will compare equal. 4. %-style formatting: `%s` and `%r` call :class:`Enum`'s :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -242,7 +242,7 @@ links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path, start=None) +.. function:: relpath(path, start=os.curdir) Return a relative filepath to *path* either from the current directory or from an optional *start* directory. This is a path computation: the diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1607,7 +1607,7 @@ The *dir_fd* argument. -.. function:: makedirs(path, mode=0o777, exist_ok=False) +.. function:: makedirs(name, mode=0o777, exist_ok=False) .. index:: single: directory; creating @@ -1763,7 +1763,7 @@ The *dir_fd* argument. -.. function:: removedirs(path) +.. function:: removedirs(name) .. index:: single: directory; deleting diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -274,9 +274,7 @@ return OrderedDict(zip(self._fields, self)) def _asdict(self): - '''Return a new OrderedDict which maps field names to their values. - This method is obsolete. Use vars(nt) or nt.__dict__ instead. - ''' + 'Return a new OrderedDict which maps field names to their values.' return self.__dict__ def __getnewargs__(self): diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -19,13 +19,19 @@ is based on PEP 282 and comments thereto in comp.lang.python, and influenced by Apache's log4j system. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, struct, traceback, re +import errno import io +import logging +import logging.handlers +import re +import struct +import sys +import traceback try: import _thread as thread @@ -38,10 +44,7 @@ DEFAULT_LOGGING_CONFIG_PORT = 9030 -if sys.platform == "win32": - RESET_ERROR = 10054 #WSAECONNRESET -else: - RESET_ERROR = 104 #ECONNRESET +RESET_ERROR = errno.ECONNRESET # # The following code implements a socket listener for on-the-fly @@ -274,6 +277,30 @@ return True +class ConvertingMixin(object): + """For ConvertingXXX's, this mixin class provides common functions""" + + def convert_with_key(self, key, value, replace=True): + result = self.configurator.convert(value) + #If the converted value is different, save for next time + if value is not result: + if replace: + self[key] = result + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + result.key = key + return result + + def convert(self, value): + result = self.configurator.convert(value) + if value is not result: + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + return result + + # The ConvertingXXX classes are wrappers around standard Python containers, # and they serve to convert any suitable values in the container. The # conversion converts base dicts, lists and tuples to their wrapped @@ -283,77 +310,37 @@ # Each wrapper should have a configurator attribute holding the actual # configurator to use for conversion. -class ConvertingDict(dict): +class ConvertingDict(dict, ConvertingMixin): """A converting dictionary wrapper.""" def __getitem__(self, key): value = dict.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def get(self, key, default=None): value = dict.get(self, key, default) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, key, default=None): value = dict.pop(self, key, default) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value, replace=False) -class ConvertingList(list): +class ConvertingList(list, ConvertingMixin): """A converting list wrapper.""" def __getitem__(self, key): value = list.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, idx=-1): value = list.pop(self, idx) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - return result + return self.convert(value) -class ConvertingTuple(tuple): +class ConvertingTuple(tuple, ConvertingMixin): """A converting tuple wrapper.""" def __getitem__(self, key): value = tuple.__getitem__(self, key) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + # Can't replace a tuple entry. + return self.convert_with_key(key, value, replace=False) class BaseConfigurator(object): """ @@ -867,12 +854,8 @@ if self.server.ready: self.server.ready.set() except OSError as e: - if not isinstance(e.args, tuple): + if e.errno != RESET_ERROR: raise - else: - errcode = e.args[0] - if errcode != RESET_ERROR: - raise class ConfigSocketReceiver(ThreadingTCPServer): """ diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -216,7 +216,7 @@ # (Inspired by Eric Raymond; the doc strings are mostly his) def makedirs(name, mode=0o777, exist_ok=False): - """makedirs(path [, mode=0o777][, exist_ok=False]) + """makedirs(name [, mode=0o777][, exist_ok=False]) Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not @@ -260,7 +260,7 @@ raise def removedirs(name): - """removedirs(path) + """removedirs(name) Super-rmdir; remove a leaf directory and all empty intermediate ones. Works like rmdir except that, if the leaf directory is diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -630,23 +630,6 @@ return archive_name -def _call_external_zip(base_dir, zip_filename, verbose=False, dry_run=False): - # XXX see if we want to keep an external call here - if verbose: - zipoptions = "-r" - else: - zipoptions = "-rq" - from distutils.errors import DistutilsExecError - from distutils.spawn import spawn - try: - spawn(["zip", zipoptions, zip_filename, base_dir], dry_run=dry_run) - except DistutilsExecError: - # XXX really should distinguish between "couldn't find - # external 'zip' command" and "zip failed". - raise ExecError("unable to create zip file '%s': " - "could neither import the 'zipfile' module nor " - "find a standalone zip utility") % zip_filename - def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None): """Create a zip file from all the files under 'base_dir'. @@ -656,6 +639,8 @@ available, raises ExecError. Returns the name of the output zip file. """ + import zipfile + zip_filename = base_name + ".zip" archive_dir = os.path.dirname(base_name) @@ -665,30 +650,20 @@ if not dry_run: os.makedirs(archive_dir) - # If zipfile module is not available, try spawning an external 'zip' - # command. - try: - import zipfile - except ImportError: - zipfile = None + if logger is not None: + logger.info("creating '%s' and adding '%s' to it", + zip_filename, base_dir) - if zipfile is None: - _call_external_zip(base_dir, zip_filename, verbose, dry_run) - else: - if logger is not None: - logger.info("creating '%s' and adding '%s' to it", - zip_filename, base_dir) - - if not dry_run: - with zipfile.ZipFile(zip_filename, "w", - compression=zipfile.ZIP_DEFLATED) as zf: - for dirpath, dirnames, filenames in os.walk(base_dir): - for name in filenames: - path = os.path.normpath(os.path.join(dirpath, name)) - if os.path.isfile(path): - zf.write(path, path) - if logger is not None: - logger.info("adding '%s'", path) + if not dry_run: + with zipfile.ZipFile(zip_filename, "w", + compression=zipfile.ZIP_DEFLATED) as zf: + for dirpath, dirnames, filenames in os.walk(base_dir): + for name in filenames: + path = os.path.normpath(os.path.join(dirpath, name)) + if os.path.isfile(path): + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) return zip_filename diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1165,7 +1165,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { struct sockaddr_can *a = (struct sockaddr_can *)addr; @@ -1589,7 +1589,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: switch (s->sock_proto) { case CAN_RAW: @@ -1796,7 +1796,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { *len_ret = sizeof (struct sockaddr_can); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 00:02:52 2014 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 21 Mar 2014 00:02:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_remove_the_ability_of_date?= =?utf-8?q?time=2Etime_to_be_considered_false_=28closes_=2313936=29?= Message-ID: <3fqkDD0JpLz7Lls@mail.python.org> http://hg.python.org/cpython/rev/89aa669dcc61 changeset: 89892:89aa669dcc61 user: Benjamin Peterson date: Thu Mar 20 18:00:35 2014 -0500 summary: remove the ability of datetime.time to be considered false (closes #13936) files: Doc/library/datetime.rst | 11 ++++-- Doc/whatsnew/3.5.rst | 5 ++- Lib/datetime.py | 6 ---- Lib/test/datetimetester.py | 21 +++---------- Misc/NEWS | 3 ++ Modules/_datetimemodule.c | 38 +------------------------- 6 files changed, 20 insertions(+), 64 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -1378,10 +1378,13 @@ * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. - +In boolean contexts, a :class:`.time` object is always considered to be true. + +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -186,4 +186,7 @@ This section lists previously described changes and other bugfixes that may require changes to your code. -* Nothing yet. +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. diff --git a/Lib/datetime.py b/Lib/datetime.py --- a/Lib/datetime.py +++ b/Lib/datetime.py @@ -1249,12 +1249,6 @@ _check_tzinfo_arg(tzinfo) return time(hour, minute, second, microsecond, tzinfo) - def __bool__(self): - if self.second or self.microsecond: - return True - offset = self.utcoffset() or timedelta(0) - return timedelta(hours=self.hour, minutes=self.minute) != offset - # Pickle support. def _getstate(self): diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -2270,13 +2270,14 @@ self.assertEqual(orig, derived) def test_bool(self): + # time is always True. cls = self.theclass self.assertTrue(cls(1)) self.assertTrue(cls(0, 1)) self.assertTrue(cls(0, 0, 1)) self.assertTrue(cls(0, 0, 0, 1)) - self.assertFalse(cls(0)) - self.assertFalse(cls()) + self.assertTrue(cls(0)) + self.assertTrue(cls()) def test_replace(self): cls = self.theclass @@ -2629,7 +2630,7 @@ self.assertEqual(derived.tzname(), 'cookie') def test_more_bool(self): - # Test cases with non-None tzinfo. + # time is always True. cls = self.theclass t = cls(0, tzinfo=FixedOffset(-300, "")) @@ -2639,23 +2640,11 @@ self.assertTrue(t) t = cls(5, tzinfo=FixedOffset(300, "")) - self.assertFalse(t) + self.assertTrue(t) t = cls(23, 59, tzinfo=FixedOffset(23*60 + 59, "")) - self.assertFalse(t) - - # Mostly ensuring this doesn't overflow internally. - t = cls(0, tzinfo=FixedOffset(23*60 + 59, "")) self.assertTrue(t) - # But this should yield a value error -- the utcoffset is bogus. - t = cls(0, tzinfo=FixedOffset(24*60, "")) - self.assertRaises(ValueError, lambda: bool(t)) - - # Likewise. - t = cls(0, tzinfo=FixedOffset(-24*60, "")) - self.assertRaises(ValueError, lambda: bool(t)) - def test_replace(self): cls = self.theclass z100 = FixedOffset(100, "+100") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #13936: Remove the ability of datetime.time instances to be considered + false in boolean contexts. + - Issue 18931: selectors module now supports /dev/poll on Solaris. Patch by Giampaolo Rodola'. diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -3805,29 +3805,6 @@ return clone; } -static int -time_bool(PyObject *self) -{ - PyObject *offset, *tzinfo; - int offsecs = 0; - - if (TIME_GET_SECOND(self) || TIME_GET_MICROSECOND(self)) { - /* Since utcoffset is in whole minutes, nothing can - * alter the conclusion that this is nonzero. - */ - return 1; - } - tzinfo = GET_TIME_TZINFO(self); - if (tzinfo != Py_None) { - offset = call_utcoffset(tzinfo, Py_None); - if (offset == NULL) - return -1; - offsecs = GET_TD_DAYS(offset)*86400 + GET_TD_SECONDS(offset); - Py_DECREF(offset); - } - return (TIME_GET_MINUTE(self)*60 - offsecs + TIME_GET_HOUR(self)*3600) != 0; -} - /* Pickle support, a simple use of __reduce__. */ /* Let basestate be the non-tzinfo data string. @@ -3895,19 +3872,6 @@ All arguments are optional. tzinfo may be None, or an instance of\n\ a tzinfo subclass. The remaining arguments may be ints.\n"); -static PyNumberMethods time_as_number = { - 0, /* nb_add */ - 0, /* nb_subtract */ - 0, /* nb_multiply */ - 0, /* nb_remainder */ - 0, /* nb_divmod */ - 0, /* nb_power */ - 0, /* nb_negative */ - 0, /* nb_positive */ - 0, /* nb_absolute */ - (inquiry)time_bool, /* nb_bool */ -}; - static PyTypeObject PyDateTime_TimeType = { PyVarObject_HEAD_INIT(NULL, 0) "datetime.time", /* tp_name */ @@ -3919,7 +3883,7 @@ 0, /* tp_setattr */ 0, /* tp_reserved */ (reprfunc)time_repr, /* tp_repr */ - &time_as_number, /* tp_as_number */ + 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)time_hash, /* tp_hash */ -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Mar 21 09:48:51 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 21 Mar 2014 09:48:51 +0100 Subject: [Python-checkins] Daily reference leaks (89aa669dcc61): sum=-4 Message-ID: results for 89aa669dcc61 on branch "default" -------------------------------------------- test_site leaked [0, 0, -2] references, sum=-2 test_site leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogBYDE8g', '-x'] From python-checkins at python.org Fri Mar 21 10:01:43 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 10:01:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Ensure_call=5Fs?= =?utf-8?b?b29uKCksIGNhbGxfbGF0ZXIoKSBhbmQgY2FsbF9hdCgpIGFyZSBpbnZva2Vk?= =?utf-8?q?_on_current?= Message-ID: <3fqzWC2mXlz7LjM@mail.python.org> http://hg.python.org/cpython/rev/30977e66a882 changeset: 89893:30977e66a882 user: Victor Stinner date: Fri Mar 21 10:00:52 2014 +0100 summary: asyncio: Ensure call_soon(), call_later() and call_at() are invoked on current loop in debug mode. Raise a RuntimeError if the event loop of the current thread is different. The check should help to debug thread-safetly issue. Patch written by David Foster. files: Lib/asyncio/base_events.py | 23 ++++++++++- Lib/test/test_asyncio/test_base_events.py | 23 +++++++++++ 2 files changed, 45 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -259,6 +259,8 @@ """Like call_later(), but uses an absolute time.""" if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_at()") + if self._debug: + self._assert_is_current_event_loop() timer = events.TimerHandle(when, callback, args, self) heapq.heappush(self._scheduled, timer) return timer @@ -273,15 +275,34 @@ Any positional arguments after the callback will be passed to the callback when it is called. """ + return self._call_soon(callback, args, check_loop=True) + + def _call_soon(self, callback, args, check_loop): if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_soon()") + if self._debug and check_loop: + self._assert_is_current_event_loop() handle = events.Handle(callback, args, self) self._ready.append(handle) return handle + def _assert_is_current_event_loop(self): + """Asserts that this event loop is the current event loop. + + Non-threadsafe methods of this class make this assumption and will + likely behave incorrectly when the assumption is violated. + + Should only be called when (self._debug == True). The caller is + responsible for checking this condition for performance reasons. + """ + if events.get_event_loop() is not self: + raise RuntimeError( + "non-threadsafe operation invoked on an event loop other " + "than the current one") + def call_soon_threadsafe(self, callback, *args): """XXX""" - handle = self.call_soon(callback, *args) + handle = self._call_soon(callback, args, check_loop=False) self._write_to_self() return handle diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -136,6 +136,29 @@ # are really slow self.assertLessEqual(dt, 0.9, dt) + def test_assert_is_current_event_loop(self): + def cb(): + pass + + other_loop = base_events.BaseEventLoop() + other_loop._selector = unittest.mock.Mock() + asyncio.set_event_loop(other_loop) + + # raise RuntimeError if the event loop is different in debug mode + self.loop.set_debug(True) + with self.assertRaises(RuntimeError): + self.loop.call_soon(cb) + with self.assertRaises(RuntimeError): + self.loop.call_later(60, cb) + with self.assertRaises(RuntimeError): + self.loop.call_at(self.loop.time() + 60, cb) + + # check disabled if debug mode is disabled + self.loop.set_debug(False) + self.loop.call_soon(cb) + self.loop.call_later(60, cb) + self.loop.call_at(self.loop.time() + 60, cb) + def test_run_once_in_executor_handle(self): def cb(): pass -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 11:46:36 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 11:46:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2321005=3A_Fix_docu?= =?utf-8?q?mentation_of_asyncio=2Esubprocess=2EDEVNULL?= Message-ID: <3fr1rD62Rcz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/70c77ff64df1 changeset: 89894:70c77ff64df1 user: Victor Stinner date: Fri Mar 21 11:44:49 2014 +0100 summary: Close #21005: Fix documentation of asyncio.subprocess.DEVNULL files: Doc/library/asyncio-subprocess.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -46,10 +46,9 @@ .. data:: asyncio.subprocess.DEVNULL - Special value that can be used as the *stderr* argument to - :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and - indicates that standard error should go into the same handle as standard - output. + Special value that can be used as the *stdin*, *stdout* or *stderr* argument + to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and + indicates that the special file :data:`os.devnull` will be used. Process -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 11:56:50 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 11:56:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321006=3A_Fix_subp?= =?utf-8?q?rocess_example_on_Windows_in_asyncio_doc?= Message-ID: <3fr2421DPkz7LjS@mail.python.org> http://hg.python.org/cpython/rev/7cca663a72eb changeset: 89895:7cca663a72eb user: Victor Stinner date: Fri Mar 21 11:56:40 2014 +0100 summary: Issue #21006: Fix subprocess example on Windows in asyncio doc files: Doc/library/asyncio-subprocess.rst | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -146,6 +146,7 @@ display the output:: import asyncio + import os import sys from asyncio import subprocess @@ -164,7 +165,11 @@ exitcode = yield from proc.wait() return (exitcode, stdout) - loop = asyncio.get_event_loop() + if os.name == 'nt': + loop = asyncio.ProactorEventLoop() + asyncio.set_event_loop(loop) + else: + loop = asyncio.get_event_loop() coro = getstatusoutput(sys.executable, '-m', 'platform') exitcode, stdout = loop.run_until_complete(coro) if not exitcode: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 12:45:41 2014 From: python-checkins at python.org (vinay.sajip) Date: Fri, 21 Mar 2014 12:45:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzEwMTQx?= =?utf-8?q?=2C_Issue_20065=3A_Changed_=23if_to_take_CAN=5FRAW_into_account?= =?utf-8?q?=2E?= Message-ID: <3fr38P5XL8z7Ljq@mail.python.org> http://hg.python.org/cpython/rev/9dc199b921eb changeset: 89896:9dc199b921eb branch: 3.4 parent: 89887:9e0626766ac0 user: Vinay Sajip date: Fri Mar 21 11:44:32 2014 +0000 summary: Issue #10141, Issue 20065: Changed #if to take CAN_RAW into account. files: Modules/socketmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1589,7 +1589,7 @@ } #endif -#ifdef AF_CAN +#if defined(AF_CAN) && defined(CAN_RAW) && defined(CAN_BCM) case AF_CAN: switch (s->sock_proto) { case CAN_RAW: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 12:45:43 2014 From: python-checkins at python.org (vinay.sajip) Date: Fri, 21 Mar 2014 12:45:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2310141=2C_Issue_20065=3A_Merged_from_3=2E4=2E?= Message-ID: <3fr38R0CCJz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/20cced06acdd changeset: 89897:20cced06acdd parent: 89895:7cca663a72eb parent: 89896:9dc199b921eb user: Vinay Sajip date: Fri Mar 21 11:45:27 2014 +0000 summary: Issue #10141, Issue 20065: Merged from 3.4. files: Modules/socketmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1589,7 +1589,7 @@ } #endif -#ifdef AF_CAN +#if defined(AF_CAN) && defined(CAN_RAW) && defined(CAN_BCM) case AF_CAN: switch (s->sock_proto) { case CAN_RAW: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 14:39:07 2014 From: python-checkins at python.org (ethan.furman) Date: Fri, 21 Mar 2014 14:39:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue19995=3A_more_informa?= =?utf-8?q?tive_error_message=3B_spelling_corrections=3B_use?= Message-ID: <3fr5gH5nDTz7LjR@mail.python.org> http://hg.python.org/cpython/rev/e266525c9294 changeset: 89898:e266525c9294 user: Ethan Furman date: Fri Mar 21 06:38:46 2014 -0700 summary: Issue19995: more informative error message; spelling corrections; use operator.mod instead of __mod__ files: Lib/test/test_unicode.py | 19 ++++++++++--------- Objects/unicodeobject.c | 25 ++++++++++++++++++++----- 2 files changed, 30 insertions(+), 14 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -8,6 +8,7 @@ import _string import codecs import itertools +import operator import struct import sys import unittest @@ -1127,20 +1128,20 @@ self.assertEqual('%.2s' % "a\xe9\u20ac", 'a\xe9') #issue 19995 - class PsuedoInt: + class PseudoInt: def __init__(self, value): self.value = int(value) def __int__(self): return self.value def __index__(self): return self.value - class PsuedoFloat: + class PseudoFloat: def __init__(self, value): self.value = float(value) def __int__(self): return int(self.value) - pi = PsuedoFloat(3.1415) - letter_m = PsuedoInt(109) + pi = PseudoFloat(3.1415) + letter_m = PseudoInt(109) self.assertEqual('%x' % 42, '2a') self.assertEqual('%X' % 15, 'F') self.assertEqual('%o' % 9, '11') @@ -1149,11 +1150,11 @@ self.assertEqual('%X' % letter_m, '6D') self.assertEqual('%o' % letter_m, '155') self.assertEqual('%c' % letter_m, 'm') - self.assertRaises(TypeError, '%x'.__mod__, pi), - self.assertRaises(TypeError, '%x'.__mod__, 3.14), - self.assertRaises(TypeError, '%X'.__mod__, 2.11), - self.assertRaises(TypeError, '%o'.__mod__, 1.79), - self.assertRaises(TypeError, '%c'.__mod__, pi), + self.assertRaisesRegex(TypeError, '%x format: an integer is required, not float', operator.mod, '%x', 3.14), + self.assertRaisesRegex(TypeError, '%X format: an integer is required, not float', operator.mod, '%X', 2.11), + self.assertRaisesRegex(TypeError, '%o format: an integer is required, not float', operator.mod, '%o', 1.79), + self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PseudoFloat', operator.mod, '%x', pi), + self.assertRaises(TypeError, operator.mod, '%c', pi), def test_formatting_with_enum(self): # issue18780 diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13986,11 +13986,13 @@ if (!PyNumber_Check(v)) goto wrongtype; - /* make sure number is a type of integer */ + /* make sure number is a type of integer for o, x, and X */ if (!PyLong_Check(v)) { if (type == 'o' || type == 'x' || type == 'X') { iobj = PyNumber_Index(v); if (iobj == NULL) { + if (PyErr_ExceptionMatches(PyExc_TypeError)) + goto wrongtype; return -1; } } @@ -14052,10 +14054,23 @@ return 0; wrongtype: - PyErr_Format(PyExc_TypeError, - "%%%c format: a number is required, " - "not %.200s", - type, Py_TYPE(v)->tp_name); + switch(type) + { + case 'o': + case 'x': + case 'X': + PyErr_Format(PyExc_TypeError, + "%%%c format: an integer is required, " + "not %.200s", + type, Py_TYPE(v)->tp_name); + break; + default: + PyErr_Format(PyExc_TypeError, + "%%%c format: a number is required, " + "not %.200s", + type, Py_TYPE(v)->tp_name); + break; + } return -1; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 15:52:20 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 15:52:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319165=3A_The_form?= =?utf-8?q?atter_module_graduates_to_full_deprecation=2E?= Message-ID: <3fr7Hm0c9kz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/455e5385752a changeset: 89899:455e5385752a user: Brett Cannon date: Fri Mar 21 10:52:33 2014 -0400 summary: Issue #19165: The formatter module graduates to full deprecation. files: Doc/whatsnew/3.5.rst | 3 ++- Lib/formatter.py | 2 +- Misc/NEWS | 3 +++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -165,7 +165,8 @@ Deprecated Python modules, functions and methods ------------------------------------------------ -* None yet. +* The :mod:`formatter` module has now graduated to full deprecation and is still + slated for removal in Python 3.6. Deprecated functions and types of the C API diff --git a/Lib/formatter.py b/Lib/formatter.py --- a/Lib/formatter.py +++ b/Lib/formatter.py @@ -21,7 +21,7 @@ import sys import warnings warnings.warn('the formatter module is deprecated and will be removed in ' - 'Python 3.6', PendingDeprecationWarning) + 'Python 3.6', DeprecationWarning) AS_IS = None diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #19165: The formatter module now raises DeprecationWarning instead of + PendingDeprecationWarning. + - Issue #13936: Remove the ability of datetime.time instances to be considered false in boolean contexts. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 16:02:53 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 16:02:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwODg0?= =?utf-8?b?OiBEb24ndCBhc3N1bWUgaW4gaW1wb3J0bGliLl9faW5pdF9fIHRoYXQgX19m?= =?utf-8?q?ile=5F=5F_is?= Message-ID: <3fr7Wx33jzz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/b5b81a3eb6e6 changeset: 89900:b5b81a3eb6e6 branch: 3.4 parent: 89871:bdad874195d6 user: Brett Cannon date: Fri Mar 21 10:58:33 2014 -0400 summary: Issue #20884: Don't assume in importlib.__init__ that __file__ is defined. files: Lib/importlib/__init__.py | 7 ++++++- Misc/NEWS | 2 ++ 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py --- a/Lib/importlib/__init__.py +++ b/Lib/importlib/__init__.py @@ -22,7 +22,12 @@ # a second copy of the module. _bootstrap.__name__ = 'importlib._bootstrap' _bootstrap.__package__ = 'importlib' - _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') + try: + _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') + except NameError: + # __file__ is not guaranteed to be defined, e.g. if this code gets + # frozen by a tool like cx_Freeze. + pass sys.modules['importlib._bootstrap'] = _bootstrap # To simplify imports in test code diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,8 @@ Library ------- +- Issue #20884: Don't assume that __file__ is defined on importlib.__init__. + - Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the initialization of the unquote_to_bytes() table of the urllib.parse module, to -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 16:02:54 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 16:02:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_for_issue_=2320884?= Message-ID: <3fr7Wy5YXBz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/42ae7b2524a2 changeset: 89901:42ae7b2524a2 parent: 89899:455e5385752a parent: 89900:b5b81a3eb6e6 user: Brett Cannon date: Fri Mar 21 11:01:02 2014 -0400 summary: Merge for issue #20884 files: Lib/importlib/__init__.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py --- a/Lib/importlib/__init__.py +++ b/Lib/importlib/__init__.py @@ -22,7 +22,12 @@ # a second copy of the module. _bootstrap.__name__ = 'importlib._bootstrap' _bootstrap.__package__ = 'importlib' - _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') + try: + _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') + except NameError: + # __file__ is not guaranteed to be defined, e.g. if this code gets + # frozen by a tool like cx_Freeze. + pass sys.modules['importlib._bootstrap'] = _bootstrap # To simplify imports in test code -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 16:02:56 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 16:02:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy40IC0+IDMuNCk6?= =?utf-8?q?_merge?= Message-ID: <3fr7X01nNQz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/09fd93f7ce02 changeset: 89902:09fd93f7ce02 branch: 3.4 parent: 89900:b5b81a3eb6e6 parent: 89896:9dc199b921eb user: Brett Cannon date: Fri Mar 21 11:02:10 2014 -0400 summary: merge files: Doc/distutils/apiref.rst | 2 +- Doc/faq/extending.rst | 4 +- Doc/library/enum.rst | 6 +- Doc/library/os.path.rst | 2 +- Doc/library/os.rst | 4 +- Lib/collections/__init__.py | 4 +- Lib/logging/config.py | 107 ++++++++++------------- Lib/os.py | 4 +- Modules/socketmodule.c | 6 +- 9 files changed, 60 insertions(+), 79 deletions(-) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -993,7 +993,7 @@ Files in *src* that begin with :file:`.nfs` are skipped (more information on these files is available in answer D2 of the `NFS FAQ page - `_. + `_). .. versionchanged:: 3.3.1 NFS files are ignored. diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -95,8 +95,8 @@ There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls -like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well -as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et. +like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc. as well +as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et al.) and mappings in the PyMapping APIs. diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -442,7 +442,7 @@ :value: What the new Enum class will record as its name. -:names: The Enum members. This can be a whitespace or comma seperated string +:names: The Enum members. This can be a whitespace or comma separated string (values will start at 1):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -453,7 +453,7 @@ or a mapping:: - {'chartruese': 7, 'sea_green': 11, 'rosemary': 42} + {'chartreuse': 7, 'sea_green': 11, 'rosemary': 42} :module: name of module where new Enum class can be found. @@ -543,7 +543,7 @@ add methods and don't specify another data type such as :class:`int` or :class:`str`. 3. When another data type is mixed in, the :attr:`value` attribute is *not the - same* as the enum member itself, although it is equivalant and will compare + same* as the enum member itself, although it is equivalent and will compare equal. 4. %-style formatting: `%s` and `%r` call :class:`Enum`'s :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -242,7 +242,7 @@ links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path, start=None) +.. function:: relpath(path, start=os.curdir) Return a relative filepath to *path* either from the current directory or from an optional *start* directory. This is a path computation: the diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1607,7 +1607,7 @@ The *dir_fd* argument. -.. function:: makedirs(path, mode=0o777, exist_ok=False) +.. function:: makedirs(name, mode=0o777, exist_ok=False) .. index:: single: directory; creating @@ -1763,7 +1763,7 @@ The *dir_fd* argument. -.. function:: removedirs(path) +.. function:: removedirs(name) .. index:: single: directory; deleting diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -274,9 +274,7 @@ return OrderedDict(zip(self._fields, self)) def _asdict(self): - '''Return a new OrderedDict which maps field names to their values. - This method is obsolete. Use vars(nt) or nt.__dict__ instead. - ''' + 'Return a new OrderedDict which maps field names to their values.' return self.__dict__ def __getnewargs__(self): diff --git a/Lib/logging/config.py b/Lib/logging/config.py --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -1,4 +1,4 @@ -# Copyright 2001-2013 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -19,13 +19,19 @@ is based on PEP 282 and comments thereto in comp.lang.python, and influenced by Apache's log4j system. -Copyright (C) 2001-2013 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ -import sys, logging, logging.handlers, struct, traceback, re +import errno import io +import logging +import logging.handlers +import re +import struct +import sys +import traceback try: import _thread as thread @@ -38,10 +44,7 @@ DEFAULT_LOGGING_CONFIG_PORT = 9030 -if sys.platform == "win32": - RESET_ERROR = 10054 #WSAECONNRESET -else: - RESET_ERROR = 104 #ECONNRESET +RESET_ERROR = errno.ECONNRESET # # The following code implements a socket listener for on-the-fly @@ -274,6 +277,30 @@ return True +class ConvertingMixin(object): + """For ConvertingXXX's, this mixin class provides common functions""" + + def convert_with_key(self, key, value, replace=True): + result = self.configurator.convert(value) + #If the converted value is different, save for next time + if value is not result: + if replace: + self[key] = result + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + result.key = key + return result + + def convert(self, value): + result = self.configurator.convert(value) + if value is not result: + if type(result) in (ConvertingDict, ConvertingList, + ConvertingTuple): + result.parent = self + return result + + # The ConvertingXXX classes are wrappers around standard Python containers, # and they serve to convert any suitable values in the container. The # conversion converts base dicts, lists and tuples to their wrapped @@ -283,77 +310,37 @@ # Each wrapper should have a configurator attribute holding the actual # configurator to use for conversion. -class ConvertingDict(dict): +class ConvertingDict(dict, ConvertingMixin): """A converting dictionary wrapper.""" def __getitem__(self, key): value = dict.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def get(self, key, default=None): value = dict.get(self, key, default) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, key, default=None): value = dict.pop(self, key, default) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value, replace=False) -class ConvertingList(list): +class ConvertingList(list, ConvertingMixin): """A converting list wrapper.""" def __getitem__(self, key): value = list.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + return self.convert_with_key(key, value) def pop(self, idx=-1): value = list.pop(self, idx) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - return result + return self.convert(value) -class ConvertingTuple(tuple): +class ConvertingTuple(tuple, ConvertingMixin): """A converting tuple wrapper.""" def __getitem__(self, key): value = tuple.__getitem__(self, key) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result + # Can't replace a tuple entry. + return self.convert_with_key(key, value, replace=False) class BaseConfigurator(object): """ @@ -867,12 +854,8 @@ if self.server.ready: self.server.ready.set() except OSError as e: - if not isinstance(e.args, tuple): + if e.errno != RESET_ERROR: raise - else: - errcode = e.args[0] - if errcode != RESET_ERROR: - raise class ConfigSocketReceiver(ThreadingTCPServer): """ diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -216,7 +216,7 @@ # (Inspired by Eric Raymond; the doc strings are mostly his) def makedirs(name, mode=0o777, exist_ok=False): - """makedirs(path [, mode=0o777][, exist_ok=False]) + """makedirs(name [, mode=0o777][, exist_ok=False]) Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not @@ -260,7 +260,7 @@ raise def removedirs(name): - """removedirs(path) + """removedirs(name) Super-rmdir; remove a leaf directory and all empty intermediate ones. Works like rmdir except that, if the leaf directory is diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1165,7 +1165,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { struct sockaddr_can *a = (struct sockaddr_can *)addr; @@ -1589,7 +1589,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#if defined(AF_CAN) && defined(CAN_RAW) && defined(CAN_BCM) case AF_CAN: switch (s->sock_proto) { case CAN_RAW: @@ -1796,7 +1796,7 @@ } #endif -#ifdef HAVE_LINUX_CAN_H +#ifdef AF_CAN case AF_CAN: { *len_ret = sizeof (struct sockaddr_can); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 16:02:57 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 16:02:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3fr7X13MS8z7Ljx@mail.python.org> http://hg.python.org/cpython/rev/0a9fbb472a27 changeset: 89903:0a9fbb472a27 parent: 89901:42ae7b2524a2 parent: 89902:09fd93f7ce02 user: Brett Cannon date: Fri Mar 21 11:03:07 2014 -0400 summary: merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 16:24:28 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 16:24:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320627=3A_xmlrpc?= =?utf-8?q?=2Eclient=2EServerProxy_is_now_a_context_manager=2E?= Message-ID: <3fr80r4mkTz7LjM@mail.python.org> http://hg.python.org/cpython/rev/395904f70d6a changeset: 89904:395904f70d6a user: Brett Cannon date: Fri Mar 21 11:24:40 2014 -0400 summary: Issue #20627: xmlrpc.client.ServerProxy is now a context manager. Patch by Claudiu Popa. files: Doc/library/xmlrpc.client.rst | 23 ++++++++++++++--------- Doc/whatsnew/3.5.rst | 3 ++- Lib/test/test_xmlrpc.py | 18 ++++++++++++++++++ Lib/xmlrpc/client.py | 6 ++++++ Misc/NEWS | 2 ++ 5 files changed, 42 insertions(+), 10 deletions(-) diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -191,6 +191,11 @@ no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: @@ -208,9 +213,9 @@ import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: @@ -518,14 +523,14 @@ from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) To access an XML-RPC server through a proxy, you need to define a custom transport. The following example shows how: diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -134,7 +134,8 @@ Improved Modules ================ -* None yet. +* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager` + (contributed by Claudiu Popa in :issue:`20627`). Optimizations diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -713,6 +713,23 @@ conn.request('POST', '/RPC2 HTTP/1.0\r\nContent-Length: 100\r\n\r\nbye') conn.close() + def test_context_manager(self): + with xmlrpclib.ServerProxy(URL) as server: + server.add(2, 3) + self.assertNotEqual(server('transport')._connection, + (None, None)) + self.assertEqual(server('transport')._connection, + (None, None)) + + def test_context_manager_method_error(self): + try: + with xmlrpclib.ServerProxy(URL) as server: + server.add(2, "a") + except xmlrpclib.Fault: + pass + self.assertEqual(server('transport')._connection, + (None, None)) + class MultiPathServerTestCase(BaseServerTestCase): threadFunc = staticmethod(http_multi_server) @@ -898,6 +915,7 @@ p = xmlrpclib.ServerProxy(self.url, transport=t) self.assertEqual(p('transport'), t) + # This is a contrived way to make a failure occur on the server side # in order to test the _send_traceback_header flag on the server class FailingMessageClass(http.client.HTTPMessage): diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -1449,6 +1449,12 @@ return self.__transport raise AttributeError("Attribute %r not found" % (attr,)) + def __enter__(self): + return self + + def __exit__(self, *args): + self.__close() + # compatibility Server = ServerProxy diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #20627: xmlrpc.client.ServerProxy is now a context manager. + - Issue #19165: The formatter module now raises DeprecationWarning instead of PendingDeprecationWarning. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 17:08:01 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 21 Mar 2014 17:08:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devinabox=3A_Issue_=2321008=3A_Update?= =?utf-8?q?_for_Python_3=2E5?= Message-ID: <3fr8z50kHHz7LjY@mail.python.org> http://hg.python.org/devinabox/rev/1eeb96fe98f1 changeset: 60:1eeb96fe98f1 user: Brett Cannon date: Fri Mar 21 12:08:18 2014 -0400 summary: Issue #21008: Update for Python 3.5 files: .hgignore | 2 - README | 70 +++++++++++++++++++----------------------- 2 files changed, 31 insertions(+), 41 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -9,8 +9,6 @@ # Downloads coverage-* mercurial-* -setuptools-* -pip-* tortoisehg-* # Generated diff --git a/README b/README --- a/README +++ b/README @@ -17,7 +17,7 @@ The following sections outline various files to download and repositories to clone into your devinabox. Make sure to **NOT** change the directories that repositories are cloned into. The default names are assumed by the other files -in the devinabox. +in devinabox. When you are done you should have in this directory everything someone needs to contribute. Simply copy the whole directory to some sort of @@ -26,7 +26,7 @@ get updates, sparing the probably taxed internet connection at the sprint from doing complete repository cloning. -If recreating from an old checkout ``hg purge --all`` in the individual +If recreating from an old checkout, ``hg purge --all`` in the individual clones is a handy way to ensure old build artifacts have been removed. You will need to enable the purge extension in ``~/.hgrc``. @@ -43,7 +43,7 @@ (http://pypi.python.org/pypi/Mercurial) and TortoiseHg for Windows users (http://tortoisehg.bitbucket.org/download/). OS X users can be told that Mercurial is available through Homebrew if they prefer -(XXX; if they use MacPorts or any other package manager then tell them they +(if they use MacPorts or any other package manager then tell them they should switch to Homebrew at home as it handles Python the best and to use the download of Mercurial you have provided to save time). @@ -60,11 +60,12 @@ OS X users should be told to download XCode from the Apple App Store **ahead of time**. It's on the order of a couple GiB in size, so you don't want to have -people downloading it at the sprint. +people downloading it at the sprint. After installation they should also make +sure to install the command-line tools (e.g. in Mavericks, +``xcode-select --install``). If new contributors think they may be doing C development, suggest LLVM + clang -for better error reporting than gcc. OS X users can get an up-to-date version -of the toolchain through Homebrew. +for better error reporting than gcc. For Windows users, tell them to download and install Visual C++ Express (http://www.microsoft.com/express/Downloads/) **ahead of time**. @@ -76,12 +77,19 @@ Clone the `CPython repository`_ and build it (you will be cleaning up your build later, though as a final step). -Also make sure to build the documentation. This not only alleviates the need for -everyone to build it from scratch, but it will also pull in copies of projects -that users can rely on, if necessary, to build other documentation -(e.g. everything needed to build the devguide). If the documentation starts -being built by a repository build of CPython (and thus Python 3) this may no -longer hold true. +Also make sure to build the documentation. This alleviates the need for +everyone to build it from scratch. To build the documentation, create a venv +with sphinx installed and point the Doc Makefile at the Python linked to in the +venv. + +All of this can be done by doing:: + + # Assuming at the top of the devinabox. + python build_cpython.py + ./cpython/python -m venv venv + ./venv/bin/pip install sphinx + cd cpython/Doc + make html PYTHON=../../venv/bin/python .. _CPython repository: http://hg.python.org/cpython @@ -89,7 +97,8 @@ PEPs ---- -Clone the `PEP repository`_ and build it. That way if people need to reference a +Clone the `PEP repository`_ and build it (use the venv you created to build the +CPython docs if necessary). That way if people need to reference a PEP they can easily find itand will be able to use the easier-to-read HTML version. @@ -104,36 +113,29 @@ Devguide -------- -Clone the `devguide repository`_ and build it. This gives people a local copy to +Clone the `devguide repository`_ and build it (again, use the venv created to +build the CPython docs if necessary). This gives people a local copy to use rather than having to use the (probably slow) internet connection at the sprint. -If a new contributor needs to be able to build the devguide, they should only -need to set their ``PYTHONPATH`` to point at the ``cpython/Doc/tools`` directory -in the CPython repository thanks to the requisite projects being pulled in when -you built the CPython documentation. - .. _devguide repository: http://hg.python.org/devguide Coverage.py ----------- -#. Download setuptools_ -#. Download coverage_ +#. Download coverage_ (need a special file that is not part of the normal + distribution of coverage, so can't just use pip) #. Build CPython: ``./build_cpython.py`` -#. Create an venv: ``./cpython/python.exe -m venv venv`` -#. Extract setuptools and coverage: ``tar -x -f setuptools-*.tar.gz; tar -x -f coverage-*.tar.gz`` -#. Install setuptools in the venv: ``../venv/bin/python3 setup.py install`` -#. Install coverage in the venv -#. Set PYTHONPATH to ``fullcoverage`` (need to change your directory): ``export PYTHONPATH=../coverage-N.N/coverage/fullcoverage`` +#. Create an venv: ``./cpython/python -m venv venv`` +#. Extract coverage: ``tar -x -f coverage-*.tar.gz`` +#. Install coverage in the venv: ``./venv/bin/python coverage-*/setup.py install`` +#. Set PYTHONPATH to ``fullcoverage`` (need to change your directory to the venv): + ``export PYTHONPATH=../coverage-N.N/coverage/fullcoverage`` #. ``unset CPPFLAGS`` in order to avoid using system Python header files #. Run coverage from the venv: ``./bin/python -m coverage run --pylib -m test`` #. Unset PYTHONPATH: ``unset PYTHONPATH`` #. Generate coverage report: ``./bin/python -m coverage html --directory=../coverage_report -i --include="../cpython/Lib/*" --title="CPython test coverage report"`` -#. Delete project directories (but not tar files!) for coverage and setuptools -#. Delete venv -#. Clean up the cpython clone: either ``make distclean`` or check it out again Do be aware that this step takes a few **hours**. If you find report generation is the bottleneck you can try using PyPy3 or your installed Python 3 interpreter @@ -165,13 +167,3 @@ While the devguide includes instructions on how to build under UNIX, the script just simplifies this by having a single command subsume both the configure and build steps. It also uses reasonable defaults (e.g. all cores on the CPU). - - -``full_coverage.py`` ---------------------- - -As discussed earlier, this generates the coverage report for the standard -library in the most thorough way possible. The ``run`` option can take specific -tests as an argument. The ``html`` directory can take an argument for a -directory to write to, but the default should not conflict with the original -coverage run you did earlier (if you followed the directions =) . -- Repository URL: http://hg.python.org/devinabox From python-checkins at python.org Fri Mar 21 17:16:22 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 17:16:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIxMDA1?= =?utf-8?q?=3A_Fix_documentation_of_asyncio=2Esubprocess=2EDEVNULL?= Message-ID: <3fr98k4FbRz7Ljc@mail.python.org> http://hg.python.org/cpython/rev/230510d0cb92 changeset: 89905:230510d0cb92 branch: 3.4 parent: 89902:09fd93f7ce02 user: Victor Stinner date: Fri Mar 21 11:44:49 2014 +0100 summary: Close #21005: Fix documentation of asyncio.subprocess.DEVNULL files: Doc/library/asyncio-subprocess.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -46,10 +46,9 @@ .. data:: asyncio.subprocess.DEVNULL - Special value that can be used as the *stderr* argument to - :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and - indicates that standard error should go into the same handle as standard - output. + Special value that can be used as the *stdin*, *stdout* or *stderr* argument + to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and + indicates that the special file :data:`os.devnull` will be used. Process -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 17:16:23 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 17:16:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDA2?= =?utf-8?q?=3A_Fix_subprocess_example_on_Windows_in_asyncio_doc?= Message-ID: <3fr98l6cDJz7LjY@mail.python.org> http://hg.python.org/cpython/rev/ab0aa412fca2 changeset: 89906:ab0aa412fca2 branch: 3.4 user: Victor Stinner date: Fri Mar 21 11:56:40 2014 +0100 summary: Issue #21006: Fix subprocess example on Windows in asyncio doc files: Doc/library/asyncio-subprocess.rst | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -146,6 +146,7 @@ display the output:: import asyncio + import os import sys from asyncio import subprocess @@ -164,7 +165,11 @@ exitcode = yield from proc.wait() return (exitcode, stdout) - loop = asyncio.get_event_loop() + if os.name == 'nt': + loop = asyncio.ProactorEventLoop() + asyncio.set_event_loop(loop) + else: + loop = asyncio.get_event_loop() coro = getstatusoutput(sys.executable, '-m', 'platform') exitcode, stdout = loop.run_until_complete(coro) if not exitcode: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 17:16:25 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 17:16:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_=28I_transplated_2_commits_from_default_to_3?= =?utf-8?b?LjQp?= Message-ID: <3fr98n1CM8z7Ljm@mail.python.org> http://hg.python.org/cpython/rev/44ccce3b4ca1 changeset: 89907:44ccce3b4ca1 parent: 89904:395904f70d6a parent: 89906:ab0aa412fca2 user: Victor Stinner date: Fri Mar 21 17:16:10 2014 +0100 summary: Null merge (I transplated 2 commits from default to 3.4) files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 17:17:40 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 17:17:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIxMDEw?= =?utf-8?q?=3A_Fix_typo_in_asyncio_doc=2E_Patch_written_by_Claudiu_Popa=2E?= Message-ID: <3fr9BD0Lgjz7LjR@mail.python.org> http://hg.python.org/cpython/rev/ef4f460e3c9e changeset: 89908:ef4f460e3c9e branch: 3.4 parent: 89906:ab0aa412fca2 user: Victor Stinner date: Fri Mar 21 17:17:15 2014 +0100 summary: Close #21010: Fix typo in asyncio doc. Patch written by Claudiu Popa. files: Doc/library/asyncio-protocol.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -89,7 +89,7 @@ .. method:: pause_reading() Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until meth:`resume_reading` + the protocol's :meth:`data_received` method until :meth:`resume_reading` is called. .. method:: resume_reading() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 21 17:17:41 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 21 Mar 2014 17:17:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Close_=2321010=3A_Fix_typo_in_asyncio_?= =?utf-8?q?doc=2E_Patch_written_by_Claudiu?= Message-ID: <3fr9BF31zMz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/7761b3dff2e8 changeset: 89909:7761b3dff2e8 parent: 89907:44ccce3b4ca1 parent: 89908:ef4f460e3c9e user: Victor Stinner date: Fri Mar 21 17:17:28 2014 +0100 summary: (Merge 3.4) Close #21010: Fix typo in asyncio doc. Patch written by Claudiu Popa. files: Doc/library/asyncio-protocol.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -89,7 +89,7 @@ .. method:: pause_reading() Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until meth:`resume_reading` + the protocol's :meth:`data_received` method until :meth:`resume_reading` is called. .. method:: resume_reading() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 02:34:39 2014 From: python-checkins at python.org (donald.stufft) Date: Sat, 22 Mar 2014 02:34:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTk1?= =?utf-8?q?=3A_Enhance_default_ciphers_used_by_the_ssl_module?= Message-ID: <3frPXv30Jmz7LjV@mail.python.org> http://hg.python.org/cpython/rev/e9749a7aa958 changeset: 89910:e9749a7aa958 branch: 3.4 parent: 89908:ef4f460e3c9e user: Donald Stufft date: Fri Mar 21 21:33:34 2014 -0400 summary: Issue #20995: Enhance default ciphers used by the ssl module Closes #20995 by Enabling better security by prioritizing ciphers such that: * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) * Prefer ECDHE over DHE for better performance * Prefer any AES-GCM over any AES-CBC for better performance and security * Then Use HIGH cipher suites as a fallback * Then Use 3DES as fallback which is secure but slow * Finally use RC4 as a fallback which is problematic but needed for compatibility some times. * Disable NULL authentication, NULL encryption, and MD5 MACs for security reasons files: Doc/library/ssl.rst | 15 +++--------- Lib/ssl.py | 37 ++++++++++++++++++++++++++------ Misc/NEWS | 3 ++ 3 files changed, 37 insertions(+), 18 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1665,17 +1665,10 @@ enabled when negotiating a SSL session is possible through the :meth:`SSLContext.set_ciphers` method. Starting from Python 3.2.3, the ssl module disables certain weak ciphers by default, but you may want -to further restrict the cipher choice. For example:: - - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) - context.set_ciphers('HIGH:!aNULL:!eNULL') - -The ``!aNULL:!eNULL`` part of the cipher spec is necessary to disable ciphers -which don't provide both encryption and authentication. Be sure to read -OpenSSL's documentation about the `cipher list -format `_. -If you want to check which ciphers are enabled by a given cipher list, -use the ``openssl ciphers`` command on your system. +to further restrict the cipher choice. Be sure to read OpenSSL's documentation +about the `cipher list format `_. +If you want to check which ciphers are enabled by a given cipher list, use the +``openssl ciphers`` command on your system. Multi-processing ^^^^^^^^^^^^^^^^ diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -162,14 +162,37 @@ # Disable weak or insecure ciphers by default # (OpenSSL's default setting is 'DEFAULT:!aNULL:!eNULL') -_DEFAULT_CIPHERS = 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2' +# Enable a better set of ciphers by default +# This list has been explicitly chosen to: +# * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) +# * Prefer ECDHE over DHE for better performance +# * Prefer any AES-GCM over any AES-CBC for better performance and security +# * Then Use HIGH cipher suites as a fallback +# * Then Use 3DES as fallback which is secure but slow +# * Finally use RC4 as a fallback which is problematic but needed for +# compatibility some times. +# * Disable NULL authentication, NULL encryption, and MD5 MACs for security +# reasons +_DEFAULT_CIPHERS = ( + 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' + 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:' + 'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5' +) -# restricted and more secure ciphers -# HIGH: high encryption cipher suites with key length >= 128 bits (no MD5) -# !aNULL: only authenticated cipher suites (no anonymous DH) -# !RC4: no RC4 streaming cipher, RC4 is broken -# !DSS: RSA is preferred over DSA -_RESTRICTED_CIPHERS = 'HIGH:!aNULL:!RC4:!DSS' +# Restricted and more secure ciphers +# This list has been explicitly chosen to: +# * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) +# * Prefer ECDHE over DHE for better performance +# * Prefer any AES-GCM over any AES-CBC for better performance and security +# * Then Use HIGH cipher suites as a fallback +# * Then Use 3DES as fallback which is secure but slow +# * Disable NULL authentication, NULL encryption, MD5 MACs, DSS, and RC4 for +# security reasons +_RESTRICTED_CIPHERS = ( + 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' + 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:' + '!eNULL:!MD5:!DSS:!RC4' +) class CertificateError(ValueError): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,9 @@ Library ------- +- Issue #20995: Enhance default ciphers used by the ssl module to enable + better security an prioritize perfect forward secrecy. + - Issue #20884: Don't assume that __file__ is defined on importlib.__init__. - Issue #20879: Delay the initialization of encoding and decoding tables for -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 02:39:02 2014 From: python-checkins at python.org (donald.stufft) Date: Sat, 22 Mar 2014 02:39:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_changes_from_3=2E4_to_bring_in_fixes_for_Issue_=23?= =?utf-8?q?20995?= Message-ID: <3frPdy5329z7LjV@mail.python.org> http://hg.python.org/cpython/rev/60f696488c4f changeset: 89911:60f696488c4f parent: 89909:7761b3dff2e8 parent: 89910:e9749a7aa958 user: Donald Stufft date: Fri Mar 21 21:38:50 2014 -0400 summary: Merge changes from 3.4 to bring in fixes for Issue #20995 files: Doc/library/ssl.rst | 15 +++--------- Lib/ssl.py | 37 ++++++++++++++++++++++++++------ Misc/NEWS | 6 +++++ 3 files changed, 40 insertions(+), 18 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1665,17 +1665,10 @@ enabled when negotiating a SSL session is possible through the :meth:`SSLContext.set_ciphers` method. Starting from Python 3.2.3, the ssl module disables certain weak ciphers by default, but you may want -to further restrict the cipher choice. For example:: - - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) - context.set_ciphers('HIGH:!aNULL:!eNULL') - -The ``!aNULL:!eNULL`` part of the cipher spec is necessary to disable ciphers -which don't provide both encryption and authentication. Be sure to read -OpenSSL's documentation about the `cipher list -format `_. -If you want to check which ciphers are enabled by a given cipher list, -use the ``openssl ciphers`` command on your system. +to further restrict the cipher choice. Be sure to read OpenSSL's documentation +about the `cipher list format `_. +If you want to check which ciphers are enabled by a given cipher list, use the +``openssl ciphers`` command on your system. Multi-processing ^^^^^^^^^^^^^^^^ diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -162,14 +162,37 @@ # Disable weak or insecure ciphers by default # (OpenSSL's default setting is 'DEFAULT:!aNULL:!eNULL') -_DEFAULT_CIPHERS = 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2' +# Enable a better set of ciphers by default +# This list has been explicitly chosen to: +# * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) +# * Prefer ECDHE over DHE for better performance +# * Prefer any AES-GCM over any AES-CBC for better performance and security +# * Then Use HIGH cipher suites as a fallback +# * Then Use 3DES as fallback which is secure but slow +# * Finally use RC4 as a fallback which is problematic but needed for +# compatibility some times. +# * Disable NULL authentication, NULL encryption, and MD5 MACs for security +# reasons +_DEFAULT_CIPHERS = ( + 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' + 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:' + 'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5' +) -# restricted and more secure ciphers -# HIGH: high encryption cipher suites with key length >= 128 bits (no MD5) -# !aNULL: only authenticated cipher suites (no anonymous DH) -# !RC4: no RC4 streaming cipher, RC4 is broken -# !DSS: RSA is preferred over DSA -_RESTRICTED_CIPHERS = 'HIGH:!aNULL:!RC4:!DSS' +# Restricted and more secure ciphers +# This list has been explicitly chosen to: +# * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) +# * Prefer ECDHE over DHE for better performance +# * Prefer any AES-GCM over any AES-CBC for better performance and security +# * Then Use HIGH cipher suites as a fallback +# * Then Use 3DES as fallback which is secure but slow +# * Disable NULL authentication, NULL encryption, MD5 MACs, DSS, and RC4 for +# security reasons +_RESTRICTED_CIPHERS = ( + 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' + 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:' + '!eNULL:!MD5:!DSS:!RC4' +) class CertificateError(ValueError): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #20995: Enhance default ciphers used by the ssl module to enable + better security an prioritize perfect forward secrecy. + - Issue #20627: xmlrpc.client.ServerProxy is now a context manager. - Issue #19165: The formatter module now raises DeprecationWarning instead of @@ -40,6 +43,9 @@ - Issue #20574: Implement incremental decoder for cp65001 code (Windows code page 65001, Microsoft UTF-8). +======= +- Issue #20884: Don't assume that __file__ is defined on importlib.__init__. +>>>>>>> other - Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 02:50:05 2014 From: python-checkins at python.org (donald.stufft) Date: Sat, 22 Mar 2014 02:50:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_an_incomplete_merge_of?= =?utf-8?q?_Misc/NEWS?= Message-ID: <3frPtj5mFPz7LjV@mail.python.org> http://hg.python.org/cpython/rev/c29301b1503f changeset: 89912:c29301b1503f user: Donald Stufft date: Fri Mar 21 21:49:38 2014 -0400 summary: Fix an incomplete merge of Misc/NEWS files: Misc/NEWS | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,8 @@ - Issue #20995: Enhance default ciphers used by the ssl module to enable better security an prioritize perfect forward secrecy. +- Issue #20884: Don't assume that __file__ is defined on importlib.__init__. + - Issue #20627: xmlrpc.client.ServerProxy is now a context manager. - Issue #19165: The formatter module now raises DeprecationWarning instead of @@ -43,9 +45,6 @@ - Issue #20574: Implement incremental decoder for cp65001 code (Windows code page 65001, Microsoft UTF-8). -======= -- Issue #20884: Don't assume that __file__ is defined on importlib.__init__. ->>>>>>> other - Issue #20879: Delay the initialization of encoding and decoding tables for base32, ascii85 and base85 codecs in the base64 module, and delay the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 04:58:26 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 22 Mar 2014 04:58:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315968=3A_Incorpor?= =?utf-8?q?ated_Tcl=2C_Tk=2C_and_Tix_builds_into_the_Windows_build?= Message-ID: <3frSkp30fNz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/c2e2dc6c8769 changeset: 89913:c2e2dc6c8769 user: Zachary Ware date: Fri Mar 21 22:58:19 2014 -0500 summary: Issue #15968: Incorporated Tcl, Tk, and Tix builds into the Windows build solution. Currently, Tix is not built in Debug configuration. This change also: - simplifies some Tcl/Tk-related msbuild properties for _tkinter - copies the Tcl and Tk DLLs into the build output directory, meaning they will always be available after a build without having to copy them manually or change PATH - removes PCbuild/build_tkinter.py: the solution does the build without needing to invoke Python (so Tcl/Tk/Tix can be built in parallel with the rest of the build using the `/m` msbuild command line switch) - removes an outdated README concerning building Tcl/Tk on AMD64 files: Misc/NEWS | 6 + PCbuild/_tkinter.vcxproj | 32 ++- PCbuild/build_tkinter.py | 78 ------- PCbuild/debug.props | 4 + PCbuild/pcbuild.sln | 52 ++++ PCbuild/pyproject.props | 34 +-- PCbuild/readme.txt | 38 +-- PCbuild/release.props | 4 + PCbuild/rt.bat | 4 +- PCbuild/tcl.vcxproj | 175 ++++++++++++++++ PCbuild/tcltk.props | 30 ++ PCbuild/tix.vcxproj | 171 ++++++++++++++++ PCbuild/tk.vcxproj | 178 +++++++++++++++++ PCbuild/x64.props | 8 +- Tools/buildbot/README.tcltk-AMD64 | 36 --- Tools/buildbot/external-amd64.bat | 15 - Tools/buildbot/external-common.bat | 7 +- Tools/buildbot/external.bat | 16 - 18 files changed, 681 insertions(+), 207 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -103,6 +103,12 @@ - Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. Patch by Arfrever Frehtes Taifersar Arahesis. +Build +----- + +- Issue #15968: Incorporated Tcl, Tk, and Tix builds into the Windows build + solution. + What's New in Python 3.4.0? =========================== diff --git a/PCbuild/_tkinter.vcxproj b/PCbuild/_tkinter.vcxproj --- a/PCbuild/_tkinter.vcxproj +++ b/PCbuild/_tkinter.vcxproj @@ -85,41 +85,49 @@ + + + + + + + + @@ -155,7 +163,7 @@ WITH_APPINIT;%(PreprocessorDefinitions) - $(tcltkLibDebug);%(AdditionalDependencies) + $(tcltkLib);%(AdditionalDependencies) @@ -163,11 +171,11 @@ X64 - $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + $(tcltkDir)\include;%(AdditionalIncludeDirectories) WITH_APPINIT;%(PreprocessorDefinitions) - $(tcltk64LibDebug);%(AdditionalDependencies) + $(tcltkLib);%(AdditionalDependencies) @@ -184,11 +192,11 @@ X64 - $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + $(tcltkDir)\include;%(AdditionalIncludeDirectories) WITH_APPINIT;%(PreprocessorDefinitions) - $(tcltk64Lib);%(AdditionalDependencies) + $(tcltkLib);%(AdditionalDependencies) @@ -205,11 +213,11 @@ X64 - $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + $(tcltkDir)\include;%(AdditionalIncludeDirectories) WITH_APPINIT;%(PreprocessorDefinitions) - $(tcltk64Lib);%(AdditionalDependencies) + $(tcltkLib);%(AdditionalDependencies) MachineX64 @@ -227,11 +235,11 @@ X64 - $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + $(tcltkDir)\include;%(AdditionalIncludeDirectories) WITH_APPINIT;%(PreprocessorDefinitions) - $(tcltk64Lib);%(AdditionalDependencies) + $(tcltkLib);%(AdditionalDependencies) MachineX64 @@ -244,6 +252,12 @@ {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} false + + {b5fd6f1d-129e-4bff-9340-03606fac7283} + + + {7e85eccf-a72c-4da4-9e52-884508e80ba1} + diff --git a/PCbuild/build_tkinter.py b/PCbuild/build_tkinter.py deleted file mode 100644 --- a/PCbuild/build_tkinter.py +++ /dev/null @@ -1,78 +0,0 @@ -"""Script to compile the dependencies of _tkinter - -Copyright (c) 2007 by Christian Heimes - -Licensed to PSF under a Contributor Agreement. -""" - -import os -import sys - -here = os.path.abspath(os.path.dirname(__file__)) -par = os.path.pardir - -TCL = "tcl8.6.1" -TK = "tk8.6.1" -TIX = "tix-8.4.3.3" - -ROOT = os.path.abspath(os.path.join(here, par, par)) -NMAKE = ('nmake /nologo /f %s %s %s') - -def nmake(makefile, command="", **kw): - defines = ' '.join(k+'='+str(v) for k, v in kw.items()) - cmd = NMAKE % (makefile, defines, command) - print("\n\n"+cmd+"\n") - if os.system(cmd) != 0: - raise RuntimeError(cmd) - -def build(platform, clean): - if platform == "Win32": - dest = os.path.join(ROOT, "tcltk") - machine = "IX86" - elif platform == "AMD64": - dest = os.path.join(ROOT, "tcltk64") - machine = "AMD64" - else: - raise ValueError(platform) - - # TCL - tcldir = os.path.join(ROOT, TCL) - if 1: - os.chdir(os.path.join(tcldir, "win")) - if clean: - nmake("makefile.vc", "clean") - nmake("makefile.vc", MACHINE=machine) - nmake("makefile.vc", "install", INSTALLDIR=dest, MACHINE=machine) - - # TK - if 1: - os.chdir(os.path.join(ROOT, TK, "win")) - if clean: - nmake("makefile.vc", "clean", DEBUG=0, TCLDIR=tcldir) - nmake("makefile.vc", DEBUG=0, MACHINE=machine, TCLDIR=tcldir) - nmake("makefile.vc", "install", DEBUG=0, INSTALLDIR=dest, MACHINE=machine, TCLDIR=tcldir) - - # TIX - if 1: - # python9.mak is available at http://svn.python.org - os.chdir(os.path.join(ROOT, TIX, "win")) - if clean: - nmake("python.mak", "clean") - nmake("python.mak", MACHINE=machine, INSTALL_DIR=dest) - nmake("python.mak", "install", MACHINE=machine, INSTALL_DIR=dest) - -def main(): - if len(sys.argv) < 2 or sys.argv[1] not in ("Win32", "AMD64"): - print("%s Win32|AMD64" % sys.argv[0]) - sys.exit(1) - - if "-c" in sys.argv: - clean = True - else: - clean = False - - build(sys.argv[1], clean) - - -if __name__ == '__main__': - main() diff --git a/PCbuild/debug.props b/PCbuild/debug.props --- a/PCbuild/debug.props +++ b/PCbuild/debug.props @@ -3,6 +3,7 @@ _d $(OutDir)kill_python_d.exe + g <_ProjectFileVersion>10.0.30319.1 @@ -23,5 +24,8 @@ $(KillPythonExe) + + $(TclDebugExt) + \ No newline at end of file diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -78,6 +78,12 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testembed", "_testembed.vcxproj", "{6DAC66D9-E703-4624-BE03-49112AB5AA62}" EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "tcl", "tcl.vcxproj", "{B5FD6F1D-129E-4BFF-9340-03606FAC7283}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "tk", "tk.vcxproj", "{7E85ECCF-A72C-4DA4-9E52-884508E80BA1}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "tix", "tix.vcxproj", "{C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -645,6 +651,52 @@ {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|Win32.Build.0 = Release|Win32 {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.ActiveCfg = Release|x64 {6DAC66D9-E703-4624-BE03-49112AB5AA62}.Release|x64.Build.0 = Release|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Debug|Win32.ActiveCfg = Debug|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Debug|Win32.Build.0 = Debug|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Debug|x64.ActiveCfg = Debug|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Debug|x64.Build.0 = Debug|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGInstrument|Win32.ActiveCfg = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGInstrument|Win32.Build.0 = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGUpdate|Win32.ActiveCfg = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGUpdate|Win32.Build.0 = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Release|Win32.ActiveCfg = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Release|Win32.Build.0 = Release|Win32 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Release|x64.ActiveCfg = Release|x64 + {B5FD6F1D-129E-4BFF-9340-03606FAC7283}.Release|x64.Build.0 = Release|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Debug|Win32.ActiveCfg = Debug|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Debug|Win32.Build.0 = Debug|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Debug|x64.ActiveCfg = Debug|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Debug|x64.Build.0 = Debug|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGInstrument|Win32.ActiveCfg = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGInstrument|Win32.Build.0 = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGUpdate|Win32.ActiveCfg = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGUpdate|Win32.Build.0 = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|Win32.ActiveCfg = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|Win32.Build.0 = Release|Win32 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|x64.ActiveCfg = Release|x64 + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|x64.Build.0 = Release|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|Win32.ActiveCfg = Debug|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|x64.ActiveCfg = Debug|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Release|Win32.ActiveCfg = Release|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Release|Win32.Build.0 = Release|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Release|x64.ActiveCfg = Release|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -21,12 +21,10 @@ $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.5 $(externalsDir)\openssl-1.0.1e + $(externalsDir)\tcl-8.6.1.0 + $(externalsDir)\tk-8.6.1.0 + $(externalsDir)\tix-8.4.3.3 $(externalsDir)\tcltk - $(externalsDir)\tcltk64 - $(tcltkDir)\lib\tcl86t.lib;$(tcltkDir)\lib\tk86t.lib - $(tcltkDir)\lib\tcl86tg.lib;$(tcltkDir)\lib\tk86tg.lib - $(tcltk64Dir)\lib\tcl86t.lib;$(tcltk64Dir)\lib\tk86t.lib - $(tcltk64Dir)\lib\tcl86tg.lib;$(tcltk64Dir)\lib\tk86tg.lib @@ -82,23 +80,17 @@ $(opensslDir) + + $(tclDir) + + + $(tkDir) + + + $(tixDir) + $(tcltkDir) - - $(tcltk64Dir) - - - $(tcltkLib) - - - $(tcltkLibDebug) - - - $(tcltk64Lib) - - - $(tcltk64LibDebug) - - + \ No newline at end of file diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -217,11 +217,19 @@ Homepage: http://www.tcl.tk/ - Unlike the other external libraries listed above, Tk must be built - separately before the _tkinter module can be built. This means that - a pre-built Tcl/Tk installation is expected in ..\..\tcltk (tcltk64 - for 64-bit) relative to this directory. See "Getting External - Sources" below for the easiest method to ensure Tcl/Tk is built. + Tkinter's dependencies are built by the tcl.vcxproj and tk.vcxproj + projects. The tix.vcxproj project also builds the Tix extended + widget set for use with Tkinter. + + Those three projects install their respective components in a + directory alongside the source directories called "tcltk" on + Win32 and "tcltk64" on x64. They also copy the Tcl and Tk DLLs + into the current output directory, which should ensure that Tkinter + is able to load Tcl/Tk without having to change your PATH. + + The tcl, tk, and tix sub-projects do not have the ability to clean + their builds; if you need to rebuild, you'll have to clean them by + hand. Getting External Sources @@ -250,26 +258,6 @@ anyway, since that is where the solution is set to look for xz. The same is true for all other external projects. -The external(-amd64).bat scripts will also build a debug build of -Tcl/Tk, but there aren't any equivalent batch files for building release -versions of Tcl/Tk currently available. If you need to build a release -version of Tcl/Tk, just take a look at the relevant external(-amd64).bat -file and find the two nmake lines, then call each one without the -'DEBUG=1' parameter, i.e.: - -The external-amd64.bat file contains this for tcl: - nmake -f makefile.vc DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install - -So for a release build, you'd call it as: - nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install - -Note that the above command is called from within ..\..\tcl-8.6.1.0\win -(relative to this directory); don't forget to build Tk as well as Tcl! - -This will be cleaned up in the future; http://bugs.python.org/issue15968 -tracks adding a new tcltk.vcxproj file that will build Tcl/Tk and Tix -the same way the other external projects listed above are built. - Building for AMD64 ------------------ diff --git a/PCbuild/release.props b/PCbuild/release.props --- a/PCbuild/release.props +++ b/PCbuild/release.props @@ -2,6 +2,7 @@ + <_ProjectFileVersion>10.0.30319.1 @@ -15,5 +16,8 @@ $(PyDebugExt) + + $(TclDebugExt) + \ No newline at end of file diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -30,15 +30,13 @@ set suffix= set qmode= set dashO= -set tcltk=tcltk :CheckOpts if "%1"=="-O" (set dashO=-O) & shift & goto CheckOpts if "%1"=="-q" (set qmode=yes) & shift & goto CheckOpts if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts -if "%1"=="-x64" (set prefix=amd64) & (set tcltk=tcltk64) & shift & goto CheckOpts +if "%1"=="-x64" (set prefix=amd64) & shift & goto CheckOpts -PATH %PATH%;%~dp0..\..\%tcltk%\bin set exe=%prefix%\python%suffix% set cmd=%exe% %dashO% -Wd -E -bb ../lib/test/regrtest.py %1 %2 %3 %4 %5 %6 %7 %8 %9 if defined qmode goto Qmode diff --git a/PCbuild/tcl.vcxproj b/PCbuild/tcl.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/tcl.vcxproj @@ -0,0 +1,175 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {B5FD6F1D-129E-4BFF-9340-03606FAC7283} + tcl + MakeFileProj + + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + +IF EXIST $(tcltkDir)\include\tcl.h ( + IF EXIST $(tcltkDir)\bin\$(tclDLLName) ( + IF EXIST $(OutDir)$(tclDLLName) ( + echo Tcl is already built and available. + exit /b 0 + ) ELSE ( + echo Copying tcl dll to $(OutDir) + xcopy $(tcltkDir)\bin\$(tclDLLName) $(OutDir) + exit /b 0 + ) + ) +) + +IF NOT EXIST $(tclDir) ( + echo error: $(tclDir) doesn't exist. + exit 1 +) + +IF "$(Platform)" EQU "Win32" set TclMachine=IX86 +IF "$(Platform)" EQU "x64" set TclMachine=AMD64 + +IF "$(Configuration)" EQU "Debug" ( + set TclOpts=symbols +) ELSE ( + set TclOpts= +) + +cd $(tclDir)\win + +nmake -f makefile.vc MACHINE=%TclMachine% OPTS=%TclOpts% core shell dlls && nmake -f makefile.vc MACHINE=%TclMachine% OPTS=%TclOpts% INSTALLDIR=$(SolutionDir)$(tcltkDir) install-binaries install-libraries && xcopy /y $(SolutionDir)$(tcltkDir)\bin\$(tclDLLName) $(OutDir) + + del $(OutDir)$(tclDLLName) +echo Tcl must be cleaned manually if you want to rebuild it. + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/tcltk.props b/PCbuild/tcltk.props new file mode 100644 --- /dev/null +++ b/PCbuild/tcltk.props @@ -0,0 +1,30 @@ +? + + + + Dummy value to avoid patching Tcl/Tk's makefile.vc + tcl86t$(TclDebugExt).dll + tk86t$(TclDebugExt).dll + tix84$(TclDebugExt).dll + $(tcltkDir)\lib\tcl86t$(TclDebugExt).lib;$(tcltkDir)\lib\tk86t$(TclDebugExt).lib + + + + + $(MSDEVDIR) + true + + + $(tclDLLName) + + + $(tkDLLName) + + + $(tixDLLName) + + + $(tcltkLib) + + + \ No newline at end of file diff --git a/PCbuild/tix.vcxproj b/PCbuild/tix.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/tix.vcxproj @@ -0,0 +1,171 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555} + tix + MakeFileProj + + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + +IF EXIST $(tcltkDir)\lib\tix8.4.3\$(tixDLLName) ( + echo Tix is already built and available. + exit /b 0 +) + +IF NOT EXIST $(tixDir) ( + echo error: $(tixDir) doesn't exist. + exit 1 +) + +IF "$(Platform)" EQU "Win32" set TclMachine=IX86 +IF "$(Platform)" EQU "x64" set TclMachine=AMD64 + +IF "$(Configuration)" EQU "Debug" ( + set TixDebug=1 +) ELSE ( + set TixDebug=0 +) + +cd $(tixDir)\win +nmake -f python.mak MACHINE=%TclMachine% DEBUG=%TixDebug% TCL_DIR=$(SolutionDir)$(tclDir) TK_DIR=$(SolutionDir)$(tkDir) all && nmake -f python.mak MACHINE=%TclMachine% DEBUG=%TixDebug% TCL_DIR=$(SolutionDir)$(tclDir) TK_DIR=$(SolutionDir)$(tkDir) INSTALL_DIR=$(SolutionDir)$(tcltkDir) install + + echo Tix must be cleaned manually if you want to rebuild it. + + + + + + {b5fd6f1d-129e-4bff-9340-03606fac7283} + + + {7e85eccf-a72c-4da4-9e52-884508e80ba1} + + + + + + \ No newline at end of file diff --git a/PCbuild/tk.vcxproj b/PCbuild/tk.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/tk.vcxproj @@ -0,0 +1,178 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {7E85ECCF-A72C-4DA4-9E52-884508E80BA1} + tk + MakeFileProj + + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + +IF EXIST $(tcltkDir)\include\tk.h ( + IF EXIST $(tcltkDir)\bin\$(tkDLLName) ( + IF EXIST $(OutDir)$(tkDLLName) ( + echo Tk is already built and available. + exit /b 0 + ) ELSE ( + echo Copying tk dll to $(OutDir) + xcopy $(tcltkDir)\bin\$(tkDLLName) $(OutDir) + exit /b 0 + ) + ) +) + +IF NOT EXIST $(tkDir) ( + echo error: $(tkDir) doesn't exist. + exit 1 +) + +IF "$(Platform)" EQU "Win32" set TclMachine=IX86 +IF "$(Platform)" EQU "x64" set TclMachine=AMD64 + +IF "$(Configuration)" EQU "Debug" ( + set TclOpts=symbols,noxp +) ELSE ( + set TclOpts=noxp +) + +cd $(tkDir)\win + +nmake -f makefile.vc MACHINE=%TclMachine% OPTS=%TclOpts% TCLDIR=$(SolutionDir)$(tclDir) all && nmake -f makefile.vc MACHINE=%TclMachine% OPTS=%TclOpts% TCLDIR=$(SolutionDir)$(tclDir) INSTALLDIR=$(SolutionDir)$(tcltkDir) install-binaries install-libraries && xcopy /y $(SolutionDir)$(tcltkDir)\bin\$(tkDLLName) $(OutDir) + + del $(OutDir)$(tkDLLName) +echo Tk must be cleaned manually if you want to rebuild it. + + + + + + {b5fd6f1d-129e-4bff-9340-03606fac7283} + + + + + + \ No newline at end of file diff --git a/PCbuild/x64.props b/PCbuild/x64.props --- a/PCbuild/x64.props +++ b/PCbuild/x64.props @@ -3,6 +3,9 @@ $(HOST_PYTHON) + + $(externalsDir)\tcltk64 + false @@ -16,5 +19,8 @@ $(PythonExe) + + $(tcltkDir) + - + \ No newline at end of file diff --git a/Tools/buildbot/README.tcltk-AMD64 b/Tools/buildbot/README.tcltk-AMD64 deleted file mode 100644 --- a/Tools/buildbot/README.tcltk-AMD64 +++ /dev/null @@ -1,36 +0,0 @@ -Comments on building tcl/tk for AMD64 with the MS SDK compiler -============================================================== - -I did have to build tcl/tk manually. - -First, I had to build the nmakehlp.exe helper utility manually by executing - cl nmakehlp.c /link bufferoverflowU.lib -in both the tcl8.4.12\win and tk8.4.12\win directories. - -Second, the AMD64 compiler refuses to compile the file -tcl8.4.12\generic\tclExecute.c because it insists on using intrinsics -for the 'ceil' and 'floor' functions: - - ..\generic\tclExecute.c(394) : error C2099: initializer is not a constant - ..\generic\tclExecute.c(398) : error C2099: initializer is not a constant - -I did comment out these lines; an alternative would have been to use -the /Oi- compiler flag to disable the intrinsic functions. -The commands then used were these: - - svn export http://svn.python.org/projects/external/tcl8.4.12 - cd tcl8.4.12\win - REM - echo patch the tcl8.4.12\generic\tclExecute.c file - pause - REM - cl nmakehlp.c /link bufferoverflowU.lib - nmake -f makefile.vc MACHINE=AMD64 - nmake -f makefile.vc INSTALLDIR=..\..\tcltk install - cd ..\.. - svn export http://svn.python.org/projects/external/tk8.4.12 - cd tk8.4.12\win - cl nmakehlp.c /link bufferoverflowU.lib - nmake -f makefile.vc TCLDIR=..\..\tcl8.4.12 MACHINE=AMD64 - nmake -f makefile.vc TCLDIR=..\..\tcl8.4.12 INSTALLDIR=..\..\tcltk install - cd ..\.. diff --git a/Tools/buildbot/external-amd64.bat b/Tools/buildbot/external-amd64.bat --- a/Tools/buildbot/external-amd64.bat +++ b/Tools/buildbot/external-amd64.bat @@ -2,20 +2,5 @@ @rem Assume we start inside the Python source directory call "Tools\buildbot\external-common.bat" -call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 -if not exist tcltk64\bin\tcl86tg.dll ( - cd tcl-8.6.1.0\win - nmake -f makefile.vc DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all - nmake -f makefile.vc DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 install - cd ..\.. -) -if not exist tcltk64\bin\tk86tg.dll ( - cd tk-8.6.1.0\win - nmake -f makefile.vc OPTS=noxp DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.6.1.0 clean - nmake -f makefile.vc OPTS=noxp DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.6.1.0 all - nmake -f makefile.vc OPTS=noxp DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.6.1.0 install - cd ..\.. -) - diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -15,7 +15,7 @@ @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 @rem if exist openssl-1.0.1e rd /s/q openssl-1.0.1e - at rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 + at rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 @rem bzip if not exist bzip2-1.0.6 ( @@ -29,12 +29,13 @@ svn export http://svn.python.org/projects/external/openssl-1.0.1e ) - at rem tcl/tk + at rem tcl/tk/tix if not exist tcl-8.6.1.0 ( - rd /s/q tcltk tcltk64 tcl-8.5.11.0 tk-8.5.11.0 + rd /s/q tcltk tcltk64 tcl-8.5.11.0 tk-8.5.11.0 tix-8.4.3.3 svn export http://svn.python.org/projects/external/tcl-8.6.1.0 ) if not exist tk-8.6.1.0 svn export http://svn.python.org/projects/external/tk-8.6.1.0 +if not exist tix-8.4.3.3 svn export http://svn.python.org/projects/external/tix-8.4.3.3 @rem sqlite3 if not exist sqlite-3.8.3.1 ( diff --git a/Tools/buildbot/external.bat b/Tools/buildbot/external.bat --- a/Tools/buildbot/external.bat +++ b/Tools/buildbot/external.bat @@ -2,20 +2,4 @@ @rem Assume we start inside the Python source directory call "Tools\buildbot\external-common.bat" -call "%VS100COMNTOOLS%\vsvars32.bat" -if not exist tcltk\bin\tcl86tg.dll ( - @rem all and install need to be separate invocations, otherwise nmakehlp is not found on install - cd tcl-8.6.1.0\win - nmake -f makefile.vc DEBUG=1 INSTALLDIR=..\..\tcltk clean all - nmake -f makefile.vc DEBUG=1 INSTALLDIR=..\..\tcltk install - cd ..\.. -) - -if not exist tcltk\bin\tk86tg.dll ( - cd tk-8.6.1.0\win - nmake -f makefile.vc OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.6.1.0 clean - nmake -f makefile.vc OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.6.1.0 all - nmake -f makefile.vc OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.6.1.0 install - cd ..\.. -) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 05:17:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 22 Mar 2014 05:17:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_improve_the_command-line_i?= =?utf-8?q?nterface_of_json=2Etool_=28closes_=2321000=29?= Message-ID: <3frT9M0F5Mz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/a2ad16e86e60 changeset: 89914:a2ad16e86e60 user: Benjamin Peterson date: Fri Mar 21 23:17:29 2014 -0500 summary: improve the command-line interface of json.tool (closes #21000) A patch from Berker Peksag. files: Doc/library/json.rst | 51 +++++++++++++++++++++ Lib/json/tool.py | 27 ++++++---- Lib/test/test_json/test_tool.py | 8 +++ Misc/NEWS | 2 + 4 files changed, 76 insertions(+), 12 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -104,6 +104,8 @@ $ echo '{1.2:3.4}' | python -mjson.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -563,3 +565,52 @@ {'x': 3} The *object_pairs_hook* parameter can be used to alter this behavior. + +.. highlight:: bash + +.. _json-commandline: + +Command Line Interface +---------------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional :option:`infile` and :option:`outfile` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: [] + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + +.. cmdoption:: [] + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: -h, --help + + Show the help message. diff --git a/Lib/json/tool.py b/Lib/json/tool.py --- a/Lib/json/tool.py +++ b/Lib/json/tool.py @@ -10,21 +10,24 @@ Expecting property name enclosed in double quotes: line 1 column 3 (char 2) """ +import argparse +import json import sys -import json + def main(): - if len(sys.argv) == 1: - infile = sys.stdin - outfile = sys.stdout - elif len(sys.argv) == 2: - infile = open(sys.argv[1], 'r') - outfile = sys.stdout - elif len(sys.argv) == 3: - infile = open(sys.argv[1], 'r') - outfile = open(sys.argv[2], 'w') - else: - raise SystemExit(sys.argv[0] + " [infile [outfile]]") + prog = 'python -m json.tool' + description = ('A simple command line interface for json module ' + 'to validate and pretty-print JSON objects.') + parser = argparse.ArgumentParser(prog=prog, description=description) + parser.add_argument('infile', nargs='?', type=argparse.FileType(), + help='a JSON file to be validated or pretty-printed') + parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), + help='write the output of infile to outfile') + options = parser.parse_args() + + infile = options.infile or sys.stdin + outfile = options.outfile or sys.stdout with infile: try: obj = json.load(infile) diff --git a/Lib/test/test_json/test_tool.py b/Lib/test/test_json/test_tool.py --- a/Lib/test/test_json/test_tool.py +++ b/Lib/test/test_json/test_tool.py @@ -55,6 +55,7 @@ def test_infile_stdout(self): infile = self._create_infile() rc, out, err = assert_python_ok('-m', 'json.tool', infile) + self.assertEqual(rc, 0) self.assertEqual(out.splitlines(), self.expect.encode().splitlines()) self.assertEqual(err, b'') @@ -65,5 +66,12 @@ self.addCleanup(os.remove, outfile) with open(outfile, "r") as fp: self.assertEqual(fp.read(), self.expect) + self.assertEqual(rc, 0) self.assertEqual(out, b'') self.assertEqual(err, b'') + + def test_help_flag(self): + rc, out, err = assert_python_ok('-m', 'json.tool', '-h') + self.assertEqual(rc, 0) + self.assertTrue(out.startswith(b'usage: ')) + self.assertEqual(err, b'') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #21000: Improve the command-line interface of json.tool. + - Issue #20995: Enhance default ciphers used by the ssl module to enable better security an prioritize perfect forward secrecy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 06:36:37 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 22 Mar 2014 06:36:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315968=3A_Temporar?= =?utf-8?q?ily_revert_change_to_PCbuild/rt=2Ebat?= Message-ID: <3frVw51xLtz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/c12cc78d59c1 changeset: 89915:c12cc78d59c1 user: Zachary Ware date: Sat Mar 22 00:36:25 2014 -0500 summary: Issue #15968: Temporarily revert change to PCbuild/rt.bat to prevent buildbot failures on test_idle. This change should go back in once #20035 is fixed. files: PCbuild/rt.bat | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -30,13 +30,15 @@ set suffix= set qmode= set dashO= +set tcltk=tcltk :CheckOpts if "%1"=="-O" (set dashO=-O) & shift & goto CheckOpts if "%1"=="-q" (set qmode=yes) & shift & goto CheckOpts if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts -if "%1"=="-x64" (set prefix=amd64) & shift & goto CheckOpts +if "%1"=="-x64" (set prefix=amd64) & (set tcltk=tcltk64) & shift & goto CheckOpts +PATH %PATH%;%~dp0..\..\%tcltk%\bin set exe=%prefix%\python%suffix% set cmd=%exe% %dashO% -Wd -E -bb ../lib/test/regrtest.py %1 %2 %3 %4 %5 %6 %7 %8 %9 if defined qmode goto Qmode -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Mar 22 09:42:14 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 22 Mar 2014 09:42:14 +0100 Subject: [Python-checkins] Daily reference leaks (c29301b1503f): sum=4 Message-ID: results for c29301b1503f on branch "default" -------------------------------------------- test_site leaked [2, -2, 2] references, sum=2 test_site leaked [2, -2, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogiZoP2h', '-x'] From python-checkins at python.org Sat Mar 22 16:36:51 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 22 Mar 2014 16:36:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321017=3A_Enable_T?= =?utf-8?q?ix_debug_build_on_Windows?= Message-ID: <3frmDg1N6qz7Ljc@mail.python.org> http://hg.python.org/cpython/rev/8cf384852680 changeset: 89916:8cf384852680 user: Zachary Ware date: Sat Mar 22 10:37:06 2014 -0500 summary: Issue #21017: Enable Tix debug build on Windows files: PCbuild/pcbuild.sln | 2 ++ PCbuild/pyproject.props | 2 +- Tools/buildbot/external-common.bat | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -684,7 +684,9 @@ {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|x64.ActiveCfg = Release|x64 {7E85ECCF-A72C-4DA4-9E52-884508E80BA1}.Release|x64.Build.0 = Release|x64 {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|Win32.ActiveCfg = Debug|Win32 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|Win32.Build.0 = Debug|Win32 {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|x64.ActiveCfg = Debug|x64 + {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.Debug|x64.Build.0 = Debug|x64 {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 {C5A3E7FB-9695-4B2E-960B-1D9F43F1E555}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -23,7 +23,7 @@ $(externalsDir)\openssl-1.0.1e $(externalsDir)\tcl-8.6.1.0 $(externalsDir)\tk-8.6.1.0 - $(externalsDir)\tix-8.4.3.3 + $(externalsDir)\tix-8.4.3.4 $(externalsDir)\tcltk diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -35,7 +35,7 @@ svn export http://svn.python.org/projects/external/tcl-8.6.1.0 ) if not exist tk-8.6.1.0 svn export http://svn.python.org/projects/external/tk-8.6.1.0 -if not exist tix-8.4.3.3 svn export http://svn.python.org/projects/external/tix-8.4.3.3 +if not exist tix-8.4.3.4 svn export http://svn.python.org/projects/external/tix-8.4.3.4 @rem sqlite3 if not exist sqlite-3.8.3.1 ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 18:15:09 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 22 Mar 2014 18:15:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDE1?= =?utf-8?q?=3A_SSL_contexts_will_now_automatically_select_an_elliptic_curv?= =?utf-8?q?e_for?= Message-ID: <3frpQ52jhNz7LjV@mail.python.org> http://hg.python.org/cpython/rev/869277faf3dc changeset: 89917:869277faf3dc branch: 3.4 parent: 89910:e9749a7aa958 user: Antoine Pitrou date: Sat Mar 22 18:13:50 2014 +0100 summary: Issue #21015: SSL contexts will now automatically select an elliptic curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise default to "prime256v1". (should also fix a buildbot failure introduced by #20995) files: Lib/test/test_ssl.py | 12 ++++++++++++ Misc/NEWS | 4 ++++ Modules/_ssl.c | 15 +++++++++++++++ 3 files changed, 31 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2567,6 +2567,18 @@ s.connect((HOST, server.port)) self.assertIn("no shared cipher", str(server.conn_errors[0])) + @unittest.skipUnless(ssl.HAS_ECDH, "test requires ECDH-enabled OpenSSL") + def test_default_ecdh_curve(self): + # Issue #21015: elliptic curve-based Diffie Hellman key exchange + # should be enabled by default on SSL contexts. + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + context.load_cert_chain(CERTFILE) + context.set_ciphers("ECDH") + with ThreadedEchoServer(context=context) as server: + with context.wrap_socket(socket.socket()) as s: + s.connect((HOST, server.port)) + self.assertIn("ECDH", s.cipher()[0]) + @unittest.skipUnless("tls-unique" in ssl.CHANNEL_BINDING_TYPES, "'tls-unique' channel binding not available") def test_tls_unique_channel_binding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,10 @@ Library ------- +- Issue #21015: SSL contexts will now automatically select an elliptic + curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise + default to "prime256v1". + - Issue #20995: Enhance default ciphers used by the ssl module to enable better security an prioritize perfect forward secrecy. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2059,6 +2059,21 @@ options |= SSL_OP_NO_SSLv2; SSL_CTX_set_options(self->ctx, options); +#ifndef OPENSSL_NO_ECDH + /* Allow automatic ECDH curve selection (on OpenSSL 1.0.2+), or use + prime256v1 by default. This is Apache mod_ssl's initialization + policy, so we should be safe. */ +#if defined(SSL_CTX_set_ecdh_auto) + SSL_CTX_set_ecdh_auto(self->ctx, 1); +#else + { + EC_KEY *key = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1); + SSL_CTX_set_tmp_ecdh(self->ctx, key); + EC_KEY_free(key); + } +#endif +#endif + #define SID_CTX "Python" SSL_CTX_set_session_id_context(self->ctx, (const unsigned char *) SID_CTX, sizeof(SID_CTX)); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 18:15:10 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 22 Mar 2014 18:15:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2321015=3A_SSL_contexts_will_now_automatically_se?= =?utf-8?q?lect_an_elliptic_curve_for?= Message-ID: <3frpQ653RYz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/3b81d1b3f9d1 changeset: 89918:3b81d1b3f9d1 parent: 89916:8cf384852680 parent: 89917:869277faf3dc user: Antoine Pitrou date: Sat Mar 22 18:14:57 2014 +0100 summary: Issue #21015: SSL contexts will now automatically select an elliptic curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise default to "prime256v1". (should also fix a buildbot failure introduced by #20995) files: Lib/test/test_ssl.py | 12 ++++++++++++ Misc/NEWS | 4 ++++ Modules/_ssl.c | 15 +++++++++++++++ 3 files changed, 31 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2567,6 +2567,18 @@ s.connect((HOST, server.port)) self.assertIn("no shared cipher", str(server.conn_errors[0])) + @unittest.skipUnless(ssl.HAS_ECDH, "test requires ECDH-enabled OpenSSL") + def test_default_ecdh_curve(self): + # Issue #21015: elliptic curve-based Diffie Hellman key exchange + # should be enabled by default on SSL contexts. + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + context.load_cert_chain(CERTFILE) + context.set_ciphers("ECDH") + with ThreadedEchoServer(context=context) as server: + with context.wrap_socket(socket.socket()) as s: + s.connect((HOST, server.port)) + self.assertIn("ECDH", s.cipher()[0]) + @unittest.skipUnless("tls-unique" in ssl.CHANNEL_BINDING_TYPES, "'tls-unique' channel binding not available") def test_tls_unique_channel_binding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,10 @@ Library ------- +- Issue #21015: SSL contexts will now automatically select an elliptic + curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise + default to "prime256v1". + - Issue #21000: Improve the command-line interface of json.tool. - Issue #20995: Enhance default ciphers used by the ssl module to enable diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2059,6 +2059,21 @@ options |= SSL_OP_NO_SSLv2; SSL_CTX_set_options(self->ctx, options); +#ifndef OPENSSL_NO_ECDH + /* Allow automatic ECDH curve selection (on OpenSSL 1.0.2+), or use + prime256v1 by default. This is Apache mod_ssl's initialization + policy, so we should be safe. */ +#if defined(SSL_CTX_set_ecdh_auto) + SSL_CTX_set_ecdh_auto(self->ctx, 1); +#else + { + EC_KEY *key = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1); + SSL_CTX_set_tmp_ecdh(self->ctx, key); + EC_KEY_free(key); + } +#endif +#endif + #define SID_CTX "Python" SSL_CTX_set_session_id_context(self->ctx, (const unsigned char *) SID_CTX, sizeof(SID_CTX)); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 18:19:45 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 22 Mar 2014 18:19:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTEz?= =?utf-8?q?=3A_improve_the_SSL_security_considerations_to_first_advocate_u?= =?utf-8?q?sing?= Message-ID: <3frpWP2S40z7LjP@mail.python.org> http://hg.python.org/cpython/rev/8661ffca4581 changeset: 89919:8661ffca4581 branch: 3.4 parent: 89917:869277faf3dc user: Antoine Pitrou date: Sat Mar 22 18:19:11 2014 +0100 summary: Issue #20913: improve the SSL security considerations to first advocate using create_default_context(). files: Doc/library/asyncio-eventloop.rst | 2 + Doc/library/ftplib.rst | 51 +++++++----------- Doc/library/http.client.rst | 38 +++++++------ Doc/library/imaplib.rst | 29 ++++++---- Doc/library/nntplib.rst | 16 +++-- Doc/library/poplib.rst | 29 ++++++---- Doc/library/smtplib.rst | 27 ++++----- Doc/library/ssl.rst | 42 +++++++++++++-- 8 files changed, 133 insertions(+), 101 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -241,6 +241,8 @@ the transport; if *ssl* is :const:`True`, a context with some unspecified default settings is used. + .. seealso:: :ref:`SSL/TLS security considerations ` + * *server_hostname*, is only for use together with *ssl*, and sets or overrides the hostname that the target server's certificate will be matched against. By default the value of the *host* argument diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -80,14 +80,14 @@ :rfc:`4217`. Connect as usual to port 21 implicitly securing the FTP control connection before authenticating. Securing the data connection requires the user to - explicitly ask for it by calling the :meth:`prot_p` method. - *keyfile* and *certfile* are optional -- they can contain a PEM formatted - private key and certificate chain file name for the SSL connection. - *context* parameter is a :class:`ssl.SSLContext` object which allows - bundling SSL configuration options, certificates and private keys into a - single (potentially long-lived) structure. *source_address* is a 2-tuple - ``(host, port)`` for the socket to bind to as its source address before - connecting. + explicitly ask for it by calling the :meth:`prot_p` method. *context* + is a :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially + long-lived) structure. Please read :ref:`ssl-security` for best practices. + + *keyfile* and *certfile* are a legacy alternative to *context* -- they + can point to PEM-formatted private key and certificate chain files + (respectively) for the SSL connection. .. versionadded:: 3.2 @@ -96,29 +96,18 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). - Here's a sample session using the :class:`FTP_TLS` class: + Here's a sample session using the :class:`FTP_TLS` class:: - >>> from ftplib import FTP_TLS - >>> ftps = FTP_TLS('ftp.python.org') - >>> ftps.login() # login anonymously before securing control channel - >>> ftps.prot_p() # switch to secure data connection - >>> ftps.retrlines('LIST') # list directory content securely - total 9 - drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . - drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. - drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin - drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc - d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming - drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib - drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub - drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr - -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg - '226 Transfer complete.' - >>> ftps.quit() - >>> + >>> ftps = FTP_TLS('ftp.pureftpd.org') + >>> ftps.login() + '230 Anonymous user logged in' + >>> ftps.prot_p() + '200 Data protection level set to "private"' + >>> ftps.nlst() + ['6jack', 'OpenBSD', 'antilink', 'blogbench', 'bsdcam', 'clockspeed', 'djbdns-jedi', 'docs', 'eaccelerator-jedi', 'favicon.ico', 'francotone', 'fugu', 'ignore', 'libpuzzle', 'metalog', 'minidentd', 'misc', 'mysql-udf-global-user-variables', 'php-jenkins-hash', 'php-skein-hash', 'php-webdav', 'phpaudit', 'phpbench', 'pincaster', 'ping', 'posto', 'pub', 'public', 'public_keys', 'pure-ftpd', 'qscan', 'qtc', 'sharedance', 'skycache', 'sound', 'tmp', 'ucarp'] .. exception:: error_reply @@ -434,8 +423,8 @@ .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: FTP_TLS.ccc() diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -43,10 +43,10 @@ For example, the following calls all create instances that connect to the server at the same host and port:: - >>> h1 = http.client.HTTPConnection('www.cwi.nl') - >>> h2 = http.client.HTTPConnection('www.cwi.nl:80') - >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80) - >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80, timeout=10) + >>> h1 = http.client.HTTPConnection('www.python.org') + >>> h2 = http.client.HTTPConnection('www.python.org:80') + >>> h3 = http.client.HTTPConnection('www.python.org', 80) + >>> h4 = http.client.HTTPConnection('www.python.org', 80, timeout=10) .. versionchanged:: 3.2 *source_address* was added. @@ -64,23 +64,27 @@ A subclass of :class:`HTTPConnection` that uses SSL for communication with secure servers. Default port is ``443``. If *context* is specified, it must be a :class:`ssl.SSLContext` instance describing the various SSL - options. If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. + options. *key_file* and *cert_file* are deprecated, please use - :meth:`ssl.SSLContext.load_cert_chain` instead. + :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. - If you access arbitrary hosts on the Internet, it is recommended to - require certificate checking and feed the *context* with a set of - trusted CA certificates:: + The recommended way to connect to HTTPS hosts on the Internet is as + follows:: - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) - context.verify_mode = ssl.CERT_REQUIRED - context.load_verify_locations('/etc/pki/tls/certs/ca-bundle.crt') - h = client.HTTPSConnection('svn.python.org', 443, context=context) + context = ssl.create_default_context() + h = client.HTTPSConnection('www.python.org', 443, context=context) + + Please read :ref:`ssl-security` for more information on best practices. + + .. note:: + If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` + of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then + by default *host* is matched against the host name(s) allowed by the + server's certificate. If you want to change that behaviour, you can + explicitly set *check_hostname* to False. .. versionchanged:: 3.2 *source_address*, *context* and *check_hostname* were added. diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -69,21 +69,25 @@ This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled with SSL support). If *host* is not specified, ``''`` (the local host) is used. - If *port* is omitted, the standard IMAP4-over-SSL port (993) is used. *keyfile* - and *certfile* are also optional - they can contain a PEM formatted private key - and certificate chain file for the SSL connection. *ssl_context* parameter is a - :class:`ssl.SSLContext` object which allows bundling SSL configuration - options, certificates and private keys into a single (potentially long-lived) - structure. Note that the *keyfile*/*certfile* parameters are mutually exclusive with *ssl_context*, - a :class:`ValueError` is raised if *keyfile*/*certfile* is provided along with *ssl_context*. + If *port* is omitted, the standard IMAP4-over-SSL port (993) is used. + *ssl_context* is a :class:`ssl.SSLContext` object which allows bundling + SSL configuration options, certificates and private keys into a single + (potentially long-lived) structure. Please read :ref:`ssl-security` for + best practices. + + *keyfile* and *certfile* are a legacy alternative to *ssl_context* - they + can point to PEM-formatted private key and certificate chain files for + the SSL connection. Note that the *keyfile*/*certfile* parameters are + mutually exclusive with *ssl_context*, a :class:`ValueError` is raised + if *keyfile*/*certfile* is provided along with *ssl_context*. .. versionchanged:: 3.3 *ssl_context* parameter added. .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). The second subclass allows for connections created by a child process: @@ -437,14 +441,15 @@ Send a ``STARTTLS`` command. The *ssl_context* argument is optional and should be a :class:`ssl.SSLContext` object. This will enable - encryption on the IMAP connection. + encryption on the IMAP connection. Please read :ref:`ssl-security` for + best practices. .. versionadded:: 3.2 .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: IMAP4.status(mailbox, names) diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -94,6 +94,7 @@ port *port*. :class:`NNTP_SSL` objects have the same methods as :class:`NNTP` objects. If *port* is omitted, port 563 (NNTPS) is used. *ssl_context* is also optional, and is a :class:`~ssl.SSLContext` object. + Please read :ref:`ssl-security` for best practices. All other parameters behave the same as for :class:`NNTP`. Note that SSL-on-563 is discouraged per :rfc:`4642`, in favor of @@ -104,8 +105,8 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. exception:: NNTPError @@ -234,9 +235,10 @@ .. method:: NNTP.starttls(ssl_context=None) - Send a ``STARTTLS`` command. The *ssl_context* argument is optional - and should be a :class:`ssl.SSLContext` object. This will enable - encryption on the NNTP connection. + Send a ``STARTTLS`` command. This will enable encryption on the NNTP + connection. The *ssl_context* argument is optional and should be a + :class:`ssl.SSLContext` object. Please read :ref:`ssl-security` for best + practices. Note that this may not be done after authentication information has been transmitted, and authentication occurs by default if possible during a @@ -247,8 +249,8 @@ .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: NNTP.newgroups(date, *, file=None) diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -43,20 +43,23 @@ This is a subclass of :class:`POP3` that connects to the server over an SSL encrypted socket. If *port* is not specified, 995, the standard POP3-over-SSL - port is used. *keyfile* and *certfile* are also optional - they can contain a - PEM formatted private key and certificate chain file for the SSL connection. - *timeout* works as in the :class:`POP3` constructor. *context* parameter is a - :class:`ssl.SSLContext` object which allows bundling SSL configuration - options, certificates and private keys into a single (potentially long-lived) - structure. + port is used. *timeout* works as in the :class:`POP3` constructor. + *context* is an optional :class:`ssl.SSLContext` object which allows + bundling SSL configuration options, certificates and private keys into a + single (potentially long-lived) structure. Please read :ref:`ssl-security` + for best practices. + + *keyfile* and *certfile* are a legacy alternative to *context* - they can + point to PEM-formatted private key and certificate chain files, + respectively, for the SSL connection. .. versionchanged:: 3.2 *context* parameter added. .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). One exception is defined as an attribute of the :mod:`poplib` module: @@ -198,10 +201,12 @@ *context* parameter is a :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into - a single (potentially long-lived) structure. This method supports - hostname checking via :attr:`SSLContext.check_hostname` - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + a single (potentially long-lived) structure. Please read :ref:`ssl-security` + for best practices. + + This method supports hostname checking via + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. versionadded:: 3.4 diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -69,20 +69,15 @@ required from the beginning of the connection and using :meth:`starttls` is not appropriate. If *host* is not specified, the local host is used. If *port* is zero, the standard SMTP-over-SSL port (465) is used. The optional - arguments *local_hostname* and *source_address* have the same meaning as - they do in the :class:`SMTP` class. *keyfile* and *certfile* are also - optional, and can contain a PEM formatted private key and certificate chain - file for the SSL connection. *context* also optional, can contain a - SSLContext, and is an alternative to keyfile and certfile; If it is - specified both keyfile and certfile must be None. The optional *timeout* - parameter specifies a timeout in seconds for blocking operations like the - connection attempt (if not specified, the global default timeout setting - will be used). The optional source_address parameter allows to bind to some - specific source address in a machine with multiple network interfaces, - and/or to some specific source tcp port. It takes a 2-tuple (host, port), - for the socket to bind to as its source address before connecting. If - omitted (or if host or port are ``''`` and/or 0 respectively) the OS default - behavior will be used. + arguments *local_hostname*, *timeout* and *source_address* have the same + meaning as they do in the :class:`SMTP` class. *context*, also optional, + can contain a :class:`~ssl.SSLContext` and allows to configure various + aspects of the secure connection. Please read :ref:`ssl-security` for + best practices. + + *keyfile* and *certfile* are a legacy alternative to *context*, and can + point to a PEM formatted private key and certificate chain file for the + SSL connection. .. versionchanged:: 3.3 *context* was added. @@ -92,8 +87,8 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1542,7 +1542,7 @@ import socket, ssl - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile") bindsocket = socket.socket() @@ -1619,9 +1619,39 @@ Security considerations ----------------------- +Best defaults +^^^^^^^^^^^^^ + +For **client use**, if you don't have any special requirements for your +security policy, it is highly recommended that you use the +:func:`create_default_context` function to create your SSL context. +It will load the system's trusted CA certificates, enable certificate +validation, and try to choose reasonably secure protocol and cipher settings. + +For example, here is how you would use the :class:`smtplib.SMTP` class to +create a trusted, secure connection to a SMTP server:: + + >>> import ssl, smtplib + >>> smtp = smtplib.SMTP("mail.python.org", port=587) + >>> context = ssl.create_default_context() + >>> smtp.starttls(context=context) + (220, b'2.0.0 Ready to start TLS') + +If a client certificate is needed for the connection, it can be added with +:meth:`SSLContext.load_cert_chain`. + +By contrast, if you create the SSL context by calling the :class:`SSLContext` +constructor yourself, it will not have certificate validation enabled by +default. If you do so, please read the paragraphs below to achieve a good +security level. + +Manual settings +^^^^^^^^^^^^^^^ + Verifying certificates -^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''' +When calling the the :class:`SSLContext` constructor directly, :const:`CERT_NONE` is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you're talking to. @@ -1645,7 +1675,7 @@ by default). Protocol versions -^^^^^^^^^^^^^^^^^ +''''''''''''''''' SSL version 2 is considered insecure and is therefore dangerous to use. If you want maximum compatibility between clients and servers, it is recommended @@ -1655,11 +1685,11 @@ context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.options |= ssl.OP_NO_SSLv2 -The SSL context created above will allow SSLv3 and TLSv1 connections, but -not SSLv2. +The SSL context created above will allow SSLv3 and TLSv1 (and later, if +supported by your system) connections, but not SSLv2. Cipher selection -^^^^^^^^^^^^^^^^ +'''''''''''''''' If you have advanced security requirements, fine-tuning of the ciphers enabled when negotiating a SSL session is possible through the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 18:19:46 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 22 Mar 2014 18:19:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320913=3A_improve_the_SSL_security_consideration?= =?utf-8?q?s_to_first_advocate_using?= Message-ID: <3frpWQ6VVTz7LjV@mail.python.org> http://hg.python.org/cpython/rev/e703fc5262b7 changeset: 89920:e703fc5262b7 parent: 89918:3b81d1b3f9d1 parent: 89919:8661ffca4581 user: Antoine Pitrou date: Sat Mar 22 18:19:36 2014 +0100 summary: Issue #20913: improve the SSL security considerations to first advocate using create_default_context(). files: Doc/library/asyncio-eventloop.rst | 2 + Doc/library/ftplib.rst | 51 +++++++----------- Doc/library/http.client.rst | 38 +++++++------ Doc/library/imaplib.rst | 29 ++++++---- Doc/library/nntplib.rst | 16 +++-- Doc/library/poplib.rst | 29 ++++++---- Doc/library/smtplib.rst | 27 ++++----- Doc/library/ssl.rst | 42 +++++++++++++-- 8 files changed, 133 insertions(+), 101 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -241,6 +241,8 @@ the transport; if *ssl* is :const:`True`, a context with some unspecified default settings is used. + .. seealso:: :ref:`SSL/TLS security considerations ` + * *server_hostname*, is only for use together with *ssl*, and sets or overrides the hostname that the target server's certificate will be matched against. By default the value of the *host* argument diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -80,14 +80,14 @@ :rfc:`4217`. Connect as usual to port 21 implicitly securing the FTP control connection before authenticating. Securing the data connection requires the user to - explicitly ask for it by calling the :meth:`prot_p` method. - *keyfile* and *certfile* are optional -- they can contain a PEM formatted - private key and certificate chain file name for the SSL connection. - *context* parameter is a :class:`ssl.SSLContext` object which allows - bundling SSL configuration options, certificates and private keys into a - single (potentially long-lived) structure. *source_address* is a 2-tuple - ``(host, port)`` for the socket to bind to as its source address before - connecting. + explicitly ask for it by calling the :meth:`prot_p` method. *context* + is a :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially + long-lived) structure. Please read :ref:`ssl-security` for best practices. + + *keyfile* and *certfile* are a legacy alternative to *context* -- they + can point to PEM-formatted private key and certificate chain files + (respectively) for the SSL connection. .. versionadded:: 3.2 @@ -96,29 +96,18 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). - Here's a sample session using the :class:`FTP_TLS` class: + Here's a sample session using the :class:`FTP_TLS` class:: - >>> from ftplib import FTP_TLS - >>> ftps = FTP_TLS('ftp.python.org') - >>> ftps.login() # login anonymously before securing control channel - >>> ftps.prot_p() # switch to secure data connection - >>> ftps.retrlines('LIST') # list directory content securely - total 9 - drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . - drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. - drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin - drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc - d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming - drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib - drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub - drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr - -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg - '226 Transfer complete.' - >>> ftps.quit() - >>> + >>> ftps = FTP_TLS('ftp.pureftpd.org') + >>> ftps.login() + '230 Anonymous user logged in' + >>> ftps.prot_p() + '200 Data protection level set to "private"' + >>> ftps.nlst() + ['6jack', 'OpenBSD', 'antilink', 'blogbench', 'bsdcam', 'clockspeed', 'djbdns-jedi', 'docs', 'eaccelerator-jedi', 'favicon.ico', 'francotone', 'fugu', 'ignore', 'libpuzzle', 'metalog', 'minidentd', 'misc', 'mysql-udf-global-user-variables', 'php-jenkins-hash', 'php-skein-hash', 'php-webdav', 'phpaudit', 'phpbench', 'pincaster', 'ping', 'posto', 'pub', 'public', 'public_keys', 'pure-ftpd', 'qscan', 'qtc', 'sharedance', 'skycache', 'sound', 'tmp', 'ucarp'] .. exception:: error_reply @@ -434,8 +423,8 @@ .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: FTP_TLS.ccc() diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -43,10 +43,10 @@ For example, the following calls all create instances that connect to the server at the same host and port:: - >>> h1 = http.client.HTTPConnection('www.cwi.nl') - >>> h2 = http.client.HTTPConnection('www.cwi.nl:80') - >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80) - >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80, timeout=10) + >>> h1 = http.client.HTTPConnection('www.python.org') + >>> h2 = http.client.HTTPConnection('www.python.org:80') + >>> h3 = http.client.HTTPConnection('www.python.org', 80) + >>> h4 = http.client.HTTPConnection('www.python.org', 80, timeout=10) .. versionchanged:: 3.2 *source_address* was added. @@ -64,23 +64,27 @@ A subclass of :class:`HTTPConnection` that uses SSL for communication with secure servers. Default port is ``443``. If *context* is specified, it must be a :class:`ssl.SSLContext` instance describing the various SSL - options. If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. + options. *key_file* and *cert_file* are deprecated, please use - :meth:`ssl.SSLContext.load_cert_chain` instead. + :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. - If you access arbitrary hosts on the Internet, it is recommended to - require certificate checking and feed the *context* with a set of - trusted CA certificates:: + The recommended way to connect to HTTPS hosts on the Internet is as + follows:: - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) - context.verify_mode = ssl.CERT_REQUIRED - context.load_verify_locations('/etc/pki/tls/certs/ca-bundle.crt') - h = client.HTTPSConnection('svn.python.org', 443, context=context) + context = ssl.create_default_context() + h = client.HTTPSConnection('www.python.org', 443, context=context) + + Please read :ref:`ssl-security` for more information on best practices. + + .. note:: + If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` + of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then + by default *host* is matched against the host name(s) allowed by the + server's certificate. If you want to change that behaviour, you can + explicitly set *check_hostname* to False. .. versionchanged:: 3.2 *source_address*, *context* and *check_hostname* were added. diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -69,21 +69,25 @@ This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled with SSL support). If *host* is not specified, ``''`` (the local host) is used. - If *port* is omitted, the standard IMAP4-over-SSL port (993) is used. *keyfile* - and *certfile* are also optional - they can contain a PEM formatted private key - and certificate chain file for the SSL connection. *ssl_context* parameter is a - :class:`ssl.SSLContext` object which allows bundling SSL configuration - options, certificates and private keys into a single (potentially long-lived) - structure. Note that the *keyfile*/*certfile* parameters are mutually exclusive with *ssl_context*, - a :class:`ValueError` is raised if *keyfile*/*certfile* is provided along with *ssl_context*. + If *port* is omitted, the standard IMAP4-over-SSL port (993) is used. + *ssl_context* is a :class:`ssl.SSLContext` object which allows bundling + SSL configuration options, certificates and private keys into a single + (potentially long-lived) structure. Please read :ref:`ssl-security` for + best practices. + + *keyfile* and *certfile* are a legacy alternative to *ssl_context* - they + can point to PEM-formatted private key and certificate chain files for + the SSL connection. Note that the *keyfile*/*certfile* parameters are + mutually exclusive with *ssl_context*, a :class:`ValueError` is raised + if *keyfile*/*certfile* is provided along with *ssl_context*. .. versionchanged:: 3.3 *ssl_context* parameter added. .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). The second subclass allows for connections created by a child process: @@ -437,14 +441,15 @@ Send a ``STARTTLS`` command. The *ssl_context* argument is optional and should be a :class:`ssl.SSLContext` object. This will enable - encryption on the IMAP connection. + encryption on the IMAP connection. Please read :ref:`ssl-security` for + best practices. .. versionadded:: 3.2 .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: IMAP4.status(mailbox, names) diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -94,6 +94,7 @@ port *port*. :class:`NNTP_SSL` objects have the same methods as :class:`NNTP` objects. If *port* is omitted, port 563 (NNTPS) is used. *ssl_context* is also optional, and is a :class:`~ssl.SSLContext` object. + Please read :ref:`ssl-security` for best practices. All other parameters behave the same as for :class:`NNTP`. Note that SSL-on-563 is discouraged per :rfc:`4642`, in favor of @@ -104,8 +105,8 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. exception:: NNTPError @@ -234,9 +235,10 @@ .. method:: NNTP.starttls(ssl_context=None) - Send a ``STARTTLS`` command. The *ssl_context* argument is optional - and should be a :class:`ssl.SSLContext` object. This will enable - encryption on the NNTP connection. + Send a ``STARTTLS`` command. This will enable encryption on the NNTP + connection. The *ssl_context* argument is optional and should be a + :class:`ssl.SSLContext` object. Please read :ref:`ssl-security` for best + practices. Note that this may not be done after authentication information has been transmitted, and authentication occurs by default if possible during a @@ -247,8 +249,8 @@ .. versionchanged:: 3.4 The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. method:: NNTP.newgroups(date, *, file=None) diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -43,20 +43,23 @@ This is a subclass of :class:`POP3` that connects to the server over an SSL encrypted socket. If *port* is not specified, 995, the standard POP3-over-SSL - port is used. *keyfile* and *certfile* are also optional - they can contain a - PEM formatted private key and certificate chain file for the SSL connection. - *timeout* works as in the :class:`POP3` constructor. *context* parameter is a - :class:`ssl.SSLContext` object which allows bundling SSL configuration - options, certificates and private keys into a single (potentially long-lived) - structure. + port is used. *timeout* works as in the :class:`POP3` constructor. + *context* is an optional :class:`ssl.SSLContext` object which allows + bundling SSL configuration options, certificates and private keys into a + single (potentially long-lived) structure. Please read :ref:`ssl-security` + for best practices. + + *keyfile* and *certfile* are a legacy alternative to *context* - they can + point to PEM-formatted private key and certificate chain files, + respectively, for the SSL connection. .. versionchanged:: 3.2 *context* parameter added. .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). One exception is defined as an attribute of the :mod:`poplib` module: @@ -198,10 +201,12 @@ *context* parameter is a :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into - a single (potentially long-lived) structure. This method supports - hostname checking via :attr:`SSLContext.check_hostname` - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + a single (potentially long-lived) structure. Please read :ref:`ssl-security` + for best practices. + + This method supports hostname checking via + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. versionadded:: 3.4 diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -69,20 +69,15 @@ required from the beginning of the connection and using :meth:`starttls` is not appropriate. If *host* is not specified, the local host is used. If *port* is zero, the standard SMTP-over-SSL port (465) is used. The optional - arguments *local_hostname* and *source_address* have the same meaning as - they do in the :class:`SMTP` class. *keyfile* and *certfile* are also - optional, and can contain a PEM formatted private key and certificate chain - file for the SSL connection. *context* also optional, can contain a - SSLContext, and is an alternative to keyfile and certfile; If it is - specified both keyfile and certfile must be None. The optional *timeout* - parameter specifies a timeout in seconds for blocking operations like the - connection attempt (if not specified, the global default timeout setting - will be used). The optional source_address parameter allows to bind to some - specific source address in a machine with multiple network interfaces, - and/or to some specific source tcp port. It takes a 2-tuple (host, port), - for the socket to bind to as its source address before connecting. If - omitted (or if host or port are ``''`` and/or 0 respectively) the OS default - behavior will be used. + arguments *local_hostname*, *timeout* and *source_address* have the same + meaning as they do in the :class:`SMTP` class. *context*, also optional, + can contain a :class:`~ssl.SSLContext` and allows to configure various + aspects of the secure connection. Please read :ref:`ssl-security` for + best practices. + + *keyfile* and *certfile* are a legacy alternative to *context*, and can + point to a PEM formatted private key and certificate chain file for the + SSL connection. .. versionchanged:: 3.3 *context* was added. @@ -92,8 +87,8 @@ .. versionchanged:: 3.4 The class now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). + :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see + :data:`ssl.HAS_SNI`). .. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1542,7 +1542,7 @@ import socket, ssl - context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile") bindsocket = socket.socket() @@ -1619,9 +1619,39 @@ Security considerations ----------------------- +Best defaults +^^^^^^^^^^^^^ + +For **client use**, if you don't have any special requirements for your +security policy, it is highly recommended that you use the +:func:`create_default_context` function to create your SSL context. +It will load the system's trusted CA certificates, enable certificate +validation, and try to choose reasonably secure protocol and cipher settings. + +For example, here is how you would use the :class:`smtplib.SMTP` class to +create a trusted, secure connection to a SMTP server:: + + >>> import ssl, smtplib + >>> smtp = smtplib.SMTP("mail.python.org", port=587) + >>> context = ssl.create_default_context() + >>> smtp.starttls(context=context) + (220, b'2.0.0 Ready to start TLS') + +If a client certificate is needed for the connection, it can be added with +:meth:`SSLContext.load_cert_chain`. + +By contrast, if you create the SSL context by calling the :class:`SSLContext` +constructor yourself, it will not have certificate validation enabled by +default. If you do so, please read the paragraphs below to achieve a good +security level. + +Manual settings +^^^^^^^^^^^^^^^ + Verifying certificates -^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''' +When calling the the :class:`SSLContext` constructor directly, :const:`CERT_NONE` is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you're talking to. @@ -1645,7 +1675,7 @@ by default). Protocol versions -^^^^^^^^^^^^^^^^^ +''''''''''''''''' SSL version 2 is considered insecure and is therefore dangerous to use. If you want maximum compatibility between clients and servers, it is recommended @@ -1655,11 +1685,11 @@ context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.options |= ssl.OP_NO_SSLv2 -The SSL context created above will allow SSLv3 and TLSv1 connections, but -not SSLv2. +The SSL context created above will allow SSLv3 and TLSv1 (and later, if +supported by your system) connections, but not SSLv2. Cipher selection -^^^^^^^^^^^^^^^^ +'''''''''''''''' If you have advanced security requirements, fine-tuning of the ciphers enabled when negotiating a SSL session is possible through the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 20:39:31 2014 From: python-checkins at python.org (georg.brandl) Date: Sat, 22 Mar 2014 20:39:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMDk3?= =?utf-8?q?5=3A_make_date_in_the_interpreter_banner_a_little_more_consiste?= =?utf-8?q?nt?= Message-ID: <3frscg2vskz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/4476b7493ee4 changeset: 89921:4476b7493ee4 branch: 3.4 parent: 89919:8661ffca4581 user: Georg Brandl date: Sat Mar 22 20:38:11 2014 +0100 summary: Closes #20975: make date in the interpreter banner a little more consistent files: Doc/tutorial/interpreter.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -96,8 +96,8 @@ before printing the first prompt:: $ python3.4 - Python 3.4 (default, Sep 24 2012, 09:25:04) - [GCC 4.6.3] on linux2 + Python 3.4 (default, Mar 16 2014, 09:25:04) + [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 20:39:32 2014 From: python-checkins at python.org (georg.brandl) Date: Sat, 22 Mar 2014 20:39:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3frsch4J4vz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/802ed5baffb3 changeset: 89922:802ed5baffb3 parent: 89920:e703fc5262b7 parent: 89921:4476b7493ee4 user: Georg Brandl date: Sat Mar 22 20:39:08 2014 +0100 summary: merge with 3.4 files: Doc/tutorial/interpreter.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -96,8 +96,8 @@ before printing the first prompt:: $ python3.5 - Python 3.5 (default, Sep 24 2012, 09:25:04) - [GCC 4.6.3] on linux2 + Python 3.5 (default, Sep 16 2015, 09:25:04) + [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 22 20:40:10 2014 From: python-checkins at python.org (georg.brandl) Date: Sat, 22 Mar 2014 20:40:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_small_additional_job_in_the_t?= =?utf-8?q?utorial?= Message-ID: <3frsdQ6s6Mz7LjV@mail.python.org> http://hg.python.org/peps/rev/a60776d88718 changeset: 5419:a60776d88718 user: Georg Brandl date: Sat Mar 22 20:40:03 2014 +0100 summary: small additional job in the tutorial files: pep-0101.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -246,7 +246,7 @@ Misc/HISTORY. ___ Doc/tutorial/interpreter.rst (2 references to '[Pp]ython32', one - to 'Python 3.2'). + to 'Python 3.2', also make the date in the banner consistent). ___ Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which have each one reference to '[Pp]ython32'. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Mar 22 21:53:51 2014 From: python-checkins at python.org (nick.coghlan) Date: Sat, 22 Mar 2014 21:53:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_Network_security_e?= =?utf-8?q?nhancement_exception?= Message-ID: <3frvGR5tWyz7LjP@mail.python.org> http://hg.python.org/peps/rev/78a067318208 changeset: 5420:78a067318208 user: Nick Coghlan date: Sun Mar 23 06:53:40 2014 +1000 summary: PEP 466: Network security enhancement exception files: pep-0466.txt | 283 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 283 insertions(+), 0 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt new file mode 100644 --- /dev/null +++ b/pep-0466.txt @@ -0,0 +1,283 @@ +PEP: 466 +Title: Network Security Enhancement Exception for All Branches +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan , +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 23-Mar-2014 +Post-History: 23-Mar-2014 + + +Abstract +======== + +Most CPython tracker issues are classified as errors in behaviour or +proposed enhancements. Most patches to fix behavioural errors are +applied to all active maintenance branches. Enhancement patches are +restricted to the default branch that becomes the next Python version. + +This PEP relaxes the latter restriction allowing enhancements to be applied +to maintenance branches for standard library components that have +implications for the overall security of the internet. In particular, the +exception will apply to: + +* the ``ssl`` module +* the ``hashlib`` module +* the ``hmac`` module +* the ``sha`` module (Python 2 only) +* the components of other networking modules that make use of these modules +* the components of the ``random`` and ``os`` modules that are relevant to + cryptographic applications +* the version of OpenSSL bundled with the binary installers + +Changes to these modules will still need to undergo normal backwards +compatibility assessments, but new features will be permitted where +appropriate, making it easier to implement secure networked software in +Python, even for software that needs to remain compatible with older feature +releases of Python. + + +Exemption Policy +================ + +Under this policy, the following network security related modules are +granted a blanket exemption to the restriction against adding new features +in maintenance releases: + +* the ``ssl`` module +* the ``hashlib`` module +* the ``hmac`` module +* the ``sha`` module (Python 2 only) + +This exemption applies to *all* proposals to backport backwards compatible +features in these modules to active maintenance branches. This choice is +made deliberately to ensure that the "feature or fix?" argument isn't simply +replaced by a "security related or not?" argument. These particular modules +are inherently security related, and all enhancements to them improve +Python's capabilities as a platform for development of secure networked +software. + +As part of this policy, permission is also granted to upgrade to newer +feature releases of OpenSSL when preparing the binary installers +for new maintenance releases of CPython. + +In addition to the above blanket exemption, a conditional exemption is +granted for these modules that may include some network security related +features: + +* the ``os`` module (primarily ``os.urandom``) +* the ``random`` module +* networking related modules that depend on one or more of the network + security related modules listed above + +This more limited exemption for these modules requires that the *specific* +enhancement being proposed for backporting needs to be justified as being +network security related. If the enhancement under discussion is designed +to take advantage of a new feature in one of the network security related +modules, then that will be taken as implying that the enhancement is +security related. + + +Backwards Compatibility Considerations +====================================== + +This PEP does *not* grant any general exemptions to the usual backwards +compatibility policy for maintenance releases. Instead, it is designed +to make it easier to provide more "secure by default" behaviour in future +feature releases, while still limiting the risk of breaking currently +working software when upgrading to a new maintenance release. + +In *all* cases where this policy is applied to backport enhancements to +maintenance releases, it MUST be possible to write cross-version compatible +code that operates by "feature detection" (for example, checking for +particular attributes in the module), without needing to explicitly check +the Python version. + +It is then up to library and framework code to provide an appropriate error +message or fallback behaviour if a desired feature is found to be missing. + +Affected APIs SHOULD be designed to allow library and application code to +perform the following actions after detecting the presence of a relevant +network security related feature: + +* explicitly opt in to more secure settings (to allow the use of enhanced + security features in older versions of Python) +* explicitly opt in to less secure settings (to allow the use of newer Python + versions in lower security environments) +* determine the default setting for the feature (this MAY require explicit + Python version checks to determine the Python feature release, but MUST + NOT depend on the specific maintenance release) + + +Documentation Requirements +========================== + +All modules that take advantage of this policy to backport network +security related enhancements to earlier Python versions MUST include +a "Security Considerations" section in their documentation. + +In addition to any other module specific contents, this section MUST +enumerate key security enhancements and fixes (with CVE identifiers where +applicable), and the + + +Evolution of this Policy +======================== + +The key requirement for a module to be considered for inclusion in this +policy (whether under a blanket or conditional exemption) is that it must +have security implications *beyond* the specific application that is written +in Python and the system that application is running on. Thus the focus on +network security protocols and related cryptographic infrastructure - Python +is a popular choice for the development of web services and clients, and +thus the capabilities of widely used Python versions have implications for +the security design of other services that may be using newer versions of +Python or other development languages. + +The intent behind this requirement is to minimise any impact that the +introduction of this policy may have on the stability and compatibility of +maintenance releases. It would be thoroughly counterproductive if end +users became as cautious about updating to new Python maintenance releases +as they are about updating to new feature releases. + + +Motivation and Rationale +======================== + +This PEP can be seen as a more targeted version of the "faster standard +library release cycle" proposals discussed in PEP 407 and PEP 413, +focusing specifically on those areas which have implications beyond the +Python community. + +The creation of this PEP was prompted primarily by the aging SSL support in +the Python 2 series. As of March 2014, the Python 2.7 SSL module is +approaching four years of age, and the SSL support in the still popular +Python 2.6 release had its feature set locked six years ago. + +These are simply too old to provide a foundation that can be recommended +in good conscience for secure networking software that operates over the +public internet, especially in an era where it is becoming quite clearly +evident that advanced persistent security threats are even more widespread +and more indiscriminate in their targeting than had previously been +understood. + +While the use of the system OpenSSL installation addresses many of these +concerns on Linux platforms, it doesn't address all of them, and in the +case of the binary installers for Windows and Mac OS X that are published +on python.org, the version of OpenSSL used is entirely within the control +of the Python core development team. + +With increased popularity comes increased responsibility, and this policy +is about recognising the fact that Python's popularity and adoption has now +reached a level where some of our design and policy decisions have +significant implications beyond the Python development community. + +As one example, the Python 2 ``ssl`` module does not support the Server +Name Identification standard. While it is possible to obtain SNI support +by using the third party ``requests`` client library, actually doing so +currently requires using not only ``requests`` and its embedded dependencies, +but also half a dozen or more additional libraries. The lack of support +in the Python 2 series thus serves as an impediment to making effective +use of SNI on servers, as Python 2 clients will frequently fail to handle +it correctly. + +Another more critical example is the lack of SSL hostname matching in the +Python 2 standard library - it is currently necessary to rely on a third +party library, such as ``requests`` or ``backports.ssl_match_hostname`` to +obtain that functionality in Python 2. + +The Python 2 series also remains more vulnerable to remote timing attacks +on security sensitive comparisons than the Python 3 series, as it lacks a +standard library equivalent to the timing attack resistant +``hmac.compare_digest()`` function. While appropriate secure comparison +functions can be implemented in third party extensions, may users don't +even consider the problem and use ordinary equality comparisons instead +- while a standard library solution doesn't automatically fix that problem, +it *does* make the barrier to resolution much lower once the problem is +pointed out. + +My position on the ongoing transition from Python 2 to Python 3 has long +been that Python 2 remains a supported platform for the core development +team, and that commercial support will remain available well after upstream +maintenance ends. However, in the absence of this network security +enhancement policy, that position is difficult to justify when it comes to +software that operates over the public internet. Just as many developers +consider it too difficult to develop truly secure modern networked software +in C/C++ (largely due to the challenges associated with manual +memory management), I consider it too difficult to develop truly secure +modern networked software using the Python 2 series without introducing this +network security enhancement policy, as doing so would mean reimplementing +substantial portions of the standard library as third party modules to gain +access to the newer underlying network security protocols and mechanisms, +and then injecting those into the module namespace to override their +standard library counterparts. + +Requiring that latent defects in an application's Unicode correctness be +addressed in order to migrate to Python 3 is not a reasonable alternative +recommendation, especially given the likely existence of legacy code that +lacks the kind of automated regression test suite needed to help support +a migration from Python 2 to Python 3. The key point of this PEP is that +those situations affect more people than just the developers and users of +the affected application: their existence becomes something that developers +of secure networked services need to take into account as part of their +security design. By making it more feasible to enhance the security of the +Python 2 series, we can help contribute to the evolution of a more secure +internet for all concerned. + +For the Python 2 series, this proposal will most obviously impact the +remaining maintenance releases of Python 2.7. However, commercial +redistributors that continue to offer full support for Python 2.6, and will +also do so for Python 2.7 after upstream maintenance ends, may choose +to take this policy into account when deciding what changes to backport to +their own maintenance updates. Providing commercial redistributors such +as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and +Red Hat that option is one of the benefits offered by requiring feature +detection based forward compatibility for the modules covered by this policy. + + +Open Questions +============== + +* What are the risks associated with allowing OpenSSL to be updated to + new feature versions in the Windows and Mac OS X binary installers for + maintenance releases? + +* Are there any other security relevant modules that should be covered + by either a blanket or conditional exemption? + +* Should we enumerate a specific list of "other networking modules" rather + than leaving it implicit? + + +Acknowledgement +=============== + +Thanks to Christian Heimes for his recent efforts on greatly improving +Python's SSL support in the Python 3 series, and a variety of members of +the Python community for helping me to better understand the implications +of the default settings we provide in our SSL modules, and the impact that +tolerating the use of SSL infrastructure that was defined in 2010 +(Python 2.7) or even 2008 (Python 2.6) potentially has for the security +of the web as a whole. + +Christian and Donald Stufft also provided valuable feedback on a preliminary +draft of this proposal. + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Mar 22 22:17:42 2014 From: python-checkins at python.org (nick.coghlan) Date: Sat, 22 Mar 2014 22:17:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_finish_incomplete_?= =?utf-8?q?sentence?= Message-ID: <3frvny1Y7Vz7LjP@mail.python.org> http://hg.python.org/peps/rev/6ed8f836c072 changeset: 5421:6ed8f836c072 user: Nick Coghlan date: Sun Mar 23 07:17:32 2014 +1000 summary: PEP 466: finish incomplete sentence files: pep-0466.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -120,7 +120,8 @@ In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where -applicable), and the +applicable), along with the feature and maintenance releases that first +included them. Evolution of this Policy -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 23 07:41:34 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Mar 2014 07:41:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_466_based_on_first?= =?utf-8?q?_round_of_feedback?= Message-ID: <3fs8JZ6Ch9z7Ljr@mail.python.org> http://hg.python.org/peps/rev/2e82209dda21 changeset: 5422:2e82209dda21 user: Nick Coghlan date: Sun Mar 23 16:41:24 2014 +1000 summary: Update PEP 466 based on first round of feedback files: pep-0466.txt | 327 ++++++++++++++++++++++++++++++++------ 1 files changed, 271 insertions(+), 56 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -1,5 +1,5 @@ PEP: 466 -Title: Network Security Enhancement Exception for All Branches +Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , @@ -18,10 +18,17 @@ applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. -This PEP relaxes the latter restriction allowing enhancements to be applied -to maintenance branches for standard library components that have -implications for the overall security of the internet. In particular, the -exception will apply to: +This cadence works reasonably well during Python's normal 18-24 month +feature release cycle, which is still applicable to the Python 3 series. +However, the age of the standard library in Python 2 has now reached a point +where it is sufficiently far behind the state of the art in network security +protocols for it to be causing real problems in commercial use cases +where upgrading to Python 3 in the near term may not be practical. + +Accordingly, this PEP relaxes the normal restrictions by allowing +enhancements to be applied in Python 2.7 maintenance releases for standard +library components that have implications for the overall security of the +internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module @@ -32,19 +39,29 @@ cryptographic applications * the version of OpenSSL bundled with the binary installers -Changes to these modules will still need to undergo normal backwards -compatibility assessments, but new features will be permitted where +Proposed backports for these modules will still need to undergo normal +backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. +While this PEP does not make any changes to the core development team's +handling of security-fix-only branches that are no longer in active +maintenance, it *does* recommend that commercial redistributors providing +extended support periods for the Python standard library either adopt a +similar approach to ensuring that the secure networking infrastructure +keeps pace with the evolution of the internet, or else disclaim support +for the use of older versions in roles that involving connecting +directly to the public internet. + Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features -in maintenance releases: +in maintenance releases, for the purpose of keeping their APIs aligned with +their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module @@ -52,7 +69,7 @@ * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible -features in these modules to active maintenance branches. This choice is +changes in these modules to Python 2.7 maintenance releases. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve @@ -67,7 +84,7 @@ granted for these modules that may include some network security related features: -* the ``os`` module (primarily ``os.urandom``) +* the ``os`` module (specifically ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above @@ -84,10 +101,12 @@ ====================================== This PEP does *not* grant any general exemptions to the usual backwards -compatibility policy for maintenance releases. Instead, it is designed -to make it easier to provide more "secure by default" behaviour in future -feature releases, while still limiting the risk of breaking currently -working software when upgrading to a new maintenance release. +compatibility policy for maintenance releases. Instead, by explicitly +encouraging the use of feature based checks and explicitly opting in to +less secure configurations, it is designed to make it easier to provide +more "secure by default" behaviour in future feature releases, while still +limiting the risk of breaking currently working software when upgrading to +a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible @@ -95,24 +114,56 @@ particular attributes in the module), without needing to explicitly check the Python version. -It is then up to library and framework code to provide an appropriate error -message or fallback behaviour if a desired feature is found to be missing. +It is then up to library and framework code to provide an appropriate warning +and fallback behaviour if a desired feature is found to be missing. While +some especially security sensitive software MAY fail outright if a desired +security feature is unavailable, most software SHOULD instead continue +operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced - security features in older versions of Python) + security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python - versions in lower security environments) + feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST - NOT depend on the specific maintenance release) + NOT require checking for a specific maintenance release) +Security related changes to other modules (such as data format processing +libraries) will continue to be made available as backports and new modules +on the Python Package Index, as independent distribution remains the +preferred approach to handling software that needs to evolve faster than +the standard library. Refer to the `Motivation and Rationale`_ section for +a review of the characteristics that make the secure networking +infrastructure worthy of special consideration. -Documentation Requirements -========================== + +Other Considerations +==================== + +Maintainability +--------------- + +This policy does NOT represent a commitment by volunteer contributors to +actually backport network security related changes from the Python 3 series +to the Python 2 series. Rather, it is intended to send a clear signal to +potential corporate contributors that the core development team are willing +to review and merge corporate contributions that put this policy into +effect. + +Backporting security related fixes and enhancements to earlier versions is +a common service for commercial redistributors to offer to their customers. +This policy represents an explicit invitation to contribute some of those +changes back to the upstream community in cases where they are likely to +have a broad impact that helps improve the security of the internet as a +whole. + + +Documentation +------------- All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include @@ -124,6 +175,33 @@ included them. +Security releases +----------------- + +This PEP does not propose any changes to the handling of security +releases - those will continue to be source only releases that +include only critical security fixes. + +However, the recommendations for library and application developers are +deliberately designed to accommodate commercial redistributors applying +this policy to any Python release series that is either in security +fix only mode, or has been declared "end of life" by the core development +team. + +Whether or not redistributors choose to exercise that option will be up +to the redistributor. + + +Integration testing +------------------- + +Third party integration testing services would likely need to start +offering users a choice of multiple Python 2.7.x versions to test against, +to ensure that the application is correctly degrading gracefully if it +attempts to use newer networking features on maintenance releases that +are too old to provide them. + + Evolution of this Policy ======================== @@ -152,6 +230,10 @@ focusing specifically on those areas which have implications beyond the Python community. + +Background +---------- + The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular @@ -162,16 +244,22 @@ public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been -understood. +understood. While they represented reasonable security infrastructure in +their time, the state of the art has moved on, and we need to investigate +mechanisms for effectively providing more up to date network security +infrastructure for users that, for whatever reason, are not currently in +a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control -of the Python core development team. +of the Python core development team, and currently limited to OpenSSL +maintenance releases for the version initially shipped with the corresponding +Python feature release. With increased popularity comes increased responsibility, and this policy -is about recognising the fact that Python's popularity and adoption has now +aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. @@ -207,35 +295,144 @@ software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual -memory management), I consider it too difficult to develop truly secure -modern networked software using the Python 2 series without introducing this -network security enhancement policy, as doing so would mean reimplementing -substantial portions of the standard library as third party modules to gain -access to the newer underlying network security protocols and mechanisms, -and then injecting those into the module namespace to override their -standard library counterparts. +memory management), I anticipate that in the not too distant future, it +will be considered too difficult to develop truly secure modern networked +software using the Python 2 series (some developers would argue that we +have already reached that point). -Requiring that latent defects in an application's Unicode correctness be -addressed in order to migrate to Python 3 is not a reasonable alternative -recommendation, especially given the likely existence of legacy code that -lacks the kind of automated regression test suite needed to help support -a migration from Python 2 to Python 3. The key point of this PEP is that -those situations affect more people than just the developers and users of -the affected application: their existence becomes something that developers -of secure networked services need to take into account as part of their -security design. By making it more feasible to enhance the security of the -Python 2 series, we can help contribute to the evolution of a more secure -internet for all concerned. -For the Python 2 series, this proposal will most obviously impact the -remaining maintenance releases of Python 2.7. However, commercial -redistributors that continue to offer full support for Python 2.6, and will -also do so for Python 2.7 after upstream maintenance ends, may choose -to take this policy into account when deciding what changes to backport to -their own maintenance updates. Providing commercial redistributors such -as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and -Red Hat that option is one of the benefits offered by requiring feature -detection based forward compatibility for the modules covered by this policy. +Alternative: advise developers of networked software to migrate to Python 3 +--------------------------------------------------------------------------- + +This alternative represents the status quo. Unfortunately, it has proven +to be unworkable in practice, as the backwards compatibility implications +mean that this is a non-trivial migration process for large applications +and integration projects. + +Now that we're fully aware of the impact the limitations in Python 2 may be +having on the evolution of internet security standards, I no longer believe +that it is reasonable to expect platform and application developers to +resolve all of the latent defects in an application's Unicode correctness +solely in order to gain access to the network security enhancements +available in Python 3. + +While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 +as its main Python interpreter for its 14.04 LTS release, Fedora still +has a lot of work to do to migrate, and it will take a non-trivial amount +of time to migrate the relevant infrastructure components. While Red Hat +are also actively working to make it easier for users to use more recent +versions of Python on our stable platforms, it's going to take time for +those efforts to start having an impact on end users' choice of version, +and those changes won't affect the core tools regardless. + +The OpenStack migration to Python 3 is also still in its infancy, and even +though that's a project with an extensive and relatively robust automated +test suite, it's large enough that it is going to take quite some time +to migrate. + +And that's just three of the highest profile open source projects that +make heavy use of Python. Given the likely existence of large amounts of +legacy code that lacks the kind of automated regression test suite needed +to help support a migration from Python 2 to Python 3. The key point of +this PEP is that those situations affect more people than just the +developers and users of the affected application: their existence becomes +something that developers of secure networked services need to take into +account as part of their security design. + +As Terry Reedy noted, if we try to persist with the status quo, the likely +outcome is that commercial redistributors will attempt to do something +like this on behalf of their customers *anyway*, but in a potentially +inconsistent and ad hoc manner. By drawing the scope definition process +into the upstream project we are in a better position to influence the +approach taken to address the situation and to help ensure some consistency +across redistributors. + +The problem is real, so *something* needs to change, and this PEP describes +my currently preferred approach to addressing the situation. + + +Alternative: create and release Python 2.8 +------------------------------------------ + +With sufficient corporate support, it likely *would* be possible to create +and release Python 2.8 (it's highly unlikely such a project would garner +enough interest to be achievable with only volunteers). However, this +wouldn't actually solve the problem, as the aim is to provide a *relatively +low impact* way to incorporate enhanced security features into integrated +products and deployments that make use of Python 2. Upgrading to a new +Python feature release would mean both more work for the core development +team, as well as a more disruptive update that most potential end users +would likely just skip entirely. + +Attempting to create a Python 2.8 release would also bring in suggestions +to backport many additional features from Python 3 (such as ``tracemalloc`` +and the improved coroutine support). + +This is not a recommended approach, as it would involve substantial +additional work for a result that is actually less effective as a solution +to the original problem (the widespread use of the aging network security +infrastructure in Python 2). + + +Alternative: distribute the security enhancements via PyPI +---------------------------------------------------------- + +While it initially appears to be an attractive and easier to manage +approach, there are actually several significant problems with this +idea. + +Firstly, this PEP encompasses a non-trivial portion of the standard library. +It's not just the underlying SSL support, but also the libraries for other +network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the +SSL infrastructure to provide secure links, and that's just the protocols +in the standard library. Even if an API compatible ``ssl2`` module was +made available, it would need to be imported and injected +into ``sys.modules`` as ``ssl`` before importing any other module that +needed it. + +Secondly, this is complex, low level, cross-platform code that integrates +with the underlying operating system across a variety of POSIX platforms +(including Mac OS X) and Windows. The CPython BuildBot fleet is already set +up to handle continuous integration in that context, but most of the +freely available continuous integration services just offer Linux, and +perhaps paid access to Windows. Those services work reasonably well for +software that largely runs on the abstraction layers offered by Python and +other dynamic languages, but won't suffice for the kind of code involved +here. + +The OpenSSL dependency for the network security support also qualifies as +the kind of "complex binary dependency" that isn't yet handled well by the +``pip`` based software distribution ecosystem. Relying on a binary +dependency also creates potential compatibility problems for ``pip`` when +running on other interpreters like ``PyPy``. + +Another practical problem with the idea is the fact that ``pip`` itself +relies on the ``ssl`` support in the standard library (with some additional +support from a bundled copy of ``requests``, which in turn bundles +``backport.ssl_match_hostname``), and hence would require any replacement +module to also be bundled within ``pip``. This wouldn't pose any +insurmountable difficulties (it's just another dependency to vendor), but +it *would* mean yet another copy of OpenSSL to keep up to +date. + +This approach also has the same flaw as all other "improve security by +renaming things" approaches: they completely miss the users who most need +help, and raise significant barriers against being able to encourage users +to do the right thing when their infrastructure supports it (since +"use this other module" is a much higher impact change than "turn on this +higher security setting"). Deprecating the aging SSL infrastructure in the +standard library in favour of an external module would be even more user +hostile than taking the risk of trying to upgrade it in place. + +Last, but certainly not least, this approach suffers from the same problem +as the idea of doing a Python 2.8 release: likely not solving the actual +problem. Commercial redistributors of Python are set up to redistribute +*Python*, and a pre-existing set of additional packages. Getting new +packages added to the pre-existing set *can* be done, but means going +around to each and every redistributor and asking them to update their +repackaging process accordingly. By contrast, the approach described in +this PEP would require redistributors to *opt out* of the security +enhancements, which most of them are unlikely to do. Open Questions @@ -243,17 +440,27 @@ * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for - maintenance releases? + maintenance releases? Currently we just upgrade to the appropriate + OpenSSL maintenance releases, rather than switching to the latest + feature release. In particular, is it possible Windows C extensions may + be linking against the Python provided OpenSSL module? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? -* Should we enumerate a specific list of "other networking modules" rather - than leaving it implicit? +Disclosure of Interest +====================== -Acknowledgement -=============== +The author of this PEP currently works for Red Hat on test automation tools. +If this proposal is accepted, I will be strongly encouraging Red Hat to take +advantage of the resulting opportunity to help improve the overall security +of the Python ecosystem. However, I do not speak for Red Hat in this matter, +and cannot make any commitments on Red Hat's behalf. + + +Acknowledgements +================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of @@ -266,6 +473,14 @@ Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. +Thanks also to participants in the python-dev mailing list thread [1]_ + + +References +========== + +.. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html + Copyright ========= -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Sun Mar 23 10:03:53 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 23 Mar 2014 10:03:53 +0100 Subject: [Python-checkins] Daily reference leaks (802ed5baffb3): sum=0 Message-ID: results for 802ed5baffb3 on branch "default" -------------------------------------------- test_site leaked [0, -2, 2] references, sum=0 test_site leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog68Oo_f', '-x'] From python-checkins at python.org Sun Mar 23 12:59:22 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 12:59:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTkw?= =?utf-8?q?=3A_Fix_issues_found_by_pyflakes_for_multiprocessing=2E?= Message-ID: <3fsHMG5F7Bz7LjM@mail.python.org> http://hg.python.org/cpython/rev/619331c67638 changeset: 89923:619331c67638 branch: 3.4 parent: 89921:4476b7493ee4 user: Richard Oudkerk date: Sun Mar 23 11:54:15 2014 +0000 summary: Issue #20990: Fix issues found by pyflakes for multiprocessing. files: Lib/multiprocessing/spawn.py | 12 ++++++++++-- Lib/multiprocessing/synchronize.py | 7 ++++--- Misc/NEWS | 2 ++ 3 files changed, 16 insertions(+), 5 deletions(-) diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -64,7 +64,14 @@ Run code for process object if this in not the main process ''' if is_forking(sys.argv): - main() + kwds = {} + for arg in sys.argv[2:]: + name, value = arg.split('=') + if value == 'None': + kwds[name] = None + else: + kwds[name] = int(value) + spawn_main(**kwds) sys.exit() @@ -73,7 +80,8 @@ Returns prefix of command line used for spawning a child process ''' if getattr(sys, 'frozen', False): - return [sys.executable, '--multiprocessing-fork'] + tmp = ' '.join('%s=%r' % item for item in kwds.items()) + return [sys.executable, '--multiprocessing-fork'] + tmp else: prog = 'from multiprocessing.spawn import spawn_main; spawn_main(%s)' prog %= ', '.join('%s=%r' % item for item in kwds.items()) diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -49,9 +49,10 @@ _rand = tempfile._RandomNameSequence() def __init__(self, kind, value, maxvalue, *, ctx): - ctx = ctx or get_context() - ctx = ctx.get_context() - unlink_now = sys.platform == 'win32' or ctx._name == 'fork' + if ctx is None: + ctx = context._default_context.get_context() + name = ctx.get_start_method() + unlink_now = sys.platform == 'win32' or name == 'fork' for i in range(100): try: sl = self._semlock = _multiprocessing.SemLock( diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,8 @@ Library ------- +- Issue #20990: Fix issues found by pyflakes for multiprocessing. + - Issue #21015: SSL contexts will now automatically select an elliptic curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise default to "prime256v1". -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 12:59:23 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 12:59:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <3fsHMH73Wwz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/3d0eacb12b5c changeset: 89924:3d0eacb12b5c parent: 89922:802ed5baffb3 parent: 89923:619331c67638 user: Richard Oudkerk date: Sun Mar 23 11:57:01 2014 +0000 summary: Merge 3.4. files: Lib/multiprocessing/spawn.py | 12 ++++++++++-- Lib/multiprocessing/synchronize.py | 7 ++++--- Misc/NEWS | 2 ++ 3 files changed, 16 insertions(+), 5 deletions(-) diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -64,7 +64,14 @@ Run code for process object if this in not the main process ''' if is_forking(sys.argv): - main() + kwds = {} + for arg in sys.argv[2:]: + name, value = arg.split('=') + if value == 'None': + kwds[name] = None + else: + kwds[name] = int(value) + spawn_main(**kwds) sys.exit() @@ -73,7 +80,8 @@ Returns prefix of command line used for spawning a child process ''' if getattr(sys, 'frozen', False): - return [sys.executable, '--multiprocessing-fork'] + tmp = ' '.join('%s=%r' % item for item in kwds.items()) + return [sys.executable, '--multiprocessing-fork'] + tmp else: prog = 'from multiprocessing.spawn import spawn_main; spawn_main(%s)' prog %= ', '.join('%s=%r' % item for item in kwds.items()) diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -49,9 +49,10 @@ _rand = tempfile._RandomNameSequence() def __init__(self, kind, value, maxvalue, *, ctx): - ctx = ctx or get_context() - ctx = ctx.get_context() - unlink_now = sys.platform == 'win32' or ctx._name == 'fork' + if ctx is None: + ctx = context._default_context.get_context() + name = ctx.get_start_method() + unlink_now = sys.platform == 'win32' or name == 'fork' for i in range(100): try: sl = self._semlock = _multiprocessing.SemLock( diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #20990: Fix issues found by pyflakes for multiprocessing. + - Issue #21015: SSL contexts will now automatically select an elliptic curve for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise default to "prime256v1". -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 13:33:18 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 13:33:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTgw?= =?utf-8?q?=3A_Stop_wrapping_exception_when_using_ThreadPool=2E?= Message-ID: <3fsJ6Q5KnSz7LjV@mail.python.org> http://hg.python.org/cpython/rev/df6a6951b2c9 changeset: 89925:df6a6951b2c9 branch: 3.4 parent: 89923:619331c67638 user: Richard Oudkerk date: Sun Mar 23 12:30:54 2014 +0000 summary: Issue #20980: Stop wrapping exception when using ThreadPool. files: Lib/multiprocessing/managers.py | 11 ++++++++--- Lib/multiprocessing/pool.py | 12 +++++++++--- Lib/test/_test_multiprocessing.py | 11 +++++++++++ Misc/NEWS | 2 ++ 4 files changed, 30 insertions(+), 6 deletions(-) diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -1077,17 +1077,22 @@ )) -PoolProxy = MakeProxyType('PoolProxy', ( +BasePoolProxy = MakeProxyType('PoolProxy', ( 'apply', 'apply_async', 'close', 'imap', 'imap_unordered', 'join', - 'map', 'map_async', 'starmap', 'starmap_async', 'terminate' + 'map', 'map_async', 'starmap', 'starmap_async', 'terminate', )) -PoolProxy._method_to_typeid_ = { +BasePoolProxy._method_to_typeid_ = { 'apply_async': 'AsyncResult', 'map_async': 'AsyncResult', 'starmap_async': 'AsyncResult', 'imap': 'Iterator', 'imap_unordered': 'Iterator' } +class PoolProxy(BasePoolProxy): + def __enter__(self): + return self + def __exit__(self, exc_type, exc_val, exc_tb): + self.terminate() # # Definition of SyncManager diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -90,7 +90,8 @@ return "" % str(self) -def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None): +def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None, + wrap_exception=False): assert maxtasks is None or (type(maxtasks) == int and maxtasks > 0) put = outqueue.put get = inqueue.get @@ -117,7 +118,8 @@ try: result = (True, func(*args, **kwds)) except Exception as e: - e = ExceptionWithTraceback(e, e.__traceback__) + if wrap_exception: + e = ExceptionWithTraceback(e, e.__traceback__) result = (False, e) try: put((job, i, result)) @@ -137,6 +139,8 @@ ''' Class which supports an async version of applying functions to arguments. ''' + _wrap_exception = True + def Process(self, *args, **kwds): return self._ctx.Process(*args, **kwds) @@ -220,7 +224,8 @@ w = self.Process(target=worker, args=(self._inqueue, self._outqueue, self._initializer, - self._initargs, self._maxtasksperchild) + self._initargs, self._maxtasksperchild, + self._wrap_exception) ) self._pool.append(w) w.name = w.name.replace('Process', 'PoolWorker') @@ -736,6 +741,7 @@ # class ThreadPool(Pool): + _wrap_exception = False @staticmethod def Process(*args, **kwds): diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -1810,6 +1810,17 @@ self.assertIn('raise RuntimeError(123) # some comment', f1.getvalue()) + @classmethod + def _test_wrapped_exception(cls): + raise RuntimeError('foo') + + def test_wrapped_exception(self): + # Issue #20980: Should not wrap exception when using thread pool + with self.Pool(1) as p: + with self.assertRaises(RuntimeError): + p.apply(self._test_wrapped_exception) + + def raising(): raise KeyError("key") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,8 @@ Library ------- +- Issue #20980: Stop wrapping exception when using ThreadPool. + - Issue #20990: Fix issues found by pyflakes for multiprocessing. - Issue #21015: SSL contexts will now automatically select an elliptic -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 13:33:20 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 13:33:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <3fsJ6S0pRNz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/ac4e1c309d48 changeset: 89926:ac4e1c309d48 parent: 89924:3d0eacb12b5c parent: 89925:df6a6951b2c9 user: Richard Oudkerk date: Sun Mar 23 12:32:12 2014 +0000 summary: Merge 3.4. files: Lib/multiprocessing/managers.py | 11 ++++++++--- Lib/multiprocessing/pool.py | 12 +++++++++--- Lib/test/_test_multiprocessing.py | 11 +++++++++++ Misc/NEWS | 2 ++ 4 files changed, 30 insertions(+), 6 deletions(-) diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -1077,17 +1077,22 @@ )) -PoolProxy = MakeProxyType('PoolProxy', ( +BasePoolProxy = MakeProxyType('PoolProxy', ( 'apply', 'apply_async', 'close', 'imap', 'imap_unordered', 'join', - 'map', 'map_async', 'starmap', 'starmap_async', 'terminate' + 'map', 'map_async', 'starmap', 'starmap_async', 'terminate', )) -PoolProxy._method_to_typeid_ = { +BasePoolProxy._method_to_typeid_ = { 'apply_async': 'AsyncResult', 'map_async': 'AsyncResult', 'starmap_async': 'AsyncResult', 'imap': 'Iterator', 'imap_unordered': 'Iterator' } +class PoolProxy(BasePoolProxy): + def __enter__(self): + return self + def __exit__(self, exc_type, exc_val, exc_tb): + self.terminate() # # Definition of SyncManager diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -90,7 +90,8 @@ return "" % str(self) -def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None): +def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None, + wrap_exception=False): assert maxtasks is None or (type(maxtasks) == int and maxtasks > 0) put = outqueue.put get = inqueue.get @@ -117,7 +118,8 @@ try: result = (True, func(*args, **kwds)) except Exception as e: - e = ExceptionWithTraceback(e, e.__traceback__) + if wrap_exception: + e = ExceptionWithTraceback(e, e.__traceback__) result = (False, e) try: put((job, i, result)) @@ -137,6 +139,8 @@ ''' Class which supports an async version of applying functions to arguments. ''' + _wrap_exception = True + def Process(self, *args, **kwds): return self._ctx.Process(*args, **kwds) @@ -220,7 +224,8 @@ w = self.Process(target=worker, args=(self._inqueue, self._outqueue, self._initializer, - self._initargs, self._maxtasksperchild) + self._initargs, self._maxtasksperchild, + self._wrap_exception) ) self._pool.append(w) w.name = w.name.replace('Process', 'PoolWorker') @@ -736,6 +741,7 @@ # class ThreadPool(Pool): + _wrap_exception = False @staticmethod def Process(*args, **kwds): diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -1810,6 +1810,17 @@ self.assertIn('raise RuntimeError(123) # some comment', f1.getvalue()) + @classmethod + def _test_wrapped_exception(cls): + raise RuntimeError('foo') + + def test_wrapped_exception(self): + # Issue #20980: Should not wrap exception when using thread pool + with self.Pool(1) as p: + with self.assertRaises(RuntimeError): + p.apply(self._test_wrapped_exception) + + def raising(): raise KeyError("key") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #20980: Stop wrapping exception when using ThreadPool. + - Issue #20990: Fix issues found by pyflakes for multiprocessing. - Issue #21015: SSL contexts will now automatically select an elliptic -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 13:54:43 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 13:54:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwNjMz?= =?utf-8?q?=3A_Replace_relative_import_by_absolute_import=2E?= Message-ID: <3fsJb72jWGz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/0b2814fc53ae changeset: 89927:0b2814fc53ae branch: 3.3 parent: 89808:df2dbeef46da user: Richard Oudkerk date: Sun Mar 23 12:42:28 2014 +0000 summary: Issue #20633: Replace relative import by absolute import. files: Lib/multiprocessing/forking.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/forking.py b/Lib/multiprocessing/forking.py --- a/Lib/multiprocessing/forking.py +++ b/Lib/multiprocessing/forking.py @@ -132,7 +132,7 @@ def wait(self, timeout=None): if self.returncode is None: if timeout is not None: - from .connection import wait + from multiprocessing.connection import wait if not wait([self.sentinel], timeout): return None # This shouldn't block if wait() returned successfully. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,8 @@ Library ------- +- Issue #20633: Replace relative import by absolute import. + - Issue #20875: Prevent possible gzip "'read' is not defined" NameError. Patch by Claudiu Popa. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 13:54:44 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 13:54:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_Merge_3=2E3=2E?= Message-ID: <3fsJb84Ktvz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/e8942abfb829 changeset: 89928:e8942abfb829 branch: 3.4 parent: 89925:df6a6951b2c9 parent: 89927:0b2814fc53ae user: Richard Oudkerk date: Sun Mar 23 12:52:16 2014 +0000 summary: Merge 3.3. files: Lib/multiprocessing/popen_fork.py | 2 +- Lib/multiprocessing/popen_forkserver.py | 2 +- Misc/NEWS | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/popen_fork.py b/Lib/multiprocessing/popen_fork.py --- a/Lib/multiprocessing/popen_fork.py +++ b/Lib/multiprocessing/popen_fork.py @@ -47,7 +47,7 @@ def wait(self, timeout=None): if self.returncode is None: if timeout is not None: - from .connection import wait + from multiprocessing.connection import wait if not wait([self.sentinel], timeout): return None # This shouldn't block if wait() returned successfully. diff --git a/Lib/multiprocessing/popen_forkserver.py b/Lib/multiprocessing/popen_forkserver.py --- a/Lib/multiprocessing/popen_forkserver.py +++ b/Lib/multiprocessing/popen_forkserver.py @@ -57,7 +57,7 @@ def poll(self, flag=os.WNOHANG): if self.returncode is None: - from .connection import wait + from multiprocessing.connection import wait timeout = 0 if flag == os.WNOHANG else None if not wait([self.sentinel], timeout): return None diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,8 @@ Library ------- +- Issue #20633: Replace relative import by absolute import. + - Issue #20980: Stop wrapping exception when using ThreadPool. - Issue #20990: Fix issues found by pyflakes for multiprocessing. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 13:54:45 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 13:54:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <3fsJb95n84z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/227bbce969df changeset: 89929:227bbce969df parent: 89926:ac4e1c309d48 parent: 89928:e8942abfb829 user: Richard Oudkerk date: Sun Mar 23 12:53:48 2014 +0000 summary: Merge 3.4. files: Lib/multiprocessing/popen_fork.py | 2 +- Lib/multiprocessing/popen_forkserver.py | 2 +- Misc/NEWS | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/popen_fork.py b/Lib/multiprocessing/popen_fork.py --- a/Lib/multiprocessing/popen_fork.py +++ b/Lib/multiprocessing/popen_fork.py @@ -47,7 +47,7 @@ def wait(self, timeout=None): if self.returncode is None: if timeout is not None: - from .connection import wait + from multiprocessing.connection import wait if not wait([self.sentinel], timeout): return None # This shouldn't block if wait() returned successfully. diff --git a/Lib/multiprocessing/popen_forkserver.py b/Lib/multiprocessing/popen_forkserver.py --- a/Lib/multiprocessing/popen_forkserver.py +++ b/Lib/multiprocessing/popen_forkserver.py @@ -57,7 +57,7 @@ def poll(self, flag=os.WNOHANG): if self.returncode is None: - from .connection import wait + from multiprocessing.connection import wait timeout = 0 if flag == os.WNOHANG else None if not wait([self.sentinel], timeout): return None diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #20633: Replace relative import by absolute import. + - Issue #20980: Stop wrapping exception when using ThreadPool. - Issue #20990: Fix issues found by pyflakes for multiprocessing. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 16:31:46 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 23 Mar 2014 16:31:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTEz?= =?utf-8?q?=3A_make_it_clear_that_create=5Fdefault=5Fcontext=28=29_also_en?= =?utf-8?q?ables_hostname?= Message-ID: <3fsN4L55cDz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/c38ce7726737 changeset: 89930:c38ce7726737 branch: 3.4 parent: 89928:e8942abfb829 user: Antoine Pitrou date: Sun Mar 23 16:31:08 2014 +0100 summary: Issue #20913: make it clear that create_default_context() also enables hostname checking files: Doc/library/ssl.rst | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1626,7 +1626,8 @@ security policy, it is highly recommended that you use the :func:`create_default_context` function to create your SSL context. It will load the system's trusted CA certificates, enable certificate -validation, and try to choose reasonably secure protocol and cipher settings. +validation and hostname checking, and try to choose reasonably secure +protocol and cipher settings. For example, here is how you would use the :class:`smtplib.SMTP` class to create a trusted, secure connection to a SMTP server:: @@ -1641,9 +1642,9 @@ :meth:`SSLContext.load_cert_chain`. By contrast, if you create the SSL context by calling the :class:`SSLContext` -constructor yourself, it will not have certificate validation enabled by -default. If you do so, please read the paragraphs below to achieve a good -security level. +constructor yourself, it will not have certificate validation nor hostname +checking enabled by default. If you do so, please read the paragraphs below +to achieve a good security level. Manual settings ^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 16:31:47 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 23 Mar 2014 16:31:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320913=3A_make_it_clear_that_create=5Fdefault=5F?= =?utf-8?q?context=28=29_also_enables_hostname?= Message-ID: <3fsN4M6W9Cz7LkD@mail.python.org> http://hg.python.org/cpython/rev/015c4d785be7 changeset: 89931:015c4d785be7 parent: 89929:227bbce969df parent: 89930:c38ce7726737 user: Antoine Pitrou date: Sun Mar 23 16:31:34 2014 +0100 summary: Issue #20913: make it clear that create_default_context() also enables hostname checking files: Doc/library/ssl.rst | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1626,7 +1626,8 @@ security policy, it is highly recommended that you use the :func:`create_default_context` function to create your SSL context. It will load the system's trusted CA certificates, enable certificate -validation, and try to choose reasonably secure protocol and cipher settings. +validation and hostname checking, and try to choose reasonably secure +protocol and cipher settings. For example, here is how you would use the :class:`smtplib.SMTP` class to create a trusted, secure connection to a SMTP server:: @@ -1641,9 +1642,9 @@ :meth:`SSLContext.load_cert_chain`. By contrast, if you create the SSL context by calling the :class:`SSLContext` -constructor yourself, it will not have certificate validation enabled by -default. If you do so, please read the paragraphs below to achieve a good -security level. +constructor yourself, it will not have certificate validation nor hostname +checking enabled by default. If you do so, please read the paragraphs below +to achieve a good security level. Manual settings ^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 18:15:55 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 23 Mar 2014 18:15:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_101=3A_as_a_new_step_in_s?= =?utf-8?q?etting_up_a_new_branch=2C_create_a_new_whatsnew?= Message-ID: <3fsQNW37Jlz7Ll0@mail.python.org> http://hg.python.org/peps/rev/2c2175cddef4 changeset: 5423:2c2175cddef4 user: Georg Brandl date: Sun Mar 23 18:14:54 2014 +0100 summary: PEP 101: as a new step in setting up a new branch, create a new whatsnew document. files: pep-0101.txt | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -245,11 +245,15 @@ ___ Move any historical "what's new" entries from Misc/NEWS to Misc/HISTORY. - ___ Doc/tutorial/interpreter.rst (2 references to '[Pp]ython32', one - to 'Python 3.2', also make the date in the banner consistent). + ___ Edit Doc/tutorial/interpreter.rst (2 references to '[Pp]ython3x', + one to 'Python 3.x', also make the date in the banner consistent). - ___ Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which have - each one reference to '[Pp]ython32'. + ___ Edit Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which + have each one reference to '[Pp]ython3x'. + + ___ Add a new whatsnew/3.x.rst file (with the comment near the top + and the toplevel sections copied from the previous file) and + and add it to the toctree in whatsnew/index.rst. ___ Update the version number in configure.ac and re-run autoconf. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 23 18:20:01 2014 From: python-checkins at python.org (donald.stufft) Date: Sun, 23 Mar 2014 18:20:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Accept_PEP_464?= Message-ID: <3fsQTF0My5z7LjM@mail.python.org> http://hg.python.org/peps/rev/825f56e02158 changeset: 5424:825f56e02158 parent: 5422:2e82209dda21 user: Donald Stufft date: Sun Mar 23 13:19:28 2014 -0400 summary: Accept PEP 464 files: pep-0464.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0464.txt b/pep-0464.txt --- a/pep-0464.txt +++ b/pep-0464.txt @@ -5,12 +5,13 @@ Author: Donald Stufft BDFL-Delegate: Richard Jones Discussions-To: distutils-sig at python.org -Status: Draft +Status: Accepted Type: Process Content-Type: text/x-rst Created: 02-Mar-2014 Post-History: 04-Mar-2014 Replaces: 381 +Resolution: https://mail.python.org/pipermail/distutils-sig/2014-March/024027.html Abstract -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 23 18:20:02 2014 From: python-checkins at python.org (donald.stufft) Date: Sun, 23 Mar 2014 18:20:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps_=28merge_default_-=3E_default=29?= =?utf-8?q?=3A_Merge?= Message-ID: <3fsQTG25SRz7LjV@mail.python.org> http://hg.python.org/peps/rev/1c946bfa6095 changeset: 5425:1c946bfa6095 parent: 5424:825f56e02158 parent: 5423:2c2175cddef4 user: Donald Stufft date: Sun Mar 23 13:19:55 2014 -0400 summary: Merge files: pep-0101.txt | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -245,11 +245,15 @@ ___ Move any historical "what's new" entries from Misc/NEWS to Misc/HISTORY. - ___ Doc/tutorial/interpreter.rst (2 references to '[Pp]ython32', one - to 'Python 3.2', also make the date in the banner consistent). + ___ Edit Doc/tutorial/interpreter.rst (2 references to '[Pp]ython3x', + one to 'Python 3.x', also make the date in the banner consistent). - ___ Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which have - each one reference to '[Pp]ython32'. + ___ Edit Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which + have each one reference to '[Pp]ython3x'. + + ___ Add a new whatsnew/3.x.rst file (with the comment near the top + and the toplevel sections copied from the previous file) and + and add it to the toctree in whatsnew/index.rst. ___ Update the version number in configure.ac and re-run autoconf. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 23 19:19:32 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 23 Mar 2014 19:19:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIwOTc2OiByZW1v?= =?utf-8?q?ve_unneeded_quopri_import_in_email=2Eutils=2E?= Message-ID: <3fsRnw5y9Tz7LjY@mail.python.org> http://hg.python.org/cpython/rev/5d645f290d6a changeset: 89932:5d645f290d6a branch: 3.4 parent: 89930:c38ce7726737 user: R David Murray date: Sun Mar 23 14:18:44 2014 -0400 summary: #20976: remove unneeded quopri import in email.utils. files: Lib/email/message.py | 3 ++- Lib/email/utils.py | 3 --- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -8,6 +8,7 @@ import re import uu +import quopri from io import BytesIO, StringIO # Intrapackage imports @@ -278,7 +279,7 @@ if not decode: return payload if cte == 'quoted-printable': - return utils._qdecode(bpayload) + return quopri.decodestring(bpayload) elif cte == 'base64': # XXX: this is a bit of a hack; decode_b should probably be factored # out somewhere, but I haven't figured out where yet. diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -36,10 +36,7 @@ from email._parseaddr import parsedate, parsedate_tz, _parsedate_tz -from quopri import decodestring as _qdecode - # Intrapackage imports -from email.encoders import _bencode, _qencode from email.charset import Charset COMMASPACE = ', ' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 19:25:07 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 23 Mar 2014 19:25:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_=2320976=3A_remove_unneeded_quopri_import_in_email?= =?utf-8?q?=2Eutils=2E?= Message-ID: <3fsRwM2JXwz7LjY@mail.python.org> http://hg.python.org/cpython/rev/d308c20bf2f4 changeset: 89933:d308c20bf2f4 parent: 89931:015c4d785be7 parent: 89932:5d645f290d6a user: R David Murray date: Sun Mar 23 14:24:44 2014 -0400 summary: Merge #20976: remove unneeded quopri import in email.utils. files: Lib/email/message.py | 3 ++- Lib/email/utils.py | 3 --- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -8,6 +8,7 @@ import re import uu +import quopri from io import BytesIO, StringIO # Intrapackage imports @@ -278,7 +279,7 @@ if not decode: return payload if cte == 'quoted-printable': - return utils._qdecode(bpayload) + return quopri.decodestring(bpayload) elif cte == 'base64': # XXX: this is a bit of a hack; decode_b should probably be factored # out somewhere, but I haven't figured out where yet. diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -36,10 +36,7 @@ from email._parseaddr import parsedate, parsedate_tz, _parsedate_tz -from quopri import decodestring as _qdecode - # Intrapackage imports -from email.encoders import _bencode, _qencode from email.charset import Charset COMMASPACE = ', ' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 19:55:20 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 19:55:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTkw?= =?utf-8?q?=3A_Correction_for_619331c67638=2E?= Message-ID: <3fsSbD4GtZz7LjV@mail.python.org> http://hg.python.org/cpython/rev/bb6377db0a9e changeset: 89934:bb6377db0a9e branch: 3.4 parent: 89932:5d645f290d6a user: Richard Oudkerk date: Sun Mar 23 18:44:11 2014 +0000 summary: Issue #20990: Correction for 619331c67638. files: Lib/multiprocessing/spawn.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -80,8 +80,8 @@ Returns prefix of command line used for spawning a child process ''' if getattr(sys, 'frozen', False): - tmp = ' '.join('%s=%r' % item for item in kwds.items()) - return [sys.executable, '--multiprocessing-fork'] + tmp + return ([sys.executable, '--multiprocessing-fork'] + + ['%s=%r' % item for item in kwds.items()]) else: prog = 'from multiprocessing.spawn import spawn_main; spawn_main(%s)' prog %= ', '.join('%s=%r' % item for item in kwds.items()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 19:55:21 2014 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 23 Mar 2014 19:55:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <3fsSbF69kdz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/71387075b6b1 changeset: 89935:71387075b6b1 parent: 89933:d308c20bf2f4 parent: 89934:bb6377db0a9e user: Richard Oudkerk date: Sun Mar 23 18:53:53 2014 +0000 summary: Merge 3.4. files: Lib/multiprocessing/spawn.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -80,8 +80,8 @@ Returns prefix of command line used for spawning a child process ''' if getattr(sys, 'frozen', False): - tmp = ' '.join('%s=%r' % item for item in kwds.items()) - return [sys.executable, '--multiprocessing-fork'] + tmp + return ([sys.executable, '--multiprocessing-fork'] + + ['%s=%r' % item for item in kwds.items()]) else: prog = 'from multiprocessing.spawn import spawn_main; spawn_main(%s)' prog %= ', '.join('%s=%r' % item for item in kwds.items()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 20:47:28 2014 From: python-checkins at python.org (r.david.murray) Date: Sun, 23 Mar 2014 20:47:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2320145=3A_assert=5BRaise?= =?utf-8?q?s=7CWarns=5DRegex_now_raise_TypeError_on_bad_regex=2E?= Message-ID: <3fsTlN5YKTz7LjS@mail.python.org> http://hg.python.org/cpython/rev/ec556e45641a changeset: 89936:ec556e45641a user: R David Murray date: Sun Mar 23 15:08:43 2014 -0400 summary: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. Previously a non-string, non-regex second argument could cause the test to always pass. Initial patch by Kamilla Holanda. files: Doc/whatsnew/3.5.rst | 6 ++++++ Lib/unittest/case.py | 2 +- Lib/unittest/test/test_case.py | 12 ++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 5 files changed, 23 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -192,3 +192,9 @@ if it represented midnight in UTC. This behavior was considered obscure and error-prone and has been removed in Python 3.5. See :issue:`13936` for full details. + +* :meth:`unittest.TestCase.assertRaisesRegex` and + :meth:`~unittest.TestCase.assertWarnsRegex` now raise a :exc:`TypeError` if + the second argument is not a string or a compiled :mod:`regex`. You may have + tests with an invalid second argument that have until 3.5 been falsely + passing which will now raise TypeErrors (:issue:`20145`). diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -143,7 +143,7 @@ self.obj_name = str(callable_obj) else: self.obj_name = None - if isinstance(expected_regex, (bytes, str)): + if expected_regex is not None: expected_regex = re.compile(expected_regex) self.expected_regex = expected_regex self.msg = None diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -1126,6 +1126,18 @@ self.assertRaisesRegex, Exception, 'x', lambda: None) + def testAssertRaisesRegexInvalidRegex(self): + # Issue 20145. + class MyExc(Exception): + pass + self.assertRaises(TypeError, self.assertRaisesRegex, MyExc, lambda: True) + + def testAssertWarnsRegexInvalidRegex(self): + # Issue 20145. + class MyWarn(Warning): + pass + self.assertRaises(TypeError, self.assertWarnsRegex, MyWarn, lambda: True) + def testAssertRaisesRegexMismatch(self): def Stub(): raise Exception('Unexpected') diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -546,6 +546,7 @@ Albert Hofkamp Tomas Hoger Jonathan Hogg +Kamilla Holanda Steve Holden Akintayo Holder Thomas Holenstein diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #20145: `assertRaisesRegex` and `assertWarnsRegex` now raise a + TypeError if the second argument is not a string or compiled regex. + - Issue #20633: Replace relative import by absolute import. - Issue #20980: Stop wrapping exception when using ThreadPool. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 22:56:00 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 23 Mar 2014 22:56:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5NTM3?= =?utf-8?q?=3A_Fix_PyUnicode=5FDATA=28=29_alignment_under_m68k=2E__Patch_b?= =?utf-8?q?y_Andreas?= Message-ID: <3fsXbh241Sz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/004ae1472a43 changeset: 89937:004ae1472a43 branch: 3.4 parent: 89934:bb6377db0a9e user: Antoine Pitrou date: Sun Mar 23 22:55:03 2014 +0100 summary: Issue #19537: Fix PyUnicode_DATA() alignment under m68k. Patch by Andreas Schwab. files: Include/unicodeobject.h | 3 +++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -343,6 +343,9 @@ the data pointer is filled out. The bit is redundant, and helps to minimize the test in PyUnicode_IS_READY(). */ unsigned int ready:1; + /* Padding to ensure that PyUnicode_DATA() is always aligned to + 4 bytes (see issue #19537 on m68k). */ + unsigned int :24; } state; wchar_t *wstr; /* wchar_t representation (null-terminated) */ } PyASCIIObject; diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1170,6 +1170,7 @@ Chad J. Schroeder Christian Schubert Sam Schulenburg +Andreas Schwab Stefan Schwarzer Dietmar Schwertberger Federico Schwindt diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19537: Fix PyUnicode_DATA() alignment under m68k. Patch by + Andreas Schwab. + - Issue #20929: Add a type cast to avoid shifting a negative number. - Issue #20731: Properly position in source code files even if they -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 22:56:01 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 23 Mar 2014 22:56:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319537=3A_Fix_PyUnicode=5FDATA=28=29_alignment_u?= =?utf-8?q?nder_m68k=2E__Patch_by_Andreas?= Message-ID: <3fsXbj3qHgz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/0128b25068de changeset: 89938:0128b25068de parent: 89936:ec556e45641a parent: 89937:004ae1472a43 user: Antoine Pitrou date: Sun Mar 23 22:55:40 2014 +0100 summary: Issue #19537: Fix PyUnicode_DATA() alignment under m68k. Patch by Andreas Schwab. files: Include/unicodeobject.h | 3 +++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -343,6 +343,9 @@ the data pointer is filled out. The bit is redundant, and helps to minimize the test in PyUnicode_IS_READY(). */ unsigned int ready:1; + /* Padding to ensure that PyUnicode_DATA() is always aligned to + 4 bytes (see issue #19537 on m68k). */ + unsigned int :24; } state; wchar_t *wstr; /* wchar_t representation (null-terminated) */ } PyASCIIObject; diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1171,6 +1171,7 @@ Chad J. Schroeder Christian Schubert Sam Schulenburg +Andreas Schwab Stefan Schwarzer Dietmar Schwertberger Federico Schwindt diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19537: Fix PyUnicode_DATA() alignment under m68k. Patch by + Andreas Schwab. + - Issue #20929: Add a type cast to avoid shifting a negative number. - Issue #20731: Properly position in source code files even if they -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 23:04:55 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 23 Mar 2014 23:04:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogdHV0b3JpYWw6IG5v?= =?utf-8?q?_=22linux2=22_sys=2Eplatform_on_3=2Ex_=28thanks_Arfrever=29?= Message-ID: <3fsXnz4Kfxz7LjM@mail.python.org> http://hg.python.org/cpython/rev/f2dc5e60a293 changeset: 89939:f2dc5e60a293 branch: 3.4 parent: 89937:004ae1472a43 user: Georg Brandl date: Sun Mar 23 23:03:59 2014 +0100 summary: tutorial: no "linux2" sys.platform on 3.x (thanks Arfrever) files: Doc/tutorial/interpreter.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -97,7 +97,7 @@ $ python3.4 Python 3.4 (default, Mar 16 2014, 09:25:04) - [GCC 4.8.2] on linux2 + [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 23 23:04:56 2014 From: python-checkins at python.org (georg.brandl) Date: Sun, 23 Mar 2014 23:04:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3fsXp05hPVz7LjM@mail.python.org> http://hg.python.org/cpython/rev/fa89769a4279 changeset: 89940:fa89769a4279 parent: 89938:0128b25068de parent: 89939:f2dc5e60a293 user: Georg Brandl date: Sun Mar 23 23:04:45 2014 +0100 summary: merge with 3.4 files: Doc/tutorial/interpreter.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -97,7 +97,7 @@ $ python3.5 Python 3.5 (default, Sep 16 2015, 09:25:04) - [GCC 4.8.2] on linux2 + [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 00:05:41 2014 From: python-checkins at python.org (donald.stufft) Date: Mon, 24 Mar 2014 00:05:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDEz?= =?utf-8?q?=3A_Enhance_ssl=2Ecreate=5Fdefault=5Fcontext=28=29_for_server_s?= =?utf-8?q?ide_contexts?= Message-ID: <3fsZ8574jhz7LjP@mail.python.org> http://hg.python.org/cpython/rev/92efd86d1a38 changeset: 89941:92efd86d1a38 branch: 3.4 parent: 89939:f2dc5e60a293 user: Donald Stufft date: Sun Mar 23 19:05:28 2014 -0400 summary: Issue #21013: Enhance ssl.create_default_context() for server side contexts Closes #21013 by modfying ssl.create_default_context() to: * Move the restricted ciphers to only apply when using ssl.Purpose.CLIENT_AUTH. The major difference between restricted and not is the lack of RC4 in the restricted. However there are servers that exist that only expose RC4 still. * Switches the default protocol to ssl.PROTOCOL_SSLv23 so that the context will select TLS1.1 or TLS1.2 if it is available. * Add ssl.OP_NO_SSLv3 by default to continue to block SSL3.0 sockets * Add ssl.OP_SINGLE_DH_USE and ssl.OP_SINGLE_ECDG_USE to improve the security of the perfect forward secrecy * Add ssl.OP_CIPHER_SERVER_PREFERENCE so that when used for a server side socket the context will prioritize our ciphers which have been carefully selected to maximize security and performance. * Documents the failure conditions when a SSL3.0 connection is required so that end users can more easily determine if they need to unset ssl.OP_NO_SSLv3. files: Doc/library/ssl.rst | 27 ++++++++++++++++++++------- Lib/ssl.py | 30 ++++++++++++++++++++++++------ Lib/test/test_ssl.py | 26 +++++++++++++++++++++++--- Misc/NEWS | 3 +++ 4 files changed, 70 insertions(+), 16 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -250,13 +250,13 @@ :const:`None`, this function can choose to trust the system's default CA certificates instead. - The settings in Python 3.4 are: :data:`PROTOCOL_TLSv1` with high encryption - cipher suites without RC4 and without unauthenticated cipher suites. - Passing :data:`~Purpose.SERVER_AUTH` as *purpose* sets - :data:`~SSLContext.verify_mode` to :data:`CERT_REQUIRED` and either - loads CA certificates (when at least one of *cafile*, *capath* or *cadata* - is given) or uses :meth:`SSLContext.load_default_certs` to load default - CA certificates. + The settings in Python 3.4 are: :data:`PROTOCOL_SSLv23`, :data:`OP_NO_SSLv2`, + and :data:`OP_NO_SSLv3` with high encryption cipher suites without RC4 and + without unauthenticated cipher suites. Passing :data:`~Purpose.SERVER_AUTH` + as *purpose* sets :data:`~SSLContext.verify_mode` to :data:`CERT_REQUIRED` + and either loads CA certificates (when at least one of *cafile*, *capath* or + *cadata* is given) or uses :meth:`SSLContext.load_default_certs` to load + default CA certificates. .. note:: The protocol, options, cipher and other settings may change to more @@ -266,6 +266,19 @@ If your application needs specific settings, you should create a :class:`SSLContext` and apply the settings yourself. + .. note:: + If you find that when certain older clients or servers attempt to connect + with a :class:`SSLContext` created by this function that they get an + error stating "Protocol or cipher suite mismatch", it may be that they + only support SSL3.0 which this function excludes using the + :data:`OP_NO_SSLv3`. SSL3.0 has problematic security due to a number of + poor implementations and it's reliance on MD5 within the protocol. If you + wish to continue to use this function but still allow SSL 3.0 connections + you can re-enable them using:: + + ctx = ssl.create_default_context(Purpose.CLIENT_AUTH) + ctx.options &= ~ssl.OP_NO_SSLv3 + .. versionadded:: 3.4 diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -179,7 +179,7 @@ 'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5' ) -# Restricted and more secure ciphers +# Restricted and more secure ciphers for the server side # This list has been explicitly chosen to: # * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) # * Prefer ECDHE over DHE for better performance @@ -188,7 +188,7 @@ # * Then Use 3DES as fallback which is secure but slow # * Disable NULL authentication, NULL encryption, MD5 MACs, DSS, and RC4 for # security reasons -_RESTRICTED_CIPHERS = ( +_RESTRICTED_SERVER_CIPHERS = ( 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:' '!eNULL:!MD5:!DSS:!RC4' @@ -404,17 +404,35 @@ """ if not isinstance(purpose, _ASN1Object): raise TypeError(purpose) - context = SSLContext(PROTOCOL_TLSv1) + + context = SSLContext(PROTOCOL_SSLv23) + # SSLv2 considered harmful. context.options |= OP_NO_SSLv2 + + # SSLv3 has problematic security and is only required for really old + # clients such as IE6 on Windows XP + context.options |= OP_NO_SSLv3 + # disable compression to prevent CRIME attacks (OpenSSL 1.0+) context.options |= getattr(_ssl, "OP_NO_COMPRESSION", 0) - # disallow ciphers with known vulnerabilities - context.set_ciphers(_RESTRICTED_CIPHERS) - # verify certs and host name in client mode + if purpose == Purpose.SERVER_AUTH: + # verify certs and host name in client mode context.verify_mode = CERT_REQUIRED context.check_hostname = True + elif purpose == Purpose.CLIENT_AUTH: + # Prefer the server's ciphers by default so that we get stronger + # encryption + context.options |= getattr(_ssl, "OP_CIPHER_SERVER_PREFERENCE", 0) + + # Use single use keys in order to improve forward secrecy + context.options |= getattr(_ssl, "OP_SINGLE_DH_USE", 0) + context.options |= getattr(_ssl, "OP_SINGLE_ECDH_USE", 0) + + # disallow ciphers with known vulnerabilities + context.set_ciphers(_RESTRICTED_SERVER_CIPHERS) + if cafile or capath or cadata: context.load_verify_locations(cafile, capath, cadata) elif context.verify_mode != CERT_NONE: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1014,23 +1014,43 @@ def test_create_default_context(self): ctx = ssl.create_default_context() - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertTrue(ctx.check_hostname) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) with open(SIGNING_CA) as f: cadata = f.read() ctx = ssl.create_default_context(cafile=SIGNING_CA, capath=CAPATH, cadata=cadata) - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) + self.assertEqual( + ctx.options & getattr(ssl, "OP_SINGLE_DH_USE", 0), + getattr(ssl, "OP_SINGLE_DH_USE", 0), + ) + self.assertEqual( + ctx.options & getattr(ssl, "OP_SINGLE_ECDH_USE", 0), + getattr(ssl, "OP_SINGLE_ECDH_USE", 0), + ) def test__create_stdlib_context(self): ctx = ssl._create_stdlib_context() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,9 @@ Library ------- +- Issue #21013: Enhance ssl.create_default_context() when used for server side + sockets to provide better security by default. + - Issue #20633: Replace relative import by absolute import. - Issue #20980: Stop wrapping exception when using ThreadPool. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 00:12:26 2014 From: python-checkins at python.org (donald.stufft) Date: Mon, 24 Mar 2014 00:12:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_the_patch_for_issue_=2321013_into_default?= Message-ID: <3fsZHt3yKMz7LjT@mail.python.org> http://hg.python.org/cpython/rev/aa2eb034c4f7 changeset: 89942:aa2eb034c4f7 parent: 89940:fa89769a4279 parent: 89941:92efd86d1a38 user: Donald Stufft date: Sun Mar 23 19:12:13 2014 -0400 summary: Merge the patch for issue #21013 into default files: Doc/library/ssl.rst | 27 ++++++++++++++++++++------- Lib/ssl.py | 30 ++++++++++++++++++++++++------ Lib/test/test_ssl.py | 26 +++++++++++++++++++++++--- Misc/NEWS | 3 +++ 4 files changed, 70 insertions(+), 16 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -250,13 +250,13 @@ :const:`None`, this function can choose to trust the system's default CA certificates instead. - The settings in Python 3.4 are: :data:`PROTOCOL_TLSv1` with high encryption - cipher suites without RC4 and without unauthenticated cipher suites. - Passing :data:`~Purpose.SERVER_AUTH` as *purpose* sets - :data:`~SSLContext.verify_mode` to :data:`CERT_REQUIRED` and either - loads CA certificates (when at least one of *cafile*, *capath* or *cadata* - is given) or uses :meth:`SSLContext.load_default_certs` to load default - CA certificates. + The settings in Python 3.4 are: :data:`PROTOCOL_SSLv23`, :data:`OP_NO_SSLv2`, + and :data:`OP_NO_SSLv3` with high encryption cipher suites without RC4 and + without unauthenticated cipher suites. Passing :data:`~Purpose.SERVER_AUTH` + as *purpose* sets :data:`~SSLContext.verify_mode` to :data:`CERT_REQUIRED` + and either loads CA certificates (when at least one of *cafile*, *capath* or + *cadata* is given) or uses :meth:`SSLContext.load_default_certs` to load + default CA certificates. .. note:: The protocol, options, cipher and other settings may change to more @@ -266,6 +266,19 @@ If your application needs specific settings, you should create a :class:`SSLContext` and apply the settings yourself. + .. note:: + If you find that when certain older clients or servers attempt to connect + with a :class:`SSLContext` created by this function that they get an + error stating "Protocol or cipher suite mismatch", it may be that they + only support SSL3.0 which this function excludes using the + :data:`OP_NO_SSLv3`. SSL3.0 has problematic security due to a number of + poor implementations and it's reliance on MD5 within the protocol. If you + wish to continue to use this function but still allow SSL 3.0 connections + you can re-enable them using:: + + ctx = ssl.create_default_context(Purpose.CLIENT_AUTH) + ctx.options &= ~ssl.OP_NO_SSLv3 + .. versionadded:: 3.4 diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -179,7 +179,7 @@ 'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5' ) -# Restricted and more secure ciphers +# Restricted and more secure ciphers for the server side # This list has been explicitly chosen to: # * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE) # * Prefer ECDHE over DHE for better performance @@ -188,7 +188,7 @@ # * Then Use 3DES as fallback which is secure but slow # * Disable NULL authentication, NULL encryption, MD5 MACs, DSS, and RC4 for # security reasons -_RESTRICTED_CIPHERS = ( +_RESTRICTED_SERVER_CIPHERS = ( 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:' 'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:' '!eNULL:!MD5:!DSS:!RC4' @@ -404,17 +404,35 @@ """ if not isinstance(purpose, _ASN1Object): raise TypeError(purpose) - context = SSLContext(PROTOCOL_TLSv1) + + context = SSLContext(PROTOCOL_SSLv23) + # SSLv2 considered harmful. context.options |= OP_NO_SSLv2 + + # SSLv3 has problematic security and is only required for really old + # clients such as IE6 on Windows XP + context.options |= OP_NO_SSLv3 + # disable compression to prevent CRIME attacks (OpenSSL 1.0+) context.options |= getattr(_ssl, "OP_NO_COMPRESSION", 0) - # disallow ciphers with known vulnerabilities - context.set_ciphers(_RESTRICTED_CIPHERS) - # verify certs and host name in client mode + if purpose == Purpose.SERVER_AUTH: + # verify certs and host name in client mode context.verify_mode = CERT_REQUIRED context.check_hostname = True + elif purpose == Purpose.CLIENT_AUTH: + # Prefer the server's ciphers by default so that we get stronger + # encryption + context.options |= getattr(_ssl, "OP_CIPHER_SERVER_PREFERENCE", 0) + + # Use single use keys in order to improve forward secrecy + context.options |= getattr(_ssl, "OP_SINGLE_DH_USE", 0) + context.options |= getattr(_ssl, "OP_SINGLE_ECDH_USE", 0) + + # disallow ciphers with known vulnerabilities + context.set_ciphers(_RESTRICTED_SERVER_CIPHERS) + if cafile or capath or cadata: context.load_verify_locations(cafile, capath, cadata) elif context.verify_mode != CERT_NONE: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1014,23 +1014,43 @@ def test_create_default_context(self): ctx = ssl.create_default_context() - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertTrue(ctx.check_hostname) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) with open(SIGNING_CA) as f: cadata = f.read() ctx = ssl.create_default_context(cafile=SIGNING_CA, capath=CAPATH, cadata=cadata) - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) - self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self.assertEqual( + ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), + getattr(ssl, "OP_NO_COMPRESSION", 0), + ) + self.assertEqual( + ctx.options & getattr(ssl, "OP_SINGLE_DH_USE", 0), + getattr(ssl, "OP_SINGLE_DH_USE", 0), + ) + self.assertEqual( + ctx.options & getattr(ssl, "OP_SINGLE_ECDH_USE", 0), + getattr(ssl, "OP_SINGLE_ECDH_USE", 0), + ) def test__create_stdlib_context(self): ctx = ssl._create_stdlib_context() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,9 @@ Library ------- +- Issue #21013: Enhance ssl.create_default_context() when used for server side + sockets to provide better security by default. + - Issue #20145: `assertRaisesRegex` and `assertWarnsRegex` now raise a TypeError if the second argument is not a string or compiled regex. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 00:31:29 2014 From: python-checkins at python.org (donald.stufft) Date: Mon, 24 Mar 2014 00:31:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_myself_to_some_=22exp?= =?utf-8?q?erts=22?= Message-ID: <3fsZjs73jhz7LjP@mail.python.org> http://hg.python.org/devguide/rev/e665cddd6cde changeset: 683:e665cddd6cde user: Donald Stufft date: Sun Mar 23 19:31:22 2014 -0400 summary: Add myself to some "experts" files: experts.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -96,12 +96,12 @@ decimal facundobatista, rhettinger, mark.dickinson difflib tim.peters (inactive) dis ncoghlan* -distutils eric.araujo +distutils eric.araujo, dstufft doctest tim.peters (inactive) dummy_threading brett.cannon email barry, r.david.murray* encodings lemburg, loewis -enum eli.bendersky*, barry, ethan.furman* +enum eli.bendersky*, barry, ethan.furman* errno exceptions fcntl @@ -204,7 +204,7 @@ socketserver spwd sqlite3 ghaering -ssl janssen, pitrou, giampaolo.rodola, christian.heimes +ssl janssen, pitrou, giampaolo.rodola, christian.heimes, dstufft stat christian.heimes statistics string georg.brandl* @@ -309,7 +309,7 @@ bytecode benjamin.peterson, pitrou, georg.brandl context managers ncoghlan coverity scan christian.heimes, brett.cannon -cryptography christian.heimes, gregory.p.smith +cryptography christian.heimes, gregory.p.smith, dstufft data formats mark.dickinson, georg.brandl database lemburg devguide ncoghlan, eric.araujo, ezio.melotti @@ -323,7 +323,7 @@ memory management tim.peters, lemburg networking giampaolo.rodola, pitrou object model benjamin.peterson -packaging tarek, lemburg, alexis, eric.araujo +packaging tarek, lemburg, alexis, eric.araujo, dstufft py3 transition benjamin.peterson release management tarek, lemburg, benjamin.peterson, barry, loewis, gvanrossum, anthonybaxter, eric.araujo, ned.deily, -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon Mar 24 04:21:54 2014 From: python-checkins at python.org (zach.ware) Date: Mon, 24 Mar 2014 04:21:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Remove_superfl?= =?utf-8?q?uous_open_parenthesis=2E__Noticed_by_cocoatomo_on_docs=40=2E?= Message-ID: <3fsgqk6Q6kz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/aba42f4f19f4 changeset: 89943:aba42f4f19f4 branch: 3.4 parent: 89941:92efd86d1a38 user: Zachary Ware date: Sun Mar 23 22:21:38 2014 -0500 summary: Remove superfluous open parenthesis. Noticed by cocoatomo on docs at . files: Doc/distutils/sourcedist.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -76,7 +76,7 @@ :option:`packages` options * all C source files mentioned in the :option:`ext_modules` or - :option:`libraries` options ( + :option:`libraries` options .. XXX getting C library sources currently broken---no :meth:`get_source_files` method in :file:`build_clib.py`! -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 04:21:56 2014 From: python-checkins at python.org (zach.ware) Date: Mon, 24 Mar 2014 04:21:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <3fsgqm12W2z7LjS@mail.python.org> http://hg.python.org/cpython/rev/dd02c32f42f7 changeset: 89944:dd02c32f42f7 parent: 89942:aa2eb034c4f7 parent: 89943:aba42f4f19f4 user: Zachary Ware date: Sun Mar 23 22:22:12 2014 -0500 summary: Merge with 3.4 files: Doc/distutils/sourcedist.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -76,7 +76,7 @@ :option:`packages` options * all C source files mentioned in the :option:`ext_modules` or - :option:`libraries` options ( + :option:`libraries` options .. XXX getting C library sources currently broken---no :meth:`get_source_files` method in :file:`build_clib.py`! -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 08:46:49 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 24 Mar 2014 08:46:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Use_deprecated-removed_dir?= =?utf-8?q?ective=2C_clean_up_a_few_markup_issues_around?= Message-ID: <3fsnjP2ZTVz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/86e00b82abc5 changeset: 89945:86e00b82abc5 user: Georg Brandl date: Mon Mar 24 08:42:37 2014 +0100 summary: Use deprecated-removed directive, clean up a few markup issues around deprecated directives. files: Doc/library/formatter.rst | 2 +- Doc/library/imp.rst | 30 +++++++++++++------------- Doc/library/importlib.rst | 4 +- Doc/library/optparse.rst | 4 +- Doc/library/pkgutil.rst | 4 +- Doc/library/site.rst | 2 +- 6 files changed, 23 insertions(+), 23 deletions(-) diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -5,7 +5,7 @@ :synopsis: Generic output formatter and device interface. :deprecated: -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.6 Due to lack of usage, the formatter module has been deprecated and is slated for removal in Python 3.6. diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -262,12 +262,12 @@ exception is made for circular imports, which by construction have to expose an incomplete module object at some point. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 .. function:: acquire_lock() @@ -282,12 +282,12 @@ On platforms without threads, this function does nothing. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 .. function:: release_lock() @@ -295,12 +295,12 @@ Release the interpreter's global import lock. On platforms without threads, this function does nothing. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 The following constants with integer values, defined in this module, are used diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -626,8 +626,8 @@ have to implement it, but it is still available for compatibility purposes. Raise :exc:`IOError` if the path cannot be handled. - .. versionchanged:: 3.4 - Raise :exc:`IOError` instead of :exc:`NotImplementedError`. + .. versionchanged:: 3.4 + Raise :exc:`IOError` instead of :exc:`NotImplementedError`. .. method:: set_data(path, data) diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -8,8 +8,8 @@ .. sectionauthor:: Greg Ward .. deprecated:: 3.2 - The :mod:`optparse` module is deprecated and will not be developed further; - development will continue with the :mod:`argparse` module. + The :mod:`optparse` module is deprecated and will not be developed further; + development will continue with the :mod:`argparse` module. **Source code:** :source:`Lib/optparse.py` diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) diff --git a/Doc/library/site.rst b/Doc/library/site.rst --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -38,7 +38,7 @@ if it refers to an existing directory, and if so, adds it to ``sys.path`` and also inspects the newly added path for configuration files. -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.5 Support for the "site-python" directory will be removed in 3.5. If a file named "pyvenv.cfg" exists one directory above sys.executable, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 09:07:32 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 24 Mar 2014 09:07:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_a_few_scop?= =?utf-8?q?ing_issues_with_versionadded/versionchanged_directives=2E?= Message-ID: <3fsp9J28rxz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/b4be6a008fa5 changeset: 89946:b4be6a008fa5 branch: 3.4 parent: 89943:aba42f4f19f4 user: Georg Brandl date: Mon Mar 24 09:06:18 2014 +0100 summary: Fix a few scoping issues with versionadded/versionchanged directives. files: Doc/c-api/exceptions.rst | 8 ++-- Doc/c-api/object.rst | 1 + Doc/library/doctest.rst | 3 +- Doc/library/faulthandler.rst | 4 +- Doc/library/functions.rst | 1 + Doc/library/imaplib.rst | 1 + Doc/library/imp.rst | 30 ++++++++++---------- Doc/library/importlib.rst | 4 +- Doc/library/multiprocessing.rst | 4 +- Doc/library/optparse.rst | 4 +- Doc/library/sys.rst | 3 +- Doc/library/urllib.request.rst | 7 ++-- 12 files changed, 38 insertions(+), 32 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -247,7 +247,7 @@ filename object, for raising errors when a function that takes two filenames fails. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: PyObject* PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename) @@ -295,7 +295,7 @@ but accepts a second filename object. Availability: Windows. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: PyObject* PyErr_SetExcFromWindowsErrWithFilename(PyObject *type, int ierr, const char *filename) @@ -321,7 +321,7 @@ attributes, which make the exception printing subsystem think the exception is a :exc:`SyntaxError`. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: void PyErr_SyntaxLocationEx(char *filename, int lineno, int col_offset) @@ -329,7 +329,7 @@ Like :c:func:`PyErr_SyntaxLocationObject`, but *filename* is a byte string decoded from the filesystem encoding (:func:`os.fsdecode`). -.. versionadded:: 3.2 + .. versionadded:: 3.2 .. c:function:: void PyErr_SyntaxLocation(char *filename, int lineno) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -364,6 +364,7 @@ .. versionadded:: 3.4 + .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key) Return element of *o* corresponding to the object *key* or *NULL* on failure. diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -502,7 +502,8 @@ :ref:`doctest directives `, and may be passed to the doctest command line interface via the ``-o`` option. -.. versionadded:: 3.4 the ``-o`` command line option +.. versionadded:: 3.4 + The ``-o`` command line option. The first group of options define test semantics, controlling aspects of how doctest decides whether actual output matches an example's expected output: diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -4,6 +4,8 @@ .. module:: faulthandler :synopsis: Dump the Python traceback. +.. versionadded:: 3.3 + This module contains functions to dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal. Call :func:`faulthandler.enable` to install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, @@ -36,8 +38,6 @@ The module is implemented in C, so tracebacks can be dumped on a crash or when Python is deadlocked. -.. versionadded:: 3.3 - Dump the traceback ------------------ diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -832,6 +832,7 @@ .. versionadded:: 3.4 The *default* keyword-only argument. + .. function:: next(iterator[, default]) Retrieve the next item from the *iterator* by calling its diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -451,6 +451,7 @@ :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. method:: IMAP4.status(mailbox, names) Request named status conditions for *mailbox*. diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -262,12 +262,12 @@ exception is made for circular imports, which by construction have to expose an incomplete module object at some point. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 .. function:: acquire_lock() @@ -282,12 +282,12 @@ On platforms without threads, this function does nothing. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 .. function:: release_lock() @@ -295,12 +295,12 @@ Release the interpreter's global import lock. On platforms without threads, this function does nothing. -.. versionchanged:: 3.3 - The locking scheme has changed to per-module locks for - the most part. A global import lock is kept for some critical tasks, - such as initializing the per-module locks. + .. versionchanged:: 3.3 + The locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. -.. deprecated:: 3.4 + .. deprecated:: 3.4 The following constants with integer values, defined in this module, are used diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -626,8 +626,8 @@ have to implement it, but it is still available for compatibility purposes. Raise :exc:`IOError` if the path cannot be handled. - .. versionchanged:: 3.4 - Raise :exc:`IOError` instead of :exc:`NotImplementedError`. + .. versionchanged:: 3.4 + Raise :exc:`IOError` instead of :exc:`NotImplementedError`. .. method:: set_data(path, data) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -138,9 +138,9 @@ .. versionchanged:: 3.4 *spawn* added on all unix platforms, and *forkserver* added for - some unix platforms. + some unix platforms. Child processes no longer inherit all of the parents inheritable - handles on Windows. + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -8,8 +8,8 @@ .. sectionauthor:: Greg Ward .. deprecated:: 3.2 - The :mod:`optparse` module is deprecated and will not be developed further; - development will continue with the :mod:`argparse` module. + The :mod:`optparse` module is deprecated and will not be developed further; + development will continue with the :mod:`argparse` module. **Source code:** :source:`Lib/optparse.py` diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -12,11 +12,12 @@ .. data:: abiflags - On POSIX systems where Python is build with the standard ``configure`` + On POSIX systems where Python was built with the standard ``configure`` script, this contains the ABI flags as specified by :pep:`3149`. .. versionadded:: 3.2 + .. data:: argv The list of command line arguments passed to a Python script. ``argv[0]`` is the diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -523,9 +523,10 @@ Return a list of tuples (header_name, header_value) of the Request headers. .. versionchanged:: 3.4 - Request methods add_data, has_data, get_data, get_type, get_host, - get_selector, get_origin_req_host and is_unverifiable deprecated since 3.3 - have been removed. + The request methods add_data, has_data, get_data, get_type, get_host, + get_selector, get_origin_req_host and is_unverifiable that were deprecated + since 3.3 have been removed. + .. _opener-director-objects: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 09:07:33 2014 From: python-checkins at python.org (georg.brandl) Date: Mon, 24 Mar 2014 09:07:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3fsp9K4vcjz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/d9cdfaf9ac04 changeset: 89947:d9cdfaf9ac04 parent: 89945:86e00b82abc5 parent: 89946:b4be6a008fa5 user: Georg Brandl date: Mon Mar 24 09:06:33 2014 +0100 summary: merge with 3.4 files: Doc/c-api/exceptions.rst | 8 ++++---- Doc/c-api/object.rst | 1 + Doc/library/doctest.rst | 3 ++- Doc/library/faulthandler.rst | 4 ++-- Doc/library/functions.rst | 1 + Doc/library/imaplib.rst | 1 + Doc/library/multiprocessing.rst | 4 ++-- Doc/library/selectors.rst | 2 +- Doc/library/sys.rst | 3 ++- Doc/library/urllib.request.rst | 7 ++++--- 10 files changed, 20 insertions(+), 14 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -247,7 +247,7 @@ filename object, for raising errors when a function that takes two filenames fails. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: PyObject* PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename) @@ -295,7 +295,7 @@ but accepts a second filename object. Availability: Windows. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: PyObject* PyErr_SetExcFromWindowsErrWithFilename(PyObject *type, int ierr, const char *filename) @@ -321,7 +321,7 @@ attributes, which make the exception printing subsystem think the exception is a :exc:`SyntaxError`. -.. versionadded:: 3.4 + .. versionadded:: 3.4 .. c:function:: void PyErr_SyntaxLocationEx(char *filename, int lineno, int col_offset) @@ -329,7 +329,7 @@ Like :c:func:`PyErr_SyntaxLocationObject`, but *filename* is a byte string decoded from the filesystem encoding (:func:`os.fsdecode`). -.. versionadded:: 3.2 + .. versionadded:: 3.2 .. c:function:: void PyErr_SyntaxLocation(char *filename, int lineno) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -364,6 +364,7 @@ .. versionadded:: 3.4 + .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key) Return element of *o* corresponding to the object *key* or *NULL* on failure. diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -502,7 +502,8 @@ :ref:`doctest directives `, and may be passed to the doctest command line interface via the ``-o`` option. -.. versionadded:: 3.4 the ``-o`` command line option +.. versionadded:: 3.4 + The ``-o`` command line option. The first group of options define test semantics, controlling aspects of how doctest decides whether actual output matches an example's expected output: diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -4,6 +4,8 @@ .. module:: faulthandler :synopsis: Dump the Python traceback. +.. versionadded:: 3.3 + This module contains functions to dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal. Call :func:`faulthandler.enable` to install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, @@ -36,8 +38,6 @@ The module is implemented in C, so tracebacks can be dumped on a crash or when Python is deadlocked. -.. versionadded:: 3.3 - Dump the traceback ------------------ diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -832,6 +832,7 @@ .. versionadded:: 3.4 The *default* keyword-only argument. + .. function:: next(iterator[, default]) Retrieve the next item from the *iterator* by calling its diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -451,6 +451,7 @@ :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. method:: IMAP4.status(mailbox, names) Request named status conditions for *mailbox*. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -138,9 +138,9 @@ .. versionchanged:: 3.4 *spawn* added on all unix platforms, and *forkserver* added for - some unix platforms. + some unix platforms. Child processes no longer inherit all of the parents inheritable - handles on Windows. + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -217,7 +217,7 @@ This returns the file descriptor used by the underlying :func:`select.devpoll` object. -.. versionadded:: 3.5 + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -12,11 +12,12 @@ .. data:: abiflags - On POSIX systems where Python is build with the standard ``configure`` + On POSIX systems where Python was built with the standard ``configure`` script, this contains the ABI flags as specified by :pep:`3149`. .. versionadded:: 3.2 + .. data:: argv The list of command line arguments passed to a Python script. ``argv[0]`` is the diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -523,9 +523,10 @@ Return a list of tuples (header_name, header_value) of the Request headers. .. versionchanged:: 3.4 - Request methods add_data, has_data, get_data, get_type, get_host, - get_selector, get_origin_req_host and is_unverifiable deprecated since 3.3 - have been removed. + The request methods add_data, has_data, get_data, get_type, get_host, + get_selector, get_origin_req_host and is_unverifiable that were deprecated + since 3.3 have been removed. + .. _opener-director-objects: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Mar 24 09:56:50 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 24 Mar 2014 09:56:50 +0100 Subject: [Python-checkins] Daily reference leaks (aa2eb034c4f7): sum=4 Message-ID: results for aa2eb034c4f7 on branch "default" -------------------------------------------- test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 0, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogMxwYeC', '-x'] From python-checkins at python.org Mon Mar 24 13:28:19 2014 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Mar 2014 13:28:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_narrow_scope=2C_ad?= =?utf-8?q?d_legacy_SSL_branch?= Message-ID: <3fsvyC2LKKz7Ljf@mail.python.org> http://hg.python.org/peps/rev/f09189910aae changeset: 5426:f09189910aae user: Nick Coghlan date: Mon Mar 24 22:28:06 2014 +1000 summary: PEP 466: narrow scope, add legacy SSL branch files: pep-0466.txt | 246 +++++++++++++++++++++++--------------- 1 files changed, 147 insertions(+), 99 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -33,17 +33,24 @@ * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module -* the ``sha`` module (Python 2 only) -* the components of other networking modules that make use of these modules -* the components of the ``random`` and ``os`` modules that are relevant to - cryptographic applications -* the version of OpenSSL bundled with the binary installers +* the components of the ``random`` and ``os`` modules that the above + modules rely on for cryptographic randomness +* the version of OpenSSL bundled with the binary installers for Windows + and Mac OS X -Proposed backports for these modules will still need to undergo normal -backwards compatibility assessments, but new features will be permitted where -appropriate, making it easier to implement secure networked software in -Python, even for software that needs to remain compatible with older feature -releases of Python. +Changes to these modules will still need to undergo normal backwards +compatibility assessments, but otherwise they will be kept entirely aligned +with the latest feature release of their Python 3 counterparts. This is +designed to make it easier to implement secure networked software in +Python, even for software that currently needs to remain compatible with +the Python 2 series (which includes a lot of network infrastructure +software). + +The development branches will be arranged in such a way that even though +any further Python 2.7 releases published by the core development team +provide updated network security infrastructure, it will remain possible +for downstream redistributors to publish "Python 2.7 with legacy SSL +infrastructure" if they choose to do so. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active @@ -66,7 +73,6 @@ * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module -* the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible changes in these modules to Python 2.7 maintenance releases. This choice is @@ -80,21 +86,22 @@ feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. +Note that the ``sha`` and ``md5`` modules are not covered by this policy, +but have been deprecated in favour of ``hashlib`` since Python 2.5 and have +been removed entirely in the Python 3 series. + In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module -* networking related modules that depend on one or more of the network - security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being -network security related. If the enhancement under discussion is designed -to take advantage of a new feature in one of the network security related -modules, then that will be taken as implying that the enhancement is -security related. +network security related. This is generally restricted to cases where the +feature in question is needed by an update to one of the modules covered +by the blanket exemption. Backwards Compatibility Considerations @@ -117,8 +124,8 @@ It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired -security feature is unavailable, most software SHOULD instead continue -operating using a slightly degraded security configuration. +security feature is unavailable, most software SHOULD instead emit a warning +and continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant @@ -132,13 +139,15 @@ Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) -Security related changes to other modules (such as data format processing -libraries) will continue to be made available as backports and new modules -on the Python Package Index, as independent distribution remains the -preferred approach to handling software that needs to evolve faster than -the standard library. Refer to the `Motivation and Rationale`_ section for -a review of the characteristics that make the secure networking -infrastructure worthy of special consideration. +Security related changes to other modules (such as higher level networking +libraries and data format processing libraries) will continue to be made +available as backports and new modules on the Python Package Index, as +independent distribution remains the preferred approach to handling +software that must continue to evolve to handle changing development +requirements independently of the Python 2 standard library. Refer to +the `Motivation and Rationale`_ section for a review of the characteristics +that make the secure networking infrastructure worthy of special +consideration. Other Considerations @@ -151,23 +160,24 @@ actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing -to review and merge corporate contributions that put this policy into -effect. +to accept offers of corporate assistance in putting this policy into +effect and handling the resulting increase in the Python 2 maintenance +load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream community in cases where they are likely to have a broad impact that helps improve the security of the internet as a -whole. +whole, rather than sitting back and waiting for unpaid volunteers to do it +for them. Documentation ------------- -All modules that take advantage of this policy to backport network -security related enhancements to earlier Python versions MUST include -a "Security Considerations" section in their documentation. +All modules covered by this policy MUST include a "Security Considerations" +section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where @@ -183,23 +193,49 @@ include only critical security fixes. However, the recommendations for library and application developers are -deliberately designed to accommodate commercial redistributors applying -this policy to any Python release series that is either in security -fix only mode, or has been declared "end of life" by the core development -team. +deliberately designed to accommodate commercial redistributors that choose +to apply this policy to additional Python release series that are either in +security fix only mode, or have been declared "end of life" by the core +development team. Whether or not redistributors choose to exercise that option will be up -to the redistributor. +to the individual redistributor. Integration testing ------------------- -Third party integration testing services would likely need to start -offering users a choice of multiple Python 2.7.x versions to test against, -to ensure that the application is correctly degrading gracefully if it -attempts to use newer networking features on maintenance releases that -are too old to provide them. +Third party integration testing services should offer users the ability +to test against both the latest Python 2.7 maintenance release and the +latest "Python 2.7 with legacy SSL infrastructure" release, to ensure that +libraries, frameworks and applications handle the legacy security +infrastructure correctly (either failing or degrading gracefully, depending +on the security sensitivity of the software). + + +Handling lower security environments with low risk tolerance +------------------------------------------------------------ + +For better or for worse (mostly worse), there are some environments where +the risk of latent security defects is more tolerated than the risk of +regressions in maintenance releases. This policy largely excludes these +environments from consideration where the modules covered by the exemption +are concerned. + +However, one concession is made for the benefit of such environments: while +the main ``2.7`` branch in Mercurial will include the updated network +security infrastructure, the development process will be updated to include +a ``2.7-legacy-ssl`` branch with the more limited feature set of the +original 2.7 network security infrastructure, allowing downstream +redistributors to continue to provide Python 2.7 with the legacy SSL +infrastructure if they choose to do so. This branch will be tested +on the CPython continuous integration infrastructure, but no releases will +be made from it by the core development team. + +As noted above, corporate redistributors and users are expected to provide +the additional development effort needed to make this practical. It is not +acceptable to expect volunteer contributors to resolve a problem created +largely by conservative corporate development practices. Evolution of this Policy @@ -212,8 +248,9 @@ network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for -the security design of other services that may be using newer versions of -Python or other development languages. +the security design of other services that may themselves be using newer +versions of Python or other development languages, but need to interoperate +with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of @@ -282,7 +319,7 @@ standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't -even consider the problem and use ordinary equality comparisons instead +even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. @@ -307,37 +344,57 @@ This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications -and integration projects. +and integration projects. While the tools for migration have evolved to +a point where it is possible to migrate even large applications +opportunistically and incrementally (rather than all at once) by updating +code to run in the large common subset of Python 2 and Python 3, using the +most recent technology often isn't a priority in commercial environments. -Now that we're fully aware of the impact the limitations in Python 2 may be -having on the evolution of internet security standards, I no longer believe -that it is reasonable to expect platform and application developers to -resolve all of the latent defects in an application's Unicode correctness -solely in order to gain access to the network security enhancements -available in Python 3. +Previously, this was considered an acceptable harm, as while it was an +unfortunate problem for the affected developers to have to face, it was +seen as an issue between them and their management chain to make the case +for infrastructure modernisation, and this case would become naturally +more compelling as the Python 3 series evolved. -While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 -as its main Python interpreter for its 14.04 LTS release, Fedora still -has a lot of work to do to migrate, and it will take a non-trivial amount -of time to migrate the relevant infrastructure components. While Red Hat -are also actively working to make it easier for users to use more recent -versions of Python on our stable platforms, it's going to take time for -those efforts to start having an impact on end users' choice of version, -and those changes won't affect the core tools regardless. +However, now that we're fully aware of the impact the limitations of the +Python 2 standard library may be having on the evolution of internet +security standards, I no longer believe that it is reasonable to expect +platform and application developers to resolve all of the latent defects +in an application's Unicode correctness solely in order to gain access to +the network security enhancements already available in Python 3. + +While Ubuntu (and to some extent Debian as well) are committed to porting all +default system services and scripts to Python 3, and to removing Python 2 +from its default distribution images (but not from its archives), this is +a mammoth task and won't be completed for the Ubuntu 14.04 LTS release +(at least for the desktop image - it may be achieved for the mobile and +server images). + +Fedora has even more work to do to migrate, and it will take a non-trivial +amount of time to migrate the relevant infrastructure components. While +Red Hat are also actively working to make it easier for users to use more +recent versions of Python on our stable platforms, it's going to take time +for those efforts to start having an impact on end users' choice of version, +and any such changes also don't benefit the core platform infrastructure +that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated -test suite, it's large enough that it is going to take quite some time +test suite, it's still large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed -to help support a migration from Python 2 to Python 3. The key point of -this PEP is that those situations affect more people than just the -developers and users of the affected application: their existence becomes -something that developers of secure networked services need to take into -account as part of their security design. +to help support a migration from Python 2 to Python 3, there are likely to +be many cases where reimplementation (perhaps even in Python 3) proves +easier than migration. The key point of this PEP is that those situations +affect more people than just the developers and users of the affected +application: the existence of clients and servers with outdated network +security infrastructure becomes something that developers of secure +networked services need to take into account as part of their security +design, and that's a problem that inhibits the adoption of better security +standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something @@ -366,45 +423,36 @@ Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` -and the improved coroutine support). +and the improved coroutine support), making the migration from Python 2.7 +to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial -additional work for a result that is actually less effective as a solution -to the original problem (the widespread use of the aging network security -infrastructure in Python 2). +additional work for a result that is actually less effective in achieving +the original aim (which is to eliminate the current widespread use of the +aging network security infrastructure in the Python 2 series). Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- -While it initially appears to be an attractive and easier to manage -approach, there are actually several significant problems with this -idea. +While this initially appears to be an attractive and easier to manage +approach, it actually suffers from several significant problems. -Firstly, this PEP encompasses a non-trivial portion of the standard library. -It's not just the underlying SSL support, but also the libraries for other -network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the -SSL infrastructure to provide secure links, and that's just the protocols -in the standard library. Even if an API compatible ``ssl2`` module was -made available, it would need to be imported and injected -into ``sys.modules`` as ``ssl`` before importing any other module that -needed it. - -Secondly, this is complex, low level, cross-platform code that integrates +Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and -other dynamic languages, but won't suffice for the kind of code involved -here. +other dynamic languages, as well as the more comprehensive abstraction +offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the -``pip`` based software distribution ecosystem. Relying on a binary -dependency also creates potential compatibility problems for ``pip`` when -running on other interpreters like ``PyPy``. +``pip`` based software distribution ecosystem. Relying on a third party +binary dependency also creates potential compatibility problems for ``pip`` +when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional @@ -412,8 +460,7 @@ ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but -it *would* mean yet another copy of OpenSSL to keep up to -date. +it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need @@ -422,17 +469,20 @@ "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user -hostile than taking the risk of trying to upgrade it in place. +hostile than accepting the slightly increased risk of regressions associated +with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new -packages added to the pre-existing set *can* be done, but means going -around to each and every redistributor and asking them to update their +packages added to the pre-existing set *can* be done, but means approaching +each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in -this PEP would require redistributors to *opt out* of the security -enhancements, which most of them are unlikely to do. +this PEP would require redistributors to deliberately *opt out* of the +security enhancements (by switching to redistributing directly from the +``2.7-legacy-ssl`` branch rather than the main ``2.7`` branch), which most +of them are unlikely to do. Open Questions @@ -445,9 +495,6 @@ feature release. In particular, is it possible Windows C extensions may be linking against the Python provided OpenSSL module? -* Are there any other security relevant modules that should be covered - by either a blanket or conditional exemption? - Disclosure of Interest ====================== @@ -473,13 +520,14 @@ Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. -Thanks also to participants in the python-dev mailing list thread [1]_ +Thanks also to participants in the python-dev mailing list threads [1,2]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html +.. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 24 14:20:31 2014 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Mar 2014 14:20:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_cover_OpenSSL_comp?= =?utf-8?q?atibility?= Message-ID: <3fsx6R5sx9z7Ljq@mail.python.org> http://hg.python.org/peps/rev/168173658ca7 changeset: 5427:168173658ca7 user: Nick Coghlan date: Mon Mar 24 23:15:06 2014 +1000 summary: PEP 466: cover OpenSSL compatibility files: pep-0466.txt | 35 +++++++++++++++++++++++++++++------ 1 files changed, 29 insertions(+), 6 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -150,6 +150,30 @@ consideration. +OpenSSL compatibility +--------------------- + +Under this policy, OpenSSL may be upgraded to more recent feature releases +in Python 2.7 maintenance releases. On Linux and most other POSIX systems, +the specific version of OpenSSL used already varies, as Python dynamically +links to the system provided OpenSSL library by default. + +For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are +statically linked with OpenSSL and the associated symbols are not exported. +Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the +``egenix-pyopenssl`` binaries has not resulted in any reported compatibility +issues [3]_ + +The Mac OS X binary installers historically followed the same policy as +other POSIX installations and dynamically linked to the Apple provided +OpenSSL libraries. However, Apple has now ceased updating these +cross-platform libraries, instead requiring that even cross-platform +developers adopt Mac OS X specific interfaces to access up to date security +infrastructure on their platform. Accordingly, and independently of this +PEP, the Mac OS X binary installers were already going to be switched to +statically linker newer versions of OpenSSL [4]_ + + Other Considerations ==================== @@ -488,12 +512,9 @@ Open Questions ============== -* What are the risks associated with allowing OpenSSL to be updated to - new feature versions in the Windows and Mac OS X binary installers for - maintenance releases? Currently we just upgrade to the appropriate - OpenSSL maintenance releases, rather than switching to the latest - feature release. In particular, is it possible Windows C extensions may - be linking against the Python provided OpenSSL module? +* I believe I've addressed all the technical and scope questions I had, or + others raised. That just leaves the question of "If we agree to this plan, + who is actually going to handle all the extra work involved?" :) Disclosure of Interest @@ -528,6 +549,8 @@ .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html +.. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html +.. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon Mar 24 22:36:15 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 24 Mar 2014 22:36:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDM2?= =?utf-8?q?=3A_Fix_typo_in_macro_name?= Message-ID: <3ft86R6CYkz7LjM@mail.python.org> http://hg.python.org/cpython/rev/2bbda947a5b3 changeset: 89948:2bbda947a5b3 branch: 3.4 parent: 89946:b4be6a008fa5 user: Victor Stinner date: Mon Mar 24 22:34:34 2014 +0100 summary: Issue #21036: Fix typo in macro name _PY_HASHTABLE_ENTRY_DATA => _Py_HASHTABLE_ENTRY_DATA files: Modules/_tracemalloc.c | 2 +- Modules/hashtable.c | 4 ++-- Modules/hashtable.h | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -1052,7 +1052,7 @@ PyObject *tracemalloc_obj; int res; - trace = (trace_t *)_PY_HASHTABLE_ENTRY_DATA(entry); + trace = (trace_t *)_Py_HASHTABLE_ENTRY_DATA(entry); tracemalloc_obj = trace_to_pyobject(trace, get_traces->tracebacks); if (tracemalloc_obj == NULL) diff --git a/Modules/hashtable.c b/Modules/hashtable.c --- a/Modules/hashtable.c +++ b/Modules/hashtable.c @@ -326,7 +326,7 @@ entry->key_hash = key_hash; assert(data_size == ht->data_size); - memcpy(_PY_HASHTABLE_ENTRY_DATA(entry), data, data_size); + memcpy(_Py_HASHTABLE_ENTRY_DATA(entry), data, data_size); _Py_slist_prepend(&ht->buckets[index], (_Py_slist_item_t*)entry); ht->entries++; @@ -504,7 +504,7 @@ err = 1; } else { - data = _PY_HASHTABLE_ENTRY_DATA(entry); + data = _Py_HASHTABLE_ENTRY_DATA(entry); err = _Py_hashtable_set(dst, entry->key, data, src->data_size); } if (err) { diff --git a/Modules/hashtable.h b/Modules/hashtable.h --- a/Modules/hashtable.h +++ b/Modules/hashtable.h @@ -26,16 +26,16 @@ /* data follows */ } _Py_hashtable_entry_t; -#define _PY_HASHTABLE_ENTRY_DATA(ENTRY) \ +#define _Py_HASHTABLE_ENTRY_DATA(ENTRY) \ ((char *)(ENTRY) + sizeof(_Py_hashtable_entry_t)) #define _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(ENTRY) \ - (*(void **)_PY_HASHTABLE_ENTRY_DATA(ENTRY)) + (*(void **)_Py_HASHTABLE_ENTRY_DATA(ENTRY)) #define _Py_HASHTABLE_ENTRY_READ_DATA(TABLE, DATA, DATA_SIZE, ENTRY) \ do { \ assert((DATA_SIZE) == (TABLE)->data_size); \ - memcpy(DATA, _PY_HASHTABLE_ENTRY_DATA(ENTRY), DATA_SIZE); \ + memcpy(DATA, _Py_HASHTABLE_ENTRY_DATA(ENTRY), DATA_SIZE); \ } while (0) typedef Py_uhash_t (*_Py_hashtable_hash_func) (const void *key); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 22:36:17 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 24 Mar 2014 22:36:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321036=3A_Fix_typo_in_macro_na?= =?utf-8?q?me?= Message-ID: <3ft86T0txkz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/0251614e853d changeset: 89949:0251614e853d parent: 89947:d9cdfaf9ac04 parent: 89948:2bbda947a5b3 user: Victor Stinner date: Mon Mar 24 22:36:01 2014 +0100 summary: (Merge 3.4) Issue #21036: Fix typo in macro name _PY_HASHTABLE_ENTRY_DATA => _Py_HASHTABLE_ENTRY_DATA files: Modules/_tracemalloc.c | 2 +- Modules/hashtable.c | 4 ++-- Modules/hashtable.h | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -1052,7 +1052,7 @@ PyObject *tracemalloc_obj; int res; - trace = (trace_t *)_PY_HASHTABLE_ENTRY_DATA(entry); + trace = (trace_t *)_Py_HASHTABLE_ENTRY_DATA(entry); tracemalloc_obj = trace_to_pyobject(trace, get_traces->tracebacks); if (tracemalloc_obj == NULL) diff --git a/Modules/hashtable.c b/Modules/hashtable.c --- a/Modules/hashtable.c +++ b/Modules/hashtable.c @@ -326,7 +326,7 @@ entry->key_hash = key_hash; assert(data_size == ht->data_size); - memcpy(_PY_HASHTABLE_ENTRY_DATA(entry), data, data_size); + memcpy(_Py_HASHTABLE_ENTRY_DATA(entry), data, data_size); _Py_slist_prepend(&ht->buckets[index], (_Py_slist_item_t*)entry); ht->entries++; @@ -504,7 +504,7 @@ err = 1; } else { - data = _PY_HASHTABLE_ENTRY_DATA(entry); + data = _Py_HASHTABLE_ENTRY_DATA(entry); err = _Py_hashtable_set(dst, entry->key, data, src->data_size); } if (err) { diff --git a/Modules/hashtable.h b/Modules/hashtable.h --- a/Modules/hashtable.h +++ b/Modules/hashtable.h @@ -26,16 +26,16 @@ /* data follows */ } _Py_hashtable_entry_t; -#define _PY_HASHTABLE_ENTRY_DATA(ENTRY) \ +#define _Py_HASHTABLE_ENTRY_DATA(ENTRY) \ ((char *)(ENTRY) + sizeof(_Py_hashtable_entry_t)) #define _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(ENTRY) \ - (*(void **)_PY_HASHTABLE_ENTRY_DATA(ENTRY)) + (*(void **)_Py_HASHTABLE_ENTRY_DATA(ENTRY)) #define _Py_HASHTABLE_ENTRY_READ_DATA(TABLE, DATA, DATA_SIZE, ENTRY) \ do { \ assert((DATA_SIZE) == (TABLE)->data_size); \ - memcpy(DATA, _PY_HASHTABLE_ENTRY_DATA(ENTRY), DATA_SIZE); \ + memcpy(DATA, _Py_HASHTABLE_ENTRY_DATA(ENTRY), DATA_SIZE); \ } while (0) typedef Py_uhash_t (*_Py_hashtable_hash_func) (const void *key); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 24 23:27:00 2014 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 24 Mar 2014 23:27:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321040=3A_socketse?= =?utf-8?q?rver=3A_Use_the_selectors_module=2E?= Message-ID: <3ft9F03NWhz7LjY@mail.python.org> http://hg.python.org/cpython/rev/3047db8f6126 changeset: 89950:3047db8f6126 user: Charles-Fran?ois Natali date: Mon Mar 24 22:25:39 2014 +0000 summary: Issue #21040: socketserver: Use the selectors module. files: Doc/library/socketserver.rst | 4 +- Lib/socketserver.py | 96 +++++++++++++--------- Lib/test/test_socketserver.py | 32 ------- 3 files changed, 58 insertions(+), 74 deletions(-) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -113,7 +113,7 @@ Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -136,7 +136,7 @@ .. method:: BaseServer.fileno() Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to + listening. This function is most commonly passed to :mod:`selectors`, to allow monitoring multiple servers in the same process. diff --git a/Lib/socketserver.py b/Lib/socketserver.py --- a/Lib/socketserver.py +++ b/Lib/socketserver.py @@ -94,7 +94,7 @@ Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor fork (or where these are too expensive or inappropriate for the service) is to maintain an -explicit table of partially finished requests and to use select() to +explicit table of partially finished requests and to use a selector to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if @@ -104,7 +104,6 @@ - Standard classes for Sun RPC (which uses either UDP or TCP) - Standard mix-in classes to implement various authentication and encryption schemes -- Standard framework for select-based multiplexing XXX Open problems: - What to do with out-of-band data? @@ -130,13 +129,17 @@ import socket -import select +import selectors import os import errno try: import threading except ImportError: import dummy_threading as threading +try: + from time import monotonic as time +except ImportError: + from time import time as time __all__ = ["TCPServer","UDPServer","ForkingUDPServer","ForkingTCPServer", "ThreadingUDPServer","ThreadingTCPServer","BaseRequestHandler", @@ -147,14 +150,13 @@ "ThreadingUnixStreamServer", "ThreadingUnixDatagramServer"]) -def _eintr_retry(func, *args): - """restart a system call interrupted by EINTR""" - while True: - try: - return func(*args) - except OSError as e: - if e.errno != errno.EINTR: - raise +# poll/select have the advantage of not requiring any extra file descriptor, +# contrarily to epoll/kqueue (also, they require a single syscall). +if hasattr(selectors, 'PollSelector'): + _ServerSelector = selectors.PollSelector +else: + _ServerSelector = selectors.SelectSelector + class BaseServer: @@ -166,7 +168,7 @@ - serve_forever(poll_interval=0.5) - shutdown() - handle_request() # if you do not use serve_forever() - - fileno() -> int # for select() + - fileno() -> int # for selector Methods that may be overridden: @@ -227,17 +229,19 @@ """ self.__is_shut_down.clear() try: - while not self.__shutdown_request: - # XXX: Consider using another file descriptor or - # connecting to the socket to wake this up instead of - # polling. Polling reduces our responsiveness to a - # shutdown request and wastes cpu at all other times. - r, w, e = _eintr_retry(select.select, [self], [], [], - poll_interval) - if self in r: - self._handle_request_noblock() + # XXX: Consider using another file descriptor or connecting to the + # socket to wake this up instead of polling. Polling reduces our + # responsiveness to a shutdown request and wastes cpu at all other + # times. + with _ServerSelector() as selector: + selector.register(self, selectors.EVENT_READ) - self.service_actions() + while not self.__shutdown_request: + ready = selector.select(poll_interval) + if ready: + self._handle_request_noblock() + + self.service_actions() finally: self.__shutdown_request = False self.__is_shut_down.set() @@ -260,16 +264,16 @@ """ pass - # The distinction between handling, getting, processing and - # finishing a request is fairly arbitrary. Remember: + # The distinction between handling, getting, processing and finishing a + # request is fairly arbitrary. Remember: # - # - handle_request() is the top-level call. It calls - # select, get_request(), verify_request() and process_request() + # - handle_request() is the top-level call. It calls selector.select(), + # get_request(), verify_request() and process_request() # - get_request() is different for stream or datagram sockets - # - process_request() is the place that may fork a new process - # or create a new thread to finish the request - # - finish_request() instantiates the request handler class; - # this constructor will handle the request all by itself + # - process_request() is the place that may fork a new process or create a + # new thread to finish the request + # - finish_request() instantiates the request handler class; this + # constructor will handle the request all by itself def handle_request(self): """Handle one request, possibly blocking. @@ -283,18 +287,30 @@ timeout = self.timeout elif self.timeout is not None: timeout = min(timeout, self.timeout) - fd_sets = _eintr_retry(select.select, [self], [], [], timeout) - if not fd_sets[0]: - self.handle_timeout() - return - self._handle_request_noblock() + if timeout is not None: + deadline = time() + timeout + + # Wait until a request arrives or the timeout expires - the loop is + # necessary to accomodate early wakeups due to EINTR. + with _ServerSelector() as selector: + selector.register(self, selectors.EVENT_READ) + + while True: + ready = selector.select(timeout) + if ready: + return self._handle_request_noblock() + else: + if timeout is not None: + timeout = deadline - time() + if timeout < 0: + return self.handle_timeout() def _handle_request_noblock(self): """Handle one request, without blocking. - I assume that select.select has returned that the socket is - readable before this function was called, so there should be - no risk of blocking in get_request(). + I assume that selector.select() has returned that the socket is + readable before this function was called, so there should be no risk of + blocking in get_request(). """ try: request, client_address = self.get_request() @@ -377,7 +393,7 @@ - serve_forever(poll_interval=0.5) - shutdown() - handle_request() # if you don't use serve_forever() - - fileno() -> int # for select() + - fileno() -> int # for selector Methods that may be overridden: @@ -459,7 +475,7 @@ def fileno(self): """Return socket file number. - Interface required by select(). + Interface required by selector. """ return self.socket.fileno() diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py --- a/Lib/test/test_socketserver.py +++ b/Lib/test/test_socketserver.py @@ -222,38 +222,6 @@ socketserver.DatagramRequestHandler, self.dgram_examine) - @contextlib.contextmanager - def mocked_select_module(self): - """Mocks the select.select() call to raise EINTR for first call""" - old_select = select.select - - class MockSelect: - def __init__(self): - self.called = 0 - - def __call__(self, *args): - self.called += 1 - if self.called == 1: - # raise the exception on first call - raise OSError(errno.EINTR, os.strerror(errno.EINTR)) - else: - # Return real select value for consecutive calls - return old_select(*args) - - select.select = MockSelect() - try: - yield select.select - finally: - select.select = old_select - - def test_InterruptServerSelectCall(self): - with self.mocked_select_module() as mock_select: - pid = self.run_server(socketserver.TCPServer, - socketserver.StreamRequestHandler, - self.stream_examine) - # Make sure select was called again: - self.assertGreater(mock_select.called, 1) - # Alas, on Linux (at least) recvfrom() doesn't return a meaningful # client address so this cannot work: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 00:27:09 2014 From: python-checkins at python.org (donald.stufft) Date: Tue, 25 Mar 2014 00:27:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDQz?= =?utf-8?q?=3A_Remove_the_recommendation_for_specific_CA_organizations?= Message-ID: <3ftBZP1F2jzSRT@mail.python.org> http://hg.python.org/cpython/rev/6f776c91da08 changeset: 89951:6f776c91da08 branch: 3.4 parent: 89948:2bbda947a5b3 user: Donald Stufft date: Mon Mar 24 19:26:03 2014 -0400 summary: Issue #21043: Remove the recommendation for specific CA organizations Closes #21043 by updating the documentation to remove specific CA organizations and update the text to no longer need to tell you to download root certificates, but instead use the OS certificates avaialble through SSLContext.load_default_certs. files: Doc/library/ssl.rst | 17 +++-------------- Misc/NEWS | 3 +++ 2 files changed, 6 insertions(+), 14 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1339,20 +1339,9 @@ certificate, you need to provide a "CA certs" file, filled with the certificate chains for each issuer you are willing to trust. Again, this file just contains these chains concatenated together. For validation, Python will use the first -chain it finds in the file which matches. Some "standard" root certificates are -available from various certification authorities: `CACert.org -`_, `Thawte -`_, `Verisign -`_, `Positive SSL -`_ -(used by python.org), `Equifax and GeoTrust -`_. - -In general, if you are using SSL3 or TLS1, you don't need to put the full chain -in your "CA certs" file; you only need the root certificates, and the remote -peer is supposed to furnish the other certificates necessary to chain from its -certificate to a root certificate. See :rfc:`4158` for more discussion of the -way in which certification chains can be built. +chain it finds in the file which matches. The platform's certificates file can +be used by calling :meth:`SSLContext.load_default_certs`, this is done +automatically with :func:`.create_default_context`. Combined key and certificate ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -79,6 +79,9 @@ Documentation ------------- +- Issue #21043: Remove the recommendation for specific CA organizations and to + mention the ability to load the OS certificates. + - Issue #20765: Add missing documentation for PurePath.with_name() and PurePath.with_suffix(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 00:28:25 2014 From: python-checkins at python.org (donald.stufft) Date: Tue, 25 Mar 2014 00:28:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_in_3=2E4_to_bring_forward_the_Issue_=2321043_chang?= =?utf-8?b?ZXMu?= Message-ID: <3ftBbs1ZkVz7LjN@mail.python.org> http://hg.python.org/cpython/rev/0485552b487e changeset: 89952:0485552b487e parent: 89950:3047db8f6126 parent: 89951:6f776c91da08 user: Donald Stufft date: Mon Mar 24 19:28:08 2014 -0400 summary: Merge in 3.4 to bring forward the Issue #21043 changes. files: Doc/library/ssl.rst | 17 +++-------------- Misc/NEWS | 3 +++ 2 files changed, 6 insertions(+), 14 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1339,20 +1339,9 @@ certificate, you need to provide a "CA certs" file, filled with the certificate chains for each issuer you are willing to trust. Again, this file just contains these chains concatenated together. For validation, Python will use the first -chain it finds in the file which matches. Some "standard" root certificates are -available from various certification authorities: `CACert.org -`_, `Thawte -`_, `Verisign -`_, `Positive SSL -`_ -(used by python.org), `Equifax and GeoTrust -`_. - -In general, if you are using SSL3 or TLS1, you don't need to put the full chain -in your "CA certs" file; you only need the root certificates, and the remote -peer is supposed to furnish the other certificates necessary to chain from its -certificate to a root certificate. See :rfc:`4158` for more discussion of the -way in which certification chains can be built. +chain it finds in the file which matches. The platform's certificates file can +be used by calling :meth:`SSLContext.load_default_certs`, this is done +automatically with :func:`.create_default_context`. Combined key and certificate ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -104,6 +104,9 @@ Documentation ------------- +- Issue #21043: Remove the recommendation for specific CA organizations and to + mention the ability to load the OS certificates. + - Issue #20765: Add missing documentation for PurePath.with_name() and PurePath.with_suffix(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 00:43:15 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 00:43:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRW5z?= =?utf-8?b?dXJlIGNhbGxfc29vbigpLCBjYWxsX2xhdGVyKCkgYW5kIGNhbGxfYXQoKSBh?= =?utf-8?q?re_invoked_on_current?= Message-ID: <3ftBwz4hY0z7LjM@mail.python.org> http://hg.python.org/cpython/rev/458181a8b48a changeset: 89953:458181a8b48a branch: 3.4 parent: 89951:6f776c91da08 user: Victor Stinner date: Fri Mar 21 10:00:52 2014 +0100 summary: asyncio: Ensure call_soon(), call_later() and call_at() are invoked on current loop in debug mode. Raise a RuntimeError if the event loop of the current thread is different. The check should help to debug thread-safetly issue. Patch written by David Foster. files: Lib/asyncio/base_events.py | 23 ++++++++++- Lib/test/test_asyncio/test_base_events.py | 23 +++++++++++ 2 files changed, 45 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -259,6 +259,8 @@ """Like call_later(), but uses an absolute time.""" if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_at()") + if self._debug: + self._assert_is_current_event_loop() timer = events.TimerHandle(when, callback, args, self) heapq.heappush(self._scheduled, timer) return timer @@ -273,15 +275,34 @@ Any positional arguments after the callback will be passed to the callback when it is called. """ + return self._call_soon(callback, args, check_loop=True) + + def _call_soon(self, callback, args, check_loop): if tasks.iscoroutinefunction(callback): raise TypeError("coroutines cannot be used with call_soon()") + if self._debug and check_loop: + self._assert_is_current_event_loop() handle = events.Handle(callback, args, self) self._ready.append(handle) return handle + def _assert_is_current_event_loop(self): + """Asserts that this event loop is the current event loop. + + Non-threadsafe methods of this class make this assumption and will + likely behave incorrectly when the assumption is violated. + + Should only be called when (self._debug == True). The caller is + responsible for checking this condition for performance reasons. + """ + if events.get_event_loop() is not self: + raise RuntimeError( + "non-threadsafe operation invoked on an event loop other " + "than the current one") + def call_soon_threadsafe(self, callback, *args): """XXX""" - handle = self.call_soon(callback, *args) + handle = self._call_soon(callback, args, check_loop=False) self._write_to_self() return handle diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -136,6 +136,29 @@ # are really slow self.assertLessEqual(dt, 0.9, dt) + def test_assert_is_current_event_loop(self): + def cb(): + pass + + other_loop = base_events.BaseEventLoop() + other_loop._selector = unittest.mock.Mock() + asyncio.set_event_loop(other_loop) + + # raise RuntimeError if the event loop is different in debug mode + self.loop.set_debug(True) + with self.assertRaises(RuntimeError): + self.loop.call_soon(cb) + with self.assertRaises(RuntimeError): + self.loop.call_later(60, cb) + with self.assertRaises(RuntimeError): + self.loop.call_at(self.loop.time() + 60, cb) + + # check disabled if debug mode is disabled + self.loop.set_debug(False) + self.loop.call_soon(cb) + self.loop.call_later(60, cb) + self.loop.call_at(self.loop.time() + 60, cb) + def test_run_once_in_executor_handle(self): def cb(): pass -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 00:43:16 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 00:43:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge=2C_patch_already_applied_to_default?= Message-ID: <3ftBx062Vxz7LjM@mail.python.org> http://hg.python.org/cpython/rev/f7a40517f0ac changeset: 89954:f7a40517f0ac parent: 89952:0485552b487e parent: 89953:458181a8b48a user: Victor Stinner date: Tue Mar 25 00:43:03 2014 +0100 summary: Null merge, patch already applied to default files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 00:49:57 2014 From: python-checkins at python.org (donald.stufft) Date: Tue, 25 Mar 2014 00:49:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDQz?= =?utf-8?q?_-_Remove_CACert=2Eorg_from_the_recommendations?= Message-ID: <3ftC4j2q7pz7LjM@mail.python.org> http://hg.python.org/cpython/rev/7ef262eafecd changeset: 89955:7ef262eafecd branch: 2.7 parent: 89885:babb9479b79f user: Donald Stufft date: Mon Mar 24 19:49:42 2014 -0400 summary: Issue #21043 - Remove CACert.org from the recommendations files: Doc/library/ssl.rst | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -490,8 +490,7 @@ chain it finds in the file which matches. Some "standard" root certificates are available from various certification -authorities: `CACert.org `_, `Thawte -`_, `Verisign +authorities: `Thawte `_, `Verisign `_, `Positive SSL `_ (used by python.org), `Equifax and GeoTrust -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 05:24:52 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 25 Mar 2014 05:24:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?release=3A_use_https?= Message-ID: <3ftK9w2WP7z7LjM@mail.python.org> http://hg.python.org/release/rev/b270b4d5cf2c changeset: 86:b270b4d5cf2c user: Benjamin Peterson date: Tue Mar 25 00:24:48 2014 -0400 summary: use https files: add-to-pydotorg.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -40,9 +40,9 @@ 'containing "username:api_key".' sys.exit() -base_url = 'http://www.python.org/api/v1/' +base_url = 'https://www.python.org/api/v1/' ftp_root = '/data/ftp.python.org/pub/python/' -download_root = 'http://www.python.org/ftp/python/' +download_root = 'https://www.python.org/ftp/python/' headers = {'Authorization': 'ApiKey %s' % auth_info, 'Content-Type': 'application/json'} -- Repository URL: http://hg.python.org/release From python-checkins at python.org Tue Mar 25 09:08:39 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:08:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDU4?= =?utf-8?q?=3A_Fix_a_leak_of_file_descriptor_in_tempfile=2ENamedTemporaryF?= =?utf-8?b?aWxlKCks?= Message-ID: <3ftQ872nzgz7LjT@mail.python.org> http://hg.python.org/cpython/rev/f2f0eec4a556 changeset: 89956:f2f0eec4a556 branch: 2.7 user: Victor Stinner date: Tue Mar 25 09:08:16 2014 +0100 summary: Issue #21058: Fix a leak of file descriptor in tempfile.NamedTemporaryFile(), close the file descriptor if os.fdopen() fails files: Lib/tempfile.py | 8 ++++++-- Lib/test/test_tempfile.py | 18 ++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -460,8 +460,12 @@ flags |= _os.O_TEMPORARY (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) - file = _os.fdopen(fd, mode, bufsize) - return _TemporaryFileWrapper(file, name, delete) + try: + file = _os.fdopen(fd, mode, bufsize) + return _TemporaryFileWrapper(file, name, delete) + except Exception: + _os.close(fd) + raise if _os.name != 'posix' or _os.sys.platform == 'cygwin': # On non-POSIX and Cygwin systems, assume that we cannot unlink a file diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -771,6 +771,24 @@ pass self.assertRaises(ValueError, use_closed) + def test_no_leak_fd(self): + # Issue #21058: don't leak file descriptor when fdopen() fails + old_close = os.close + old_fdopen = os.fdopen + closed = [] + def close(fd): + closed.append(fd) + def fdopen(*args): + raise ValueError() + os.close = close + os.fdopen = fdopen + try: + self.assertRaises(ValueError, tempfile.NamedTemporaryFile) + self.assertEqual(len(closed), 1) + finally: + os.close = old_close + os.fdopen = old_fdopen + # How to test the mode and bufsize parameters? test_classes.append(test_NamedTemporaryFile) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -40,6 +40,9 @@ Library ------- +- Issue #21058: Fix a leak of file descriptor in tempfile.NamedTemporaryFile(), + close the file descriptor if os.fdopen() fails + - Issue #20283: RE pattern methods now accept the string keyword parameters as documented. The pattern and source keyword parameters are left as deprecated aliases. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:11:10 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:11:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDU4?= =?utf-8?q?=3A_NamedTemporaryFile=28=29_closes_the_FD_on_any_error=2C_not_?= =?utf-8?q?only?= Message-ID: <3ftQC23FJpz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/182f08c0dd45 changeset: 89957:182f08c0dd45 branch: 2.7 user: Victor Stinner date: Tue Mar 25 09:10:59 2014 +0100 summary: Issue #21058: NamedTemporaryFile() closes the FD on any error, not only Exception files: Lib/tempfile.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -463,7 +463,7 @@ try: file = _os.fdopen(fd, mode, bufsize) return _TemporaryFileWrapper(file, name, delete) - except Exception: + except: _os.close(fd) raise -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:19:53 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:19:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDU4?= =?utf-8?q?=3A_Fix_a_leak_of_file_descriptor_in_tempfile=2ENamedTemporaryF?= =?utf-8?b?aWxlKCks?= Message-ID: <3ftQP55Zw8z7LjX@mail.python.org> http://hg.python.org/cpython/rev/7f87c26f59ab changeset: 89958:7f87c26f59ab branch: 3.4 parent: 89953:458181a8b48a user: Victor Stinner date: Tue Mar 25 09:18:04 2014 +0100 summary: Issue #21058: Fix a leak of file descriptor in tempfile.NamedTemporaryFile(), close the file descriptor if io.open() fails files: Lib/tempfile.py | 10 +++++++--- Lib/test/test_tempfile.py | 24 ++++++++++++++++++++++++ Misc/NEWS | 4 ++++ 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -458,10 +458,14 @@ flags |= _os.O_TEMPORARY (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) - file = _io.open(fd, mode, buffering=buffering, - newline=newline, encoding=encoding) + try: + file = _io.open(fd, mode, buffering=buffering, + newline=newline, encoding=encoding) - return _TemporaryFileWrapper(file, name, delete) + return _TemporaryFileWrapper(file, name, delete) + except Exception: + _os.close(fd) + raise if _os.name != 'posix' or _os.sys.platform == 'cygwin': # On non-POSIX and Cygwin systems, assume that we cannot unlink a file diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -9,6 +9,7 @@ import warnings import contextlib import weakref +from unittest import mock import unittest from test import support, script_helper @@ -758,6 +759,17 @@ pass self.assertRaises(ValueError, use_closed) + def test_no_leak_fd(self): + # Issue #21058: don't leak file descriptor when io.pen() fails + closed = [] + def close(fd): + closed.append(fd) + + with mock.patch('os.close', side_effect=close): + with mock.patch('io.open', side_effect=ValueError): + self.assertRaises(ValueError, tempfile.NamedTemporaryFile) + self.assertEqual(len(closed), 1) + # How to test the mode and bufsize parameters? @@ -1061,6 +1073,18 @@ roundtrip("\u039B", "w+", encoding="utf-16") roundtrip("foo\r\n", "w+", newline="") + def test_no_leak_fd(self): + # Issue #21058: don't leak file descriptor when io.open() fails + closed = [] + def close(fd): + closed.append(fd) + + with mock.patch('os.close', side_effect=close): + with mock.patch('io.open', side_effect=ValueError): + self.assertRaises(ValueError, tempfile.TemporaryFile) + self.assertEqual(len(closed), 1) + + # Helper for test_del_on_shutdown class NulledModules: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,10 @@ Library ------- +- Issue #21058: Fix a leak of file descriptor in + :func:`tempfile.NamedTemporaryFile`, close the file descriptor if + :func:`io.open` fails + - Issue #21013: Enhance ssl.create_default_context() when used for server side sockets to provide better security by default. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:19:55 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:19:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321058=3A_Fix_a_leak_of_file_d?= =?utf-8?q?escriptor_in?= Message-ID: <3ftQP70GCBz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/b1e3035216f8 changeset: 89959:b1e3035216f8 parent: 89954:f7a40517f0ac parent: 89958:7f87c26f59ab user: Victor Stinner date: Tue Mar 25 09:19:14 2014 +0100 summary: (Merge 3.4) Issue #21058: Fix a leak of file descriptor in tempfile.NamedTemporaryFile(), close the file descriptor if io.open() fails files: Lib/tempfile.py | 10 +++++++--- Lib/test/test_tempfile.py | 24 ++++++++++++++++++++++++ Misc/NEWS | 4 ++++ 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -458,10 +458,14 @@ flags |= _os.O_TEMPORARY (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) - file = _io.open(fd, mode, buffering=buffering, - newline=newline, encoding=encoding) + try: + file = _io.open(fd, mode, buffering=buffering, + newline=newline, encoding=encoding) - return _TemporaryFileWrapper(file, name, delete) + return _TemporaryFileWrapper(file, name, delete) + except Exception: + _os.close(fd) + raise if _os.name != 'posix' or _os.sys.platform == 'cygwin': # On non-POSIX and Cygwin systems, assume that we cannot unlink a file diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -9,6 +9,7 @@ import warnings import contextlib import weakref +from unittest import mock import unittest from test import support, script_helper @@ -758,6 +759,17 @@ pass self.assertRaises(ValueError, use_closed) + def test_no_leak_fd(self): + # Issue #21058: don't leak file descriptor when io.pen() fails + closed = [] + def close(fd): + closed.append(fd) + + with mock.patch('os.close', side_effect=close): + with mock.patch('io.open', side_effect=ValueError): + self.assertRaises(ValueError, tempfile.NamedTemporaryFile) + self.assertEqual(len(closed), 1) + # How to test the mode and bufsize parameters? @@ -1061,6 +1073,18 @@ roundtrip("\u039B", "w+", encoding="utf-16") roundtrip("foo\r\n", "w+", newline="") + def test_no_leak_fd(self): + # Issue #21058: don't leak file descriptor when io.open() fails + closed = [] + def close(fd): + closed.append(fd) + + with mock.patch('os.close', side_effect=close): + with mock.patch('io.open', side_effect=ValueError): + self.assertRaises(ValueError, tempfile.TemporaryFile) + self.assertEqual(len(closed), 1) + + # Helper for test_del_on_shutdown class NulledModules: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,10 @@ Library ------- +- Issue #21058: Fix a leak of file descriptor in + :func:`tempfile.NamedTemporaryFile`, close the file descriptor if + :func:`io.open` fails + - Issue #21013: Enhance ssl.create_default_context() when used for server side sockets to provide better security by default. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:35:13 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 09:35:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMxODQ1?= =?utf-8?q?6=3A_Doc_fix=3A_PyDict=5FUpdate_only_works_with_dict-like_objec?= =?utf-8?q?ts=2C_not?= Message-ID: <3ftQkn6Nq9z7LjY@mail.python.org> http://hg.python.org/cpython/rev/8e1637e3a099 changeset: 89960:8e1637e3a099 branch: 3.4 parent: 89958:7f87c26f59ab user: Georg Brandl date: Tue Mar 25 09:34:30 2014 +0100 summary: Closes #18456: Doc fix: PyDict_Update only works with dict-like objects, not key-value sequences. Patch by priyapappachan. files: Doc/c-api/dict.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -201,8 +201,11 @@ .. c:function:: int PyDict_Update(PyObject *a, PyObject *b) - This is the same as ``PyDict_Merge(a, b, 1)`` in C, or ``a.update(b)`` in - Python. Return ``0`` on success or ``-1`` if an exception was raised. + This is the same as ``PyDict_Merge(a, b, 1)`` in C, and is similar to + ``a.update(b)`` in Python except that :c:func:`PyDict_Update` doesn't fall + back to the iterating over a sequence of key value pairs if the second + argument has no "keys" attribute. Return ``0`` on success or ``-1`` if an + exception was raised. .. c:function:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:35:15 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 09:35:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3ftQkq0tTHz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/9f7345e23b32 changeset: 89961:9f7345e23b32 parent: 89959:b1e3035216f8 parent: 89960:8e1637e3a099 user: Georg Brandl date: Tue Mar 25 09:34:55 2014 +0100 summary: merge with 3.4 files: Doc/c-api/dict.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -201,8 +201,11 @@ .. c:function:: int PyDict_Update(PyObject *a, PyObject *b) - This is the same as ``PyDict_Merge(a, b, 1)`` in C, or ``a.update(b)`` in - Python. Return ``0`` on success or ``-1`` if an exception was raised. + This is the same as ``PyDict_Merge(a, b, 1)`` in C, and is similar to + ``a.update(b)`` in Python except that :c:func:`PyDict_Update` doesn't fall + back to the iterating over a sequence of key value pairs if the second + argument has no "keys" attribute. Return ``0`` on success or ``-1`` if an + exception was raised. .. c:function:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:35:36 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 09:35:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMxODQ1?= =?utf-8?q?6=3A_Doc_fix=3A_PyDict=5FUpdate_only_works_with_dict-like_objec?= =?utf-8?q?ts=2C_not?= Message-ID: <3ftQlD5JVMz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/2e51845a99e2 changeset: 89962:2e51845a99e2 branch: 2.7 parent: 89957:182f08c0dd45 user: Georg Brandl date: Tue Mar 25 09:34:30 2014 +0100 summary: Closes #18456: Doc fix: PyDict_Update only works with dict-like objects, not key-value sequences. Patch by priyapappachan. files: Doc/c-api/dict.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -210,8 +210,11 @@ .. c:function:: int PyDict_Update(PyObject *a, PyObject *b) - This is the same as ``PyDict_Merge(a, b, 1)`` in C, or ``a.update(b)`` in - Python. Return ``0`` on success or ``-1`` if an exception was raised. + This is the same as ``PyDict_Merge(a, b, 1)`` in C, and is similar to + ``a.update(b)`` in Python except that :c:func:`PyDict_Update` doesn't fall + back to the iterating over a sequence of key value pairs if the second + argument has no "keys" attribute. Return ``0`` on success or ``-1`` if an + exception was raised. .. versionadded:: 2.2 -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Mar 25 09:39:34 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 25 Mar 2014 09:39:34 +0100 Subject: [Python-checkins] Daily reference leaks (f7a40517f0ac): sum=0 Message-ID: results for f7a40517f0ac on branch "default" -------------------------------------------- test_site leaked [0, -2, 2] references, sum=0 test_site leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogluyjrz', '-x'] From python-checkins at python.org Tue Mar 25 09:40:50 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:40:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDA2?= =?utf-8?q?=3A_asyncio_doc=3A_reorganize_subprocess_doc?= Message-ID: <3ftQsG13G2z7Ljd@mail.python.org> http://hg.python.org/cpython/rev/dd2c7cca5980 changeset: 89963:dd2c7cca5980 branch: 3.4 parent: 89960:8e1637e3a099 user: Victor Stinner date: Tue Mar 25 09:40:26 2014 +0100 summary: Issue #21006: asyncio doc: reorganize subprocess doc files: Doc/library/asyncio-eventloop.rst | 84 +--------------- Doc/library/asyncio-subprocess.rst | 94 +++++++++++++++++- 2 files changed, 94 insertions(+), 84 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -450,84 +450,8 @@ :meth:`socket.getnameinfo` function but non-blocking. -Running subprocesses --------------------- - -Run subprocesses asynchronously using the :mod:`subprocess` module. - -.. note:: - - On Windows, the default event loop uses - :class:`selectors.SelectSelector` which only supports sockets. The - :class:`ProactorEventLoop` should be used to support subprocesses. - -.. note:: - - On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector` - does not support character devices like PTY, whereas it is used by the - default event loop. The :class:`SelectorEventLoop` can be used with - :class:`SelectSelector` or :class:`PollSelector` to handle character devices - on Mac OS X 10.6 (Snow Leopard) and later. - -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - - Create a subprocess from one or more string arguments, where the first string - specifies the program to execute, and the remaining strings specify the - program's arguments. (Thus, together the string arguments form the - ``sys.argv`` value of the program, assuming it is a Python script.) This is - similar to the standard library :class:`subprocess.Popen` class called with - shell=False and the list of strings passed as the first argument; - however, where :class:`~subprocess.Popen` takes a single argument which is - list of strings, :func:`subprocess_exec` takes multiple string arguments. - - Other parameters: - - * *stdin*: Either a file-like object representing the pipe to be connected - to the subprocess's standard input stream using - :meth:`~BaseEventLoop.connect_write_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. - - * *stdout*: Either a file-like object representing the pipe to be connected - to the subprocess's standard output stream using - :meth:`~BaseEventLoop.connect_read_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. - - * *stderr*: Either a file-like object representing the pipe to be connected - to the subprocess's standard error stream using - :meth:`~BaseEventLoop.connect_read_pipe`, or one of the constants - :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. - By default a new pipe will be created and connected. When - :const:`subprocess.STDOUT` is specified, the subprocess's standard error - stream will be connected to the same pipe as the standard output stream. - - * All other keyword arguments are passed to :class:`subprocess.Popen` - without interpretation, except for *bufsize*, *universal_newlines* and - *shell*, which should not be specified at all. - - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. - - This method is a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - - Create a subprocess from *cmd*, which is a string using the platform's - "shell" syntax. This is similar to the standard library - :class:`subprocess.Popen` class called with ``shell=True``. - - See :meth:`~BaseEventLoop.subprocess_exec` for more details about - the remaining arguments. - - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. - - This method is a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. +Connect pipes +------------- .. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) @@ -553,8 +477,8 @@ .. seealso:: - The :func:`create_subprocess_exec` and :func:`create_subprocess_shell` - functions. + The :meth:`BaseEventLoop.subprocess_exec` and + :meth:`BaseEventLoop.subprocess_shell` methods. UNIX signals diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -3,29 +3,115 @@ Subprocess ========== -Create a subprocess -------------------- +Operating system support +------------------------ + +On Windows, the default event loop uses :class:`selectors.SelectSelector` +which only supports sockets. The :class:`ProactorEventLoop` should be used to +support subprocesses. + +On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector` +does not support character devices like PTY, whereas it is used by the +default event loop. The :class:`SelectorEventLoop` can be used with +:class:`SelectSelector` or :class:`PollSelector` to handle character +devices on Mac OS X 10.6 (Snow Leopard) and later. + + +Create a subprocess: high-level API using Process +------------------------------------------------- .. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) Run the shell command *cmd* given as a string. Return a :class:`~asyncio.subprocess.Process` instance. + The optional *limit* parameter sets the buffer limit passed to the + :class:`StreamReader`. + This function is a :ref:`coroutine `. .. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) Create a subprocess. Return a :class:`~asyncio.subprocess.Process` instance. + The optional *limit* parameter sets the buffer limit passed to the + :class:`StreamReader`. + This function is a :ref:`coroutine `. Use the :meth:`BaseEventLoop.connect_read_pipe` and :meth:`BaseEventLoop.connect_write_pipe` methods to connect pipes. + +Create a subprocess: low-level API using subprocess.Popen +--------------------------------------------------------- + +Run subprocesses asynchronously using the :mod:`subprocess` module. + +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from one or more string arguments, where the first string + specifies the program to execute, and the remaining strings specify the + program's arguments. (Thus, together the string arguments form the + ``sys.argv`` value of the program, assuming it is a Python script.) This is + similar to the standard library :class:`subprocess.Popen` class called with + shell=False and the list of strings passed as the first argument; + however, where :class:`~subprocess.Popen` takes a single argument which is + list of strings, :func:`subprocess_exec` takes multiple string arguments. + + Other parameters: + + * *stdin*: Either a file-like object representing the pipe to be connected + to the subprocess's standard input stream using + :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stdout*: Either a file-like object representing the pipe to be connected + to the subprocess's standard output stream using + :meth:`~BaseEventLoop.connect_read_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stderr*: Either a file-like object representing the pipe to be connected + to the subprocess's standard error stream using + :meth:`~BaseEventLoop.connect_read_pipe`, or one of the constants + :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. + By default a new pipe will be created and connected. When + :const:`subprocess.STDOUT` is specified, the subprocess's standard error + stream will be connected to the same pipe as the standard output stream. + + * All other keyword arguments are passed to :class:`subprocess.Popen` + without interpretation, except for *bufsize*, *universal_newlines* and + *shell*, which should not be specified at all. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. + + This method is a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from *cmd*, which is a string using the platform's + "shell" syntax. This is similar to the standard library + :class:`subprocess.Popen` class called with ``shell=True``. + + See :meth:`~BaseEventLoop.subprocess_exec` for more details about + the remaining arguments. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. + + This method is a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + .. seealso:: - The :meth:`BaseEventLoop.subprocess_exec` and - :meth:`BaseEventLoop.subprocess_shell` methods. + The :meth:`BaseEventLoop.connect_read_pipe` and + :meth:`BaseEventLoop.connect_write_pipe` methods. Constants -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 09:40:51 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 09:40:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321006=3A_asyncio_doc=3A_reorg?= =?utf-8?q?anize_subprocess_doc?= Message-ID: <3ftQsH3plPz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/c45b124e9af4 changeset: 89964:c45b124e9af4 parent: 89961:9f7345e23b32 parent: 89963:dd2c7cca5980 user: Victor Stinner date: Tue Mar 25 09:40:39 2014 +0100 summary: (Merge 3.4) Issue #21006: asyncio doc: reorganize subprocess doc files: Doc/library/asyncio-eventloop.rst | 84 +--------------- Doc/library/asyncio-subprocess.rst | 94 +++++++++++++++++- 2 files changed, 94 insertions(+), 84 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -450,84 +450,8 @@ :meth:`socket.getnameinfo` function but non-blocking. -Running subprocesses --------------------- - -Run subprocesses asynchronously using the :mod:`subprocess` module. - -.. note:: - - On Windows, the default event loop uses - :class:`selectors.SelectSelector` which only supports sockets. The - :class:`ProactorEventLoop` should be used to support subprocesses. - -.. note:: - - On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector` - does not support character devices like PTY, whereas it is used by the - default event loop. The :class:`SelectorEventLoop` can be used with - :class:`SelectSelector` or :class:`PollSelector` to handle character devices - on Mac OS X 10.6 (Snow Leopard) and later. - -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - - Create a subprocess from one or more string arguments, where the first string - specifies the program to execute, and the remaining strings specify the - program's arguments. (Thus, together the string arguments form the - ``sys.argv`` value of the program, assuming it is a Python script.) This is - similar to the standard library :class:`subprocess.Popen` class called with - shell=False and the list of strings passed as the first argument; - however, where :class:`~subprocess.Popen` takes a single argument which is - list of strings, :func:`subprocess_exec` takes multiple string arguments. - - Other parameters: - - * *stdin*: Either a file-like object representing the pipe to be connected - to the subprocess's standard input stream using - :meth:`~BaseEventLoop.connect_write_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. - - * *stdout*: Either a file-like object representing the pipe to be connected - to the subprocess's standard output stream using - :meth:`~BaseEventLoop.connect_read_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. - - * *stderr*: Either a file-like object representing the pipe to be connected - to the subprocess's standard error stream using - :meth:`~BaseEventLoop.connect_read_pipe`, or one of the constants - :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. - By default a new pipe will be created and connected. When - :const:`subprocess.STDOUT` is specified, the subprocess's standard error - stream will be connected to the same pipe as the standard output stream. - - * All other keyword arguments are passed to :class:`subprocess.Popen` - without interpretation, except for *bufsize*, *universal_newlines* and - *shell*, which should not be specified at all. - - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. - - This method is a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) - - Create a subprocess from *cmd*, which is a string using the platform's - "shell" syntax. This is similar to the standard library - :class:`subprocess.Popen` class called with ``shell=True``. - - See :meth:`~BaseEventLoop.subprocess_exec` for more details about - the remaining arguments. - - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. - - This method is a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. +Connect pipes +------------- .. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) @@ -553,8 +477,8 @@ .. seealso:: - The :func:`create_subprocess_exec` and :func:`create_subprocess_shell` - functions. + The :meth:`BaseEventLoop.subprocess_exec` and + :meth:`BaseEventLoop.subprocess_shell` methods. UNIX signals diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -3,29 +3,115 @@ Subprocess ========== -Create a subprocess -------------------- +Operating system support +------------------------ + +On Windows, the default event loop uses :class:`selectors.SelectSelector` +which only supports sockets. The :class:`ProactorEventLoop` should be used to +support subprocesses. + +On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector` +does not support character devices like PTY, whereas it is used by the +default event loop. The :class:`SelectorEventLoop` can be used with +:class:`SelectSelector` or :class:`PollSelector` to handle character +devices on Mac OS X 10.6 (Snow Leopard) and later. + + +Create a subprocess: high-level API using Process +------------------------------------------------- .. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) Run the shell command *cmd* given as a string. Return a :class:`~asyncio.subprocess.Process` instance. + The optional *limit* parameter sets the buffer limit passed to the + :class:`StreamReader`. + This function is a :ref:`coroutine `. .. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) Create a subprocess. Return a :class:`~asyncio.subprocess.Process` instance. + The optional *limit* parameter sets the buffer limit passed to the + :class:`StreamReader`. + This function is a :ref:`coroutine `. Use the :meth:`BaseEventLoop.connect_read_pipe` and :meth:`BaseEventLoop.connect_write_pipe` methods to connect pipes. + +Create a subprocess: low-level API using subprocess.Popen +--------------------------------------------------------- + +Run subprocesses asynchronously using the :mod:`subprocess` module. + +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from one or more string arguments, where the first string + specifies the program to execute, and the remaining strings specify the + program's arguments. (Thus, together the string arguments form the + ``sys.argv`` value of the program, assuming it is a Python script.) This is + similar to the standard library :class:`subprocess.Popen` class called with + shell=False and the list of strings passed as the first argument; + however, where :class:`~subprocess.Popen` takes a single argument which is + list of strings, :func:`subprocess_exec` takes multiple string arguments. + + Other parameters: + + * *stdin*: Either a file-like object representing the pipe to be connected + to the subprocess's standard input stream using + :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stdout*: Either a file-like object representing the pipe to be connected + to the subprocess's standard output stream using + :meth:`~BaseEventLoop.connect_read_pipe`, or the constant + :const:`subprocess.PIPE` (the default). By default a new pipe will be + created and connected. + + * *stderr*: Either a file-like object representing the pipe to be connected + to the subprocess's standard error stream using + :meth:`~BaseEventLoop.connect_read_pipe`, or one of the constants + :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. + By default a new pipe will be created and connected. When + :const:`subprocess.STDOUT` is specified, the subprocess's standard error + stream will be connected to the same pipe as the standard output stream. + + * All other keyword arguments are passed to :class:`subprocess.Popen` + without interpretation, except for *bufsize*, *universal_newlines* and + *shell*, which should not be specified at all. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. + + This method is a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from *cmd*, which is a string using the platform's + "shell" syntax. This is similar to the standard library + :class:`subprocess.Popen` class called with ``shell=True``. + + See :meth:`~BaseEventLoop.subprocess_exec` for more details about + the remaining arguments. + + Returns a pair of ``(transport, protocol)``, where *transport* is an + instance of :class:`BaseSubprocessTransport`. + + This method is a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + .. seealso:: - The :meth:`BaseEventLoop.subprocess_exec` and - :meth:`BaseEventLoop.subprocess_shell` methods. + The :meth:`BaseEventLoop.connect_read_pipe` and + :meth:`BaseEventLoop.connect_write_pipe` methods. Constants -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 10:13:11 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 10:13:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMTA0?= =?utf-8?q?5=3A_make_the_Qt_help_build_not_look_completely_stupid?= Message-ID: <3ftRZb0qjbz7LjM@mail.python.org> http://hg.python.org/cpython/rev/d6831d94dd1e changeset: 89965:d6831d94dd1e branch: 2.7 parent: 89962:2e51845a99e2 user: Georg Brandl date: Tue Mar 25 10:12:47 2014 +0100 summary: Closes #21045: make the Qt help build not look completely stupid files: Doc/tools/sphinxext/layout.html | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/tools/sphinxext/layout.html b/Doc/tools/sphinxext/layout.html --- a/Doc/tools/sphinxext/layout.html +++ b/Doc/tools/sphinxext/layout.html @@ -12,11 +12,19 @@ {%- endif %} {% endblock %} +{% block relbar1 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} +{% block relbar2 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} {% block extrahead %} {% if not embedded %}{% endif %} {% if versionswitcher is defined and not embedded %}{% endif %} {{ super() }} + {% if builder == 'qthelp' %} + + {% endif %} {% endblock %} {% block footer %}
-- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 10:13:56 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 10:13:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMTA0?= =?utf-8?q?5=3A_make_the_Qt_help_build_not_look_completely_stupid?= Message-ID: <3ftRbS6ljrz7LjM@mail.python.org> http://hg.python.org/cpython/rev/2a6c33592e40 changeset: 89966:2a6c33592e40 branch: 3.4 parent: 89963:dd2c7cca5980 user: Georg Brandl date: Tue Mar 25 10:12:47 2014 +0100 summary: Closes #21045: make the Qt help build not look completely stupid files: Doc/tools/sphinxext/layout.html | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/tools/sphinxext/layout.html b/Doc/tools/sphinxext/layout.html --- a/Doc/tools/sphinxext/layout.html +++ b/Doc/tools/sphinxext/layout.html @@ -12,6 +12,8 @@ {%- endif %} {% endblock %} +{% block relbar1 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} +{% block relbar2 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} {% block extrahead %} {% if not embedded %}{% endif %} -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 10:13:58 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 25 Mar 2014 10:13:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3ftRbV1GKmz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/1d523b48efa3 changeset: 89967:1d523b48efa3 parent: 89964:c45b124e9af4 parent: 89966:2a6c33592e40 user: Georg Brandl date: Tue Mar 25 10:13:44 2014 +0100 summary: merge with 3.4 files: Doc/tools/sphinxext/layout.html | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/tools/sphinxext/layout.html b/Doc/tools/sphinxext/layout.html --- a/Doc/tools/sphinxext/layout.html +++ b/Doc/tools/sphinxext/layout.html @@ -12,6 +12,8 @@ {%- endif %} {% endblock %} +{% block relbar1 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} +{% block relbar2 %} {% if builder != 'qthelp' %} {{ relbar() }} {% endif %} {% endblock %} {% block extrahead %} {% if not embedded %}{% endif %} -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:22:37 2014 From: python-checkins at python.org (vinay.sajip) Date: Tue, 25 Mar 2014 12:22:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Updated_extern?= =?utf-8?q?al_link_in_documentation=2E?= Message-ID: <3ftVRx574kz7LjV@mail.python.org> http://hg.python.org/cpython/rev/a22e5abf95db changeset: 89968:a22e5abf95db branch: 2.7 parent: 89965:d6831d94dd1e user: Vinay Sajip date: Tue Mar 25 11:20:22 2014 +0000 summary: Updated external link in documentation. files: Doc/howto/logging-cookbook.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -706,7 +706,7 @@ } For more information about this configuration, you can see the `relevant -section `_ +section `_ of the Django documentation. Inserting a BOM into messages sent to a SysLogHandler -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:22:38 2014 From: python-checkins at python.org (vinay.sajip) Date: Tue, 25 Mar 2014 12:22:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Updated_extern?= =?utf-8?q?al_link_in_documentation=2E?= Message-ID: <3ftVRy6ZQxz7LjV@mail.python.org> http://hg.python.org/cpython/rev/2d953924979c changeset: 89969:2d953924979c branch: 3.4 parent: 89966:2a6c33592e40 user: Vinay Sajip date: Tue Mar 25 11:21:41 2014 +0000 summary: Updated external link in documentation. files: Doc/howto/logging-cookbook.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1308,7 +1308,7 @@ } For more information about this configuration, you can see the `relevant -section `_ +section `_ of the Django documentation. .. _cookbook-rotator-namer: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:22:40 2014 From: python-checkins at python.org (vinay.sajip) Date: Tue, 25 Mar 2014 12:22:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merged_documentation_update_from_3=2E4=2E?= Message-ID: <3ftVS01CyWz7LkB@mail.python.org> http://hg.python.org/cpython/rev/a9389e1fa4e6 changeset: 89970:a9389e1fa4e6 parent: 89967:1d523b48efa3 parent: 89969:2d953924979c user: Vinay Sajip date: Tue Mar 25 11:22:24 2014 +0000 summary: Merged documentation update from 3.4. files: Doc/howto/logging-cookbook.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1308,7 +1308,7 @@ } For more information about this configuration, you can see the `relevant -section `_ +section `_ of the Django documentation. .. _cookbook-rotator-namer: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:35:39 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:35:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzEyMjA5?= =?utf-8?q?=3A_Minor_edits_to_faulthandler_doc=2E_Patch_written_by_=C3=89r?= =?utf-8?q?ic_Araujo=2E?= Message-ID: <3ftVkz12mqz7LjS@mail.python.org> http://hg.python.org/cpython/rev/a97dcdee35f9 changeset: 89971:a97dcdee35f9 branch: 3.4 parent: 89969:2d953924979c user: Victor Stinner date: Tue Mar 25 12:33:56 2014 +0100 summary: Issue #12209: Minor edits to faulthandler doc. Patch written by ?ric Araujo. files: Doc/library/faulthandler.rst | 28 ++++++++++++++--------- 1 files changed, 17 insertions(+), 11 deletions(-) diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -11,7 +11,7 @@ install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS`, and :const:`SIGILL` signals. You can also enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER` environment -variable or by using :option:`-X` ``faulthandler`` command line option. +variable or by using the :option:`-X` ``faulthandler`` command line option. The fault handler is compatible with system fault handlers like Apport or the Windows fault handler. The module uses an alternative stack for signal handlers @@ -39,8 +39,8 @@ Python is deadlocked. -Dump the traceback ------------------- +Dumping the traceback +--------------------- .. function:: dump_traceback(file=sys.stderr, all_threads=True) @@ -69,8 +69,8 @@ Check if the fault handler is enabled. -Dump the tracebacks after a timeout ------------------------------------ +Dumping the tracebacks after a timeout +-------------------------------------- .. function:: dump_traceback_later(timeout, repeat=False, file=sys.stderr, exit=False) @@ -90,8 +90,8 @@ Cancel the last call to :func:`dump_traceback_later`. -Dump the traceback on a user signal ------------------------------------ +Dumping the traceback on a user signal +-------------------------------------- .. function:: register(signum, file=sys.stderr, all_threads=True, chain=False) @@ -110,8 +110,8 @@ Not available on Windows. -File descriptor issue ---------------------- +Issue with file descriptors +--------------------------- :func:`enable`, :func:`dump_traceback_later` and :func:`register` keep the file descriptor of their *file* argument. If the file is closed and its file @@ -123,9 +123,15 @@ Example ------- -Example of a segmentation fault on Linux: :: +.. highlight:: sh - $ python -q -X faulthandler +Example of a segmentation fault on Linux with and without enabling the fault +handler:: + + $ python3 -c "import ctypes; ctypes.string_at(0)" + Segmentation fault + + $ python3 -q -X faulthandler >>> import ctypes >>> ctypes.string_at(0) Fatal Python error: Segmentation fault -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:35:40 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:35:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2312209=3A_Minor_edits_to_fault?= =?utf-8?q?handler_doc=2E_Patch_written_by?= Message-ID: <3ftVl03z4sz7LkB@mail.python.org> http://hg.python.org/cpython/rev/6f80ca0012ae changeset: 89972:6f80ca0012ae parent: 89970:a9389e1fa4e6 parent: 89971:a97dcdee35f9 user: Victor Stinner date: Tue Mar 25 12:34:17 2014 +0100 summary: (Merge 3.4) Issue #12209: Minor edits to faulthandler doc. Patch written by ?ric Araujo. files: Doc/library/faulthandler.rst | 28 ++++++++++++++--------- 1 files changed, 17 insertions(+), 11 deletions(-) diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -11,7 +11,7 @@ install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS`, and :const:`SIGILL` signals. You can also enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER` environment -variable or by using :option:`-X` ``faulthandler`` command line option. +variable or by using the :option:`-X` ``faulthandler`` command line option. The fault handler is compatible with system fault handlers like Apport or the Windows fault handler. The module uses an alternative stack for signal handlers @@ -39,8 +39,8 @@ Python is deadlocked. -Dump the traceback ------------------- +Dumping the traceback +--------------------- .. function:: dump_traceback(file=sys.stderr, all_threads=True) @@ -69,8 +69,8 @@ Check if the fault handler is enabled. -Dump the tracebacks after a timeout ------------------------------------ +Dumping the tracebacks after a timeout +-------------------------------------- .. function:: dump_traceback_later(timeout, repeat=False, file=sys.stderr, exit=False) @@ -90,8 +90,8 @@ Cancel the last call to :func:`dump_traceback_later`. -Dump the traceback on a user signal ------------------------------------ +Dumping the traceback on a user signal +-------------------------------------- .. function:: register(signum, file=sys.stderr, all_threads=True, chain=False) @@ -110,8 +110,8 @@ Not available on Windows. -File descriptor issue ---------------------- +Issue with file descriptors +--------------------------- :func:`enable`, :func:`dump_traceback_later` and :func:`register` keep the file descriptor of their *file* argument. If the file is closed and its file @@ -123,9 +123,15 @@ Example ------- -Example of a segmentation fault on Linux: :: +.. highlight:: sh - $ python -q -X faulthandler +Example of a segmentation fault on Linux with and without enabling the fault +handler:: + + $ python3 -c "import ctypes; ctypes.string_at(0)" + Segmentation fault + + $ python3 -q -X faulthandler >>> import ctypes >>> ctypes.string_at(0) Fatal Python error: Segmentation fault -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:51:25 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:51:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDM4?= =?utf-8?q?=3A_Cleanup_test=5Fepoll=2Epy?= Message-ID: <3ftW5925HQz7LjV@mail.python.org> http://hg.python.org/cpython/rev/3cd216b56599 changeset: 89973:3cd216b56599 branch: 3.4 parent: 89971:a97dcdee35f9 user: Victor Stinner date: Tue Mar 25 12:49:53 2014 +0100 summary: Issue #21038: Cleanup test_epoll.py Remove useless test and unused variables. Initial patch by Andreas Schwab. files: Lib/test/test_epoll.py | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_epoll.py b/Lib/test/test_epoll.py --- a/Lib/test/test_epoll.py +++ b/Lib/test/test_epoll.py @@ -174,11 +174,8 @@ expected.sort() self.assertEqual(events, expected) - self.assertFalse(then - now > 0.01, then - now) - now = time.time() events = ep.poll(timeout=2.1, maxevents=4) - then = time.time() self.assertFalse(events) client.send(b"Hello!") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:51:26 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:51:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDM4?= =?utf-8?q?=3A_Use_monotonic_clock_to_compute_timeout=2C_not_the_system_cl?= =?utf-8?q?ock?= Message-ID: <3ftW5B3s0dz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/945d7dd3b455 changeset: 89974:945d7dd3b455 branch: 3.4 user: Victor Stinner date: Tue Mar 25 12:50:50 2014 +0100 summary: Issue #21038: Use monotonic clock to compute timeout, not the system clock files: Lib/test/test_epoll.py | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_epoll.py b/Lib/test/test_epoll.py --- a/Lib/test/test_epoll.py +++ b/Lib/test/test_epoll.py @@ -163,9 +163,9 @@ ep.register(client.fileno(), select.EPOLLIN | select.EPOLLOUT | select.EPOLLET) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.1, then - now) events.sort() @@ -181,9 +181,9 @@ client.send(b"Hello!") server.send(b"world!!!") - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) events.sort() @@ -195,9 +195,9 @@ ep.unregister(client.fileno()) ep.modify(server.fileno(), select.EPOLLOUT) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) expected = [(server.fileno(), select.EPOLLOUT)] @@ -214,9 +214,9 @@ ep = select.epoll(16) ep.register(server) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) server.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:51:27 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:51:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40?= Message-ID: <3ftW5C5bJNz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/6a0def54c63d changeset: 89975:6a0def54c63d parent: 89972:6f80ca0012ae parent: 89974:945d7dd3b455 user: Victor Stinner date: Tue Mar 25 12:51:08 2014 +0100 summary: Merge 3.4 files: Lib/test/test_epoll.py | 19 ++++++++----------- 1 files changed, 8 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_epoll.py b/Lib/test/test_epoll.py --- a/Lib/test/test_epoll.py +++ b/Lib/test/test_epoll.py @@ -163,9 +163,9 @@ ep.register(client.fileno(), select.EPOLLIN | select.EPOLLOUT | select.EPOLLET) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.1, then - now) events.sort() @@ -174,19 +174,16 @@ expected.sort() self.assertEqual(events, expected) - self.assertFalse(then - now > 0.01, then - now) - now = time.time() events = ep.poll(timeout=2.1, maxevents=4) - then = time.time() self.assertFalse(events) client.send(b"Hello!") server.send(b"world!!!") - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) events.sort() @@ -198,9 +195,9 @@ ep.unregister(client.fileno()) ep.modify(server.fileno(), select.EPOLLOUT) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) expected = [(server.fileno(), select.EPOLLOUT)] @@ -217,9 +214,9 @@ ep = select.epoll(16) ep.register(server) - now = time.time() + now = time.monotonic() events = ep.poll(1, 4) - then = time.time() + then = time.monotonic() self.assertFalse(then - now > 0.01) server.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 12:54:03 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 12:54:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDM4?= =?utf-8?q?=3A_Cleanup_test=5Fepoll=2Epy?= Message-ID: <3ftW8C6dCKz7LjV@mail.python.org> http://hg.python.org/cpython/rev/d09032a9adee changeset: 89976:d09032a9adee branch: 2.7 parent: 89968:a22e5abf95db user: Victor Stinner date: Tue Mar 25 12:53:47 2014 +0100 summary: Issue #21038: Cleanup test_epoll.py Remove useless test and unused variables. Initial patch by Andreas Schwab. files: Lib/test/test_epoll.py | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_epoll.py b/Lib/test/test_epoll.py --- a/Lib/test/test_epoll.py +++ b/Lib/test/test_epoll.py @@ -164,11 +164,8 @@ expected.sort() self.assertEqual(events, expected) - self.assertFalse(then - now > 0.01, then - now) - now = time.time() events = ep.poll(timeout=2.1, maxevents=4) - then = time.time() self.assertFalse(events) client.send("Hello!") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 13:42:26 2014 From: python-checkins at python.org (nick.coghlan) Date: Tue, 25 Mar 2014 13:42:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_further_simplify_t?= =?utf-8?q?he_proposal?= Message-ID: <3ftXD23NMLz7M7q@mail.python.org> http://hg.python.org/peps/rev/4eb8de143801 changeset: 5428:4eb8de143801 user: Nick Coghlan date: Tue Mar 25 22:40:26 2014 +1000 summary: PEP 466: further simplify the proposal - dropped the "legacy ssl" branch idea again - explained why selective backports are a bad idea - better explained how long term support differs from short term - better explained some downstream constraints - explained that downstream redistributors have been respecting our upstream policies, potentially against their own better judgement - used Red Hat's support cycle as a reference model - removed the snarky comments that hit unintended targets - note the need for Windows specific contributions as an open question - not the general need for contributions of time as an open question - last open question is whether or not we're happy this is the best available solution given downstream constraints files: pep-0466.txt | 267 ++++++++++++++++++++++++++------------ 1 files changed, 179 insertions(+), 88 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -25,10 +25,13 @@ protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. -Accordingly, this PEP relaxes the normal restrictions by allowing -enhancements to be applied in Python 2.7 maintenance releases for standard -library components that have implications for the overall security of the -internet. In particular, the exception will apply to: +In recognition of the additional practical considerations that have arisen +during the 4+ year maintenance cycle for Python 2.7, this PEP allows +Python 2.7 standard library components that have implications for the +overall security of the internet to be updated in line with the +corresponding Python 3 feature releases. + +Specifically, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module @@ -39,52 +42,46 @@ and Mac OS X Changes to these modules will still need to undergo normal backwards -compatibility assessments, but otherwise they will be kept entirely aligned -with the latest feature release of their Python 3 counterparts. This is -designed to make it easier to implement secure networked software in -Python, even for software that currently needs to remain compatible with -the Python 2 series (which includes a lot of network infrastructure -software). - -The development branches will be arranged in such a way that even though -any further Python 2.7 releases published by the core development team -provide updated network security infrastructure, it will remain possible -for downstream redistributors to publish "Python 2.7 with legacy SSL -infrastructure" if they choose to do so. +compatibility assessments to ensure their default behaviour remains +consistent with earlier Python 2.7 releases, but otherwise they will be +kept entirely aligned with the latest feature release of their Python 3 +counterparts. This is designed to make it easier to implement secure +networked software in Python, even for software that currently needs to +remain compatible with the Python 2 series (which includes a lot of +network infrastructure software). While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure -keeps pace with the evolution of the internet, or else disclaim support -for the use of older versions in roles that involving connecting -directly to the public internet. +keeps pace with the evolution of the internet, or else explicitly +disclaim support for the use of older versions in roles that involve +connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are -granted a blanket exemption to the restriction against adding new features -in maintenance releases, for the purpose of keeping their APIs aligned with -their counterparts in the latest feature release of Python 3: +granted a blanket exemption to the normal restriction against adding new +features in Python 2.7 maintenance releases, for the purpose of keeping +their APIs aligned with their counterparts in the latest feature release +of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module -This exemption applies to *all* proposals to backport backwards compatible -changes in these modules to Python 2.7 maintenance releases. This choice is -made deliberately to ensure that the "feature or fix?" argument isn't simply -replaced by a "security related or not?" argument. These particular modules -are inherently security related, and all enhancements to them improve -Python's capabilities as a platform for development of secure networked -software. +Under this exemption, these modules are updated to provide identical +functionality to their Python 3 counterparts after every new Python 3 +feature release. The default behaviour of the backported modules will be +adjusted as appropriate to meet the backwards compatibility requirements +of a Python 2.7 maintenance release. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers -for new maintenance releases of CPython. +for new maintenance releases of Python 2.7. Note that the ``sha`` and ``md5`` modules are not covered by this policy, but have been deprecated in favour of ``hashlib`` since Python 2.5 and have @@ -107,18 +104,18 @@ Backwards Compatibility Considerations ====================================== -This PEP does *not* grant any general exemptions to the usual backwards -compatibility policy for maintenance releases. Instead, by explicitly -encouraging the use of feature based checks and explicitly opting in to -less secure configurations, it is designed to make it easier to provide -more "secure by default" behaviour in future feature releases, while still +This PEP does *not* grant Python 2.7 any general exemptions to the usual +backwards compatibility policy for maintenance releases. Instead, by +explicitly encouraging the use of feature based checks and explicitly +opting in to less secure configurations, it is designed to make it easier +to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to -a new maintenance release. +a new Python 2.7 maintenance release. In *all* cases where this policy is applied to backport enhancements to -maintenance releases, it MUST be possible to write cross-version compatible -code that operates by "feature detection" (for example, checking for -particular attributes in the module), without needing to explicitly check +Python 2.7 maintenance releases, it MUST be possible to write cross-version +compatible code that operates by "feature detection" (for example, checking +for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning @@ -155,7 +152,7 @@ Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, -the specific version of OpenSSL used already varies, as Python dynamically +the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are @@ -191,17 +188,18 @@ Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those -changes back to the upstream community in cases where they are likely to -have a broad impact that helps improve the security of the internet as a -whole, rather than sitting back and waiting for unpaid volunteers to do it -for them. +changes back to the upstream Python community in cases where they are likely +to have a broad impact that helps improve the security of the internet as a +whole, with the assurance that the existing core development team not only +won't object to such contributions, but will actively encourage their +incorporation into the next Python 2.7 maintenance release. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" -section in their documentation. +section in their documentation in order to take advantage of this policy. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where @@ -230,36 +228,31 @@ ------------------- Third party integration testing services should offer users the ability -to test against both the latest Python 2.7 maintenance release and the -latest "Python 2.7 with legacy SSL infrastructure" release, to ensure that -libraries, frameworks and applications handle the legacy security -infrastructure correctly (either failing or degrading gracefully, depending -on the security sensitivity of the software). +to test against specific Python 2.7 maintenance releases, to ensure that +libraries, frameworks and applications can still test their handling of the +legacy security infrastructure correctly (either failing or degrading +gracefully, depending on the security sensitivity of the software), even +after the latest Python 2.7 maintenance release has been synchronised with +a new Python 3 feature release for the modules covered by this policy. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where -the risk of latent security defects is more tolerated than the risk of -regressions in maintenance releases. This policy largely excludes these -environments from consideration where the modules covered by the exemption -are concerned. +the risk of latent security defects is more tolerated than even a slightly +increased risk of regressions in maintenance releases. This policy largely +excludes these environments from consideration where the modules covered by +the exemption are concerned - this approach is entirely inappropriate for +software connected to the public internet, and defence in depth security +principles suggest that it is not appropriate for most private networks +either. -However, one concession is made for the benefit of such environments: while -the main ``2.7`` branch in Mercurial will include the updated network -security infrastructure, the development process will be updated to include -a ``2.7-legacy-ssl`` branch with the more limited feature set of the -original 2.7 network security infrastructure, allowing downstream -redistributors to continue to provide Python 2.7 with the legacy SSL -infrastructure if they choose to do so. This branch will be tested -on the CPython continuous integration infrastructure, but no releases will -be made from it by the core development team. - -As noted above, corporate redistributors and users are expected to provide -the additional development effort needed to make this practical. It is not -acceptable to expect volunteer contributors to resolve a problem created -largely by conservative corporate development practices. +Downstream redistributors may still choose to cater to such environments, +but they will need to handle the process of downgrading the security +related modules and doing the associated regression testing themselves. +The main CPython continuous integration infrastructure will not cover this +scenario. Evolution of this Policy @@ -279,8 +272,9 @@ The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end -users became as cautious about updating to new Python maintenance releases -as they are about updating to new feature releases. +users became as cautious about updating to new Python 2.7 maintenance +releases as they are about updating to new feature releases within the +same release series. Motivation and Rationale @@ -312,12 +306,13 @@ a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these -concerns on Linux platforms, it doesn't address all of them, and in the -case of the binary installers for Windows and Mac OS X that are published -on python.org, the version of OpenSSL used is entirely within the control -of the Python core development team, and currently limited to OpenSSL -maintenance releases for the version initially shipped with the corresponding -Python feature release. +concerns on Linux platforms, it doesn't address all of them (in particular, +it is still difficult for sotware to explicitly require some higher level +security settings). In the case of the binary installers for Windows and +Mac OS X that are published on python.org, the version of OpenSSL used is +entirely within the control of the Python core development team, but is +currently limited to OpenSSL maintenance releases for the version initially +shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now @@ -342,7 +337,7 @@ on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison -functions can be implemented in third party extensions, may users don't +functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is @@ -361,6 +356,34 @@ software using the Python 2 series (some developers would argue that we have already reached that point). +Python 2.7 represents the only long term maintenance release the core +development team has provided, and it is natural that there will be things +that worked over a historically shorter maintenance lifespan that don't work +over this longer support period. In the specific case of the problem +described in this PEP, the simplest available solution is to acknowledge +that long term maintenance of network security related modules *requires* +the ability to add new features, even while retaining backwards compatibility +for existing interfaces. + +It is worth comparing the approach described in this PEP with Red Hat's +handling of its long term support commitments: it isn't the RHEL 6.0 release +itself that receives 10 years worth of support, but the overall RHEL 6 +*series*. The individual RHEL 6.x point releases within the series then +receive a wide variety of new features, including security enhancements, +all while meeting strict backwards compatibility guarantees for existing +software. Subscribers *are* able to continue using a given point release +after the next point release has become available, but a corresponding +add-on subscription for `Extended Update Support +`__ +is needed to cover the additional backporting work involved. + +To date, downstream redistributors have respected our upstream policy of +"no new features in Python maintenance releases". This PEP explicitly +accepts that a more nuanced policy is appropriate in the case of network +security related features, and the specific one it describes is deliberately +designed such that it at least has some chance of being applied to Red Hat +Enterprise Linux and its downstream derivatives. + Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- @@ -429,7 +452,7 @@ across redistributors. The problem is real, so *something* needs to change, and this PEP describes -my currently preferred approach to addressing the situation. +my preferred approach to addressing the situation. Alternative: create and release Python 2.8 @@ -440,10 +463,11 @@ enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated -products and deployments that make use of Python 2. Upgrading to a new -Python feature release would mean both more work for the core development -team, as well as a more disruptive update that most potential end users -would likely just skip entirely. +products and deployments that make use of Python 2. + +Upgrading to a new Python feature release would mean both more work for the +core development team, as well as a more disruptive update that most +potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` @@ -455,6 +479,11 @@ the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). +Furthermore, while I can't make any commitments to actually addressing +this issue on Red Hat platforms, I *can* categorically rule out the idea +of a Python 2.8 being of any use to me in even attempting to get it +addressed. + Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- @@ -504,17 +533,78 @@ each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the -security enhancements (by switching to redistributing directly from the -``2.7-legacy-ssl`` branch rather than the main ``2.7`` branch), which most -of them are unlikely to do. +security enhancements by deliberately downgrading the provided network +security infrastructure, which most of them are unlikely to do. + + +Alternative: provide a "legacy SSL infrastructure" branch +--------------------------------------------------------- + +Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` +branch that preserved the exact feature set of the Python 2.7.6 network +security infrastructure. + +It is the opinion of the PEP author that anyone that actually wants this is +almost certainly making a mistake, and if they insist they really do want +it in their specific situation, they're welcome to either make it themselves +or arrange for a downstream redistributor to make it for them. + +If they are made publicly available, any such rebuilds should be referred to +as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official +Python 2.7 releases that include more up to date network security +infrastructure. + +After the first Python 2.7 maintenance release that has the security +infrastructure updated to match Python 3.4, it would also be appropriate to +refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". + + +Alternative: selectively backport particular APIs +------------------------------------------------- + +An instinctively minimalist reaction to this proposal is to only backport +particular APIs in the affected modules that are judged to be "security +critical". However, this ends up providing a worse end user experience, +as well as a worse developer experience. + +For end users, the selective backporting approach means learning not only +the legacy Python 2.7 API and the current Python 3 APIs, but also the +hybrid API created by the selective backporting process. It is much +simpler to just learn the APIs of the original Python 2.7 feature release +and the relevant Python 3 features releases, without trying to define a new +hybrid API that only exists in later Python 2.7 maintenance branches. + +For developers, the main benefit of the "just align the available +functionality with Python 3" approach is that it reduces the amount of +design work needing when updating Python 2.7 with new security features. +The "feature or fix?" and "security related or not?" debates are both +handled in advance by this policy, leaving only the matter of ensuring +that backwards compatibility is preserved as needed by adjusting the +default behaviour in the Python 2.7 backport when appropriate. Open Questions ============== +* MvL has indicated he is not prepared to tackle the task of trying to + integrate a newer OpenSSL into the also aging Python 2.7 build + infrastructure on Windows (unfortunately, we've looked into upgrading + that build infrastructure, and the backwards compatibility issues + appear to be effectively insurmountable). We would require a commitment + from another trusted contributor to handle at least this task, and + potentially also taking over the task of creating the official + Python 2.7 Windows installers for the remaining Python 2.7 maintenance + releases. + +* We would need commitments to create and review full backports of the + components covered by this policy from Python 3.4 to Python 2.7, as well + as support for handling any more specific security issues affecting these + modules. + * I believe I've addressed all the technical and scope questions I had, or - others raised. That just leaves the question of "If we agree to this plan, - who is actually going to handle all the extra work involved?" :) + others raised. That just leaves the question of "Do we agree this plan + actually makes sense, given the constraints on downstream redistributors + that would also like to see this problem solved?" :) Disclosure of Interest @@ -541,7 +631,7 @@ Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. -Thanks also to participants in the python-dev mailing list threads [1,2]_ +Thanks also to participants in the python-dev mailing list threads [1,2,5]_ References @@ -551,6 +641,7 @@ .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html .. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html .. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html +.. [5] https://mail.python.org/pipermail/python-dev/2014-March/133442.html Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Mar 25 18:19:59 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 18:19:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDU4?= =?utf-8?q?=3A_fix_typo_in_a_comment=2E_Patch_written_by_Vajrasky_Kok=2E?= Message-ID: <3ftfNH55Tsz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/aa2a05fe46ae changeset: 89977:aa2a05fe46ae branch: 3.4 parent: 89974:945d7dd3b455 user: Victor Stinner date: Tue Mar 25 18:19:17 2014 +0100 summary: Issue #21058: fix typo in a comment. Patch written by Vajrasky Kok. files: Lib/test/test_tempfile.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -760,7 +760,7 @@ self.assertRaises(ValueError, use_closed) def test_no_leak_fd(self): - # Issue #21058: don't leak file descriptor when io.pen() fails + # Issue #21058: don't leak file descriptor when io.open() fails closed = [] def close(fd): closed.append(fd) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 18:20:00 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 25 Mar 2014 18:20:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321058=3A_fix_typo_in_a_commen?= =?utf-8?q?t=2E_Patch_written_by_Vajrasky_Kok=2E?= Message-ID: <3ftfNJ6rR8z7LkK@mail.python.org> http://hg.python.org/cpython/rev/4e3c76cb0e8a changeset: 89978:4e3c76cb0e8a parent: 89975:6a0def54c63d parent: 89977:aa2a05fe46ae user: Victor Stinner date: Tue Mar 25 18:19:49 2014 +0100 summary: (Merge 3.4) Issue #21058: fix typo in a comment. Patch written by Vajrasky Kok. files: Lib/test/test_tempfile.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -760,7 +760,7 @@ self.assertRaises(ValueError, use_closed) def test_no_leak_fd(self): - # Issue #21058: don't leak file descriptor when io.pen() fails + # Issue #21058: don't leak file descriptor when io.open() fails closed = [] def close(fd): closed.append(fd) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 20:35:37 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 25 Mar 2014 20:35:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogYmFja3BvcnQ6ICMy?= =?utf-8?q?0145=3A_assertRaisesRegexp_now_raises_a_TypeError_on_bad_regex?= =?utf-8?q?=2E?= Message-ID: <3ftjNn1z7Lz7LjN@mail.python.org> http://hg.python.org/cpython/rev/3f8b801e7e76 changeset: 89979:3f8b801e7e76 branch: 2.7 parent: 89976:d09032a9adee user: R David Murray date: Tue Mar 25 15:29:42 2014 -0400 summary: backport: #20145: assertRaisesRegexp now raises a TypeError on bad regex. Previously a non-string, non-regex second argument and no callable argument could cause the test to appear to always pass. files: Lib/unittest/case.py | 4 ++-- Lib/unittest/test/test_case.py | 6 ++++++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -122,8 +122,6 @@ return True expected_regexp = self.expected_regexp - if isinstance(expected_regexp, basestring): - expected_regexp = re.compile(expected_regexp) if not expected_regexp.search(str(exc_value)): raise self.failureException('"%s" does not match "%s"' % (expected_regexp.pattern, str(exc_value))) @@ -986,6 +984,8 @@ args: Extra args. kwargs: Extra kwargs. """ + if expected_regexp is not None: + expected_regexp = re.compile(expected_regexp) context = _AssertRaisesContext(expected_exception, self, expected_regexp) if callable_obj is None: return context diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -979,6 +979,12 @@ self.assertRaisesRegexp, Exception, u'x', lambda: None) + def testAssertRaisesRegexpInvalidRegexp(self): + # Issue 20145. + class MyExc(Exception): + pass + self.assertRaises(TypeError, self.assertRaisesRegexp, MyExc, lambda: True) + def testAssertRaisesRegexpMismatch(self): def Stub(): raise Exception('Unexpected') diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -546,6 +546,7 @@ Albert Hofkamp Tomas Hoger Jonathan Hogg +Kamilla Holanda Steve Holden Akintayo Holder Thomas Holenstein diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -40,6 +40,9 @@ Library ------- +- Issue #20145: `assertRaisesRegex` now raises a TypeError if the second + argument is not a string or compiled regex. + - Issue #21058: Fix a leak of file descriptor in tempfile.NamedTemporaryFile(), close the file descriptor if os.fdopen() fails -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 20:35:42 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 25 Mar 2014 20:35:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYmFja3BvcnQ6ICMy?= =?utf-8?q?0145=3A_assert=5BRaises=7CWarns=5DRegex_now_raise_TypeError_on_?= =?utf-8?q?bad_regex=2E?= Message-ID: <3ftjNt6VJgz7LkS@mail.python.org> http://hg.python.org/cpython/rev/32407a677215 changeset: 89980:32407a677215 branch: 3.4 parent: 89977:aa2a05fe46ae user: R David Murray date: Tue Mar 25 15:31:50 2014 -0400 summary: backport: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. Previously a non-string, non-regex second argument and missing callable argument could cause the test to appear to always pass. Initial patch by Kamilla Holanda. files: Lib/unittest/case.py | 2 +- Lib/unittest/test/test_case.py | 12 ++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 17 insertions(+), 1 deletions(-) diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -143,7 +143,7 @@ self.obj_name = str(callable_obj) else: self.obj_name = None - if isinstance(expected_regex, (bytes, str)): + if expected_regex is not None: expected_regex = re.compile(expected_regex) self.expected_regex = expected_regex self.msg = None diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -1126,6 +1126,18 @@ self.assertRaisesRegex, Exception, 'x', lambda: None) + def testAssertRaisesRegexInvalidRegex(self): + # Issue 20145. + class MyExc(Exception): + pass + self.assertRaises(TypeError, self.assertRaisesRegex, MyExc, lambda: True) + + def testAssertWarnsRegexInvalidRegex(self): + # Issue 20145. + class MyWarn(Warning): + pass + self.assertRaises(TypeError, self.assertWarnsRegex, MyWarn, lambda: True) + def testAssertRaisesRegexMismatch(self): def Stub(): raise Exception('Unexpected') diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -546,6 +546,7 @@ Albert Hofkamp Tomas Hoger Jonathan Hogg +Kamilla Holanda Steve Holden Akintayo Holder Thomas Holenstein diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,9 @@ Library ------- +- Issue #20145: `assertRaisesRegex` and `assertWarnsRegex` now raise a + TypeError if the second argument is not a string or compiled regex. + - Issue #21058: Fix a leak of file descriptor in :func:`tempfile.NamedTemporaryFile`, close the file descriptor if :func:`io.open` fails -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 20:35:44 2014 From: python-checkins at python.org (r.david.murray) Date: Tue, 25 Mar 2014 20:35:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_=2320145_backport=3A_delete_whatsnew_entry=2E?= Message-ID: <3ftjNw0p62z7Lkq@mail.python.org> http://hg.python.org/cpython/rev/8f72f8359987 changeset: 89981:8f72f8359987 parent: 89978:4e3c76cb0e8a parent: 89980:32407a677215 user: R David Murray date: Tue Mar 25 15:34:17 2014 -0400 summary: Merge #20145 backport: delete whatsnew entry. files: Doc/whatsnew/3.5.rst | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -192,9 +192,3 @@ if it represented midnight in UTC. This behavior was considered obscure and error-prone and has been removed in Python 3.5. See :issue:`13936` for full details. - -* :meth:`unittest.TestCase.assertRaisesRegex` and - :meth:`~unittest.TestCase.assertWarnsRegex` now raise a :exc:`TypeError` if - the second argument is not a string or a compiled :mod:`regex`. You may have - tests with an invalid second argument that have until 3.5 been falsely - passing which will now raise TypeErrors (:issue:`20145`). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Mar 25 23:33:59 2014 From: python-checkins at python.org (ethan.furman) Date: Tue, 25 Mar 2014 23:33:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_incorporated_=25a_comments=3B?= =?utf-8?q?_general_clean-up?= Message-ID: <3ftnLb25fXz7LjQ@mail.python.org> http://hg.python.org/peps/rev/829c7e796eb8 changeset: 5429:829c7e796eb8 user: Ethan Furman date: Tue Mar 25 15:33:49 2014 -0700 summary: incorporated %a comments; general clean-up files: pep-0461.txt | 91 +++++++++++++++++++++------------------ 1 files changed, 48 insertions(+), 43 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -8,8 +8,8 @@ Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 -Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22 -Resolution: +Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25 +Resolution: Abstract @@ -40,13 +40,8 @@ restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. - -Overriding Principles -===================== - -In order to avoid the problems of auto-conversion and Unicode exceptions -that could plague Python 2 code, ``str`` objects will not be supported as -interpolation values [4]_ [5]_. +Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` +formats, and ``FTP`` and ``HTTP`` communications, among many others. Proposed semantics for ``bytes`` and ``bytearray`` formatting @@ -57,23 +52,31 @@ All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, ``%g``, etc.) will be supported, and will work as they do for str, including -the padding, justification and other related modifiers. +the padding, justification and other related modifiers. The only difference +will be that the results from these codes will be ASCII-encoded text, not +unicode. In other words, for any numeric formatting code `%x`:: -Example:: + b"%x" % val + +is equivalent to + + ("%x" % val).encode("ascii") + +Examples:: >>> b'%4x' % 10 b' a' - >>> '%#4x' % 10 + >>> b'%#4x' % 10 ' 0xa' - >>> '%04X' % 10 + >>> b'%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. -Example:: +Examples:: >>> b'%c' % 48 b'0' @@ -81,7 +84,9 @@ >>> b'%c' % b'a' b'a' -``%s`` is restricted in what it will accept:: +``%s`` is included for two reasons: 1) `b` is already a format code for +``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x +code uses ``%s``; however, it is restricted in what it will accept:: - input type supports ``Py_buffer`` [6]_? use it to collect the necessary bytes @@ -89,40 +94,46 @@ - input type is something else? use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError`` +In particular, ``%s`` will not accept numbers (use a numeric format code for +that), nor ``str`` (encode it to ``bytes``). + Examples:: >>> b'%s' % b'abc' b'abc' + >>> b'%s' % 'some string'.encode('utf8') + b'some string' + >>> b'%s' % 3.14 Traceback (most recent call last): ... - TypeError: 3.14 has no __bytes__ method, use a numeric code instead + TypeError: b'%s' does not accept numbers, use a numeric code instead >>> b'%s' % 'hello world!' Traceback (most recent call last): ... - TypeError: 'hello world' has no __bytes__ method, perhaps you need to encode it? + TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` + + +``%a`` will call ``ascii()`` on the interpolated value. This is intended +as a debugging aid, rather than something that should be used in production. +Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` +representation. Use cases include developing a new protocol and writing +landmarks into the stream; debugging data going into an existing protocol +to see if the problem is the protocol itself or bad data; a fall-back for a +serialization format; or even a rudimentary serialization format when +defining ``__bytes__`` would not be appropriate [8]. .. note:: - Because the ``str`` type does not have a ``__bytes__`` method, attempts to - directly use ``'a string'`` as a bytes interpolation value will raise an - exception. To use strings they must be encoded or otherwise transformed - into a ``bytes`` sequence:: - - 'a string'.encode('latin-1') - -``%a`` will call ``ascii()`` on the interpolated value's ``repr()``. -This is intended as a debugging aid, rather than something that should be used -in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` -representation. + If a ``str`` is passed into ``%a``, it will be surrounded by quotes. Unsupported codes ----------------- -``%r`` (which calls ``__repr__`` and returns a '`str`') is not supported. +``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. Proposed variations @@ -131,6 +142,9 @@ It was suggested to let ``%s`` accept numbers, but since numbers have their own format codes this idea was discarded. +It has been suggested to use ``%b`` for bytes as well as ``%s``. This was +rejected as not adding any value either in clarity or simplicity. + It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%s``. @@ -171,19 +185,11 @@ ``bytes`` and ``bytearray`` already have several methods which assume an ASCII compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name just a few. %-interpolation, with its very restricted mini-language, will not -be any more of a nuisance than the already existing methdods. +be any more of a nuisance than the already existing methods. - -Open Questions -============== - -It has been suggested to use ``%b`` for bytes as well as ``%s``. - - - Pro: clearly says 'this is bytes'; should be used for new code. - - - Con: does not exist in Python 2.x, so we would have two ways of doing the - same thing, ``%s`` and ``%b``, with no difference between them. - +Some have objected to allowing the full range of numeric formatting codes with +the claim that decimal alone would be sufficient. However, at least two +formats (dbf and pdf) make use of non-decimal numbers. Footnotes @@ -197,8 +203,7 @@ .. [6] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ -.. [8] mainly implicit encode/decode, with intermittent errors when the data - was not ASCII compatible +.. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html Copyright -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Wed Mar 26 09:49:04 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 26 Mar 2014 09:49:04 +0100 Subject: [Python-checkins] Daily reference leaks (8f72f8359987): sum=0 Message-ID: results for 8f72f8359987 on branch "default" -------------------------------------------- test_site leaked [2, -2, 0] references, sum=0 test_site leaked [2, -2, 0] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogFL9O6d', '-x'] From python-checkins at python.org Wed Mar 26 10:01:10 2014 From: python-checkins at python.org (raymond.hettinger) Date: Wed, 26 Mar 2014 10:01:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Broaden_the_early-out_test?= =?utf-8?q?_for_nsmallest_and_nlargest?= Message-ID: <3fv3GG0Rzdz7LjS@mail.python.org> http://hg.python.org/cpython/rev/0f6c518183ef changeset: 89982:0f6c518183ef user: Raymond Hettinger date: Wed Mar 26 02:00:54 2014 -0700 summary: Broaden the early-out test for nsmallest and nlargest files: Lib/heapq.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/heapq.py b/Lib/heapq.py --- a/Lib/heapq.py +++ b/Lib/heapq.py @@ -197,7 +197,7 @@ Equivalent to: sorted(iterable, reverse=True)[:n] """ - if n < 0: + if n <= 0: return [] it = iter(iterable) result = list(islice(it, n)) @@ -215,7 +215,7 @@ Equivalent to: sorted(iterable)[:n] """ - if n < 0: + if n <= 0: return [] it = iter(iterable) result = list(islice(it, n)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Mar 26 12:46:52 2014 From: python-checkins at python.org (nick.coghlan) Date: Wed, 26 Mar 2014 12:46:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_switch_to_more_sel?= =?utf-8?q?ective_backports?= Message-ID: <3fv6xS5lpHz7LkD@mail.python.org> http://hg.python.org/peps/rev/8527f6e2beb0 changeset: 5430:8527f6e2beb0 user: Nick Coghlan date: Wed Mar 26 21:46:42 2014 +1000 summary: PEP 466: switch to more selective backports files: pep-0466.txt | 268 ++++++++++++++++++-------------------- 1 files changed, 125 insertions(+), 143 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -7,7 +7,7 @@ Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 -Post-History: 23-Mar-2014 +Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 Abstract @@ -22,8 +22,8 @@ feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security -protocols for it to be causing real problems in commercial use cases -where upgrading to Python 3 in the near term may not be practical. +protocols for it to be causing real problems in use cases where upgrading to +Python 3 in the near term may not be feasible. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows @@ -31,91 +31,74 @@ overall security of the internet to be updated in line with the corresponding Python 3 feature releases. -Specifically, the exception will apply to: - -* the ``ssl`` module -* the ``hashlib`` module -* the ``hmac`` module -* the components of the ``random`` and ``os`` modules that the above - modules rely on for cryptographic randomness -* the version of OpenSSL bundled with the binary installers for Windows - and Mac OS X - -Changes to these modules will still need to undergo normal backwards -compatibility assessments to ensure their default behaviour remains -consistent with earlier Python 2.7 releases, but otherwise they will be -kept entirely aligned with the latest feature release of their Python 3 -counterparts. This is designed to make it easier to implement secure -networked software in Python, even for software that currently needs to -remain compatible with the Python 2 series (which includes a lot of -network infrastructure software). +Specifically, the exception allows a critical set of network security +related features to be backported from Python 3.4 to the upcoming Python +2.7.7 maintenance release. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing -extended support periods for the Python standard library either adopt a -similar approach to ensuring that the secure networking infrastructure -keeps pace with the evolution of the internet, or else explicitly -disclaim support for the use of older versions in roles that involve -connecting directly to the public internet. +extended support periods for the Python standard library either backport +these features to their supported versions, or else explicitly disclaim +support for the use of older versions in roles that involve connecting +directly to the public internet. Exemption Policy ================ -Under this policy, the following network security related modules are -granted a blanket exemption to the normal restriction against adding new -features in Python 2.7 maintenance releases, for the purpose of keeping -their APIs aligned with their counterparts in the latest feature release -of Python 3: +Under this policy, the following features SHOULD be backported from Python +3.4 to the upcoming Python 2.7.7 maintenance release: -* the ``ssl`` module -* the ``hashlib`` module -* the ``hmac`` module +* in the ``os`` module: -Under this exemption, these modules are updated to provide identical -functionality to their Python 3 counterparts after every new Python 3 -feature release. The default behaviour of the backported modules will be -adjusted as appropriate to meet the backwards compatibility requirements -of a Python 2.7 maintenance release. + * persistent file descriptor for ``os.urandom()``. + +* in the ``hmac`` module: + + * constant time comparison function (``hmac.compare_digest()``). + +* in the ``hashlib`` module: + + * password hashing function (``hashlib.pbkdf2_hmac()``). + * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` + and ``hashlib.algorithms_guaranteed``). + +* in the ``ssl`` module: + + * this module is almost entirely synchronised with its Python 3 + counterpart, bringing TLSv2, SSLContext manipulation, Server Name + Identification, access to platform certificate stores, standard + library support for peer hostname validation and more to the Python 2 + series. + * the only ``ssl`` module features *not* backported under this policy are + the ``ssl.RAND_*`` functions that provide access to OpenSSL's random + number generation capabilities - use ``os.urandom()`` instead. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. -Note that the ``sha`` and ``md5`` modules are not covered by this policy, -but have been deprecated in favour of ``hashlib`` since Python 2.5 and have -been removed entirely in the Python 3 series. - -In addition to the above blanket exemption, a conditional exemption is -granted for these modules that may include some network security related -features: - -* the ``os`` module (specifically ``os.urandom``) -* the ``random`` module - -This more limited exemption for these modules requires that the *specific* -enhancement being proposed for backporting needs to be justified as being -network security related. This is generally restricted to cases where the -feature in question is needed by an update to one of the modules covered -by the blanket exemption. - Backwards Compatibility Considerations ====================================== -This PEP does *not* grant Python 2.7 any general exemptions to the usual -backwards compatibility policy for maintenance releases. Instead, by -explicitly encouraging the use of feature based checks and explicitly -opting in to less secure configurations, it is designed to make it easier +As in the Python 3 series, the backported ``ssl.create_default_context()`` +API is granted a backwards compatibility exemption that permits the +protocol, options, cipher and other settings of the created SSL context to +be made + +This PEP does *not* grant any exemptions to the usual backwards +compatibility policy for maintenance releases. Instead, by explicitly +encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. -In *all* cases where this policy is applied to backport enhancements to -Python 2.7 maintenance releases, it MUST be possible to write cross-version +In all cases where this policy allows new features to be backported to +the Python 2.7 release series, it is possible to write cross-version compatible code that operates by "feature detection" (for example, checking -for particular attributes in the module), without needing to explicitly check +for particular attributes in a module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning @@ -124,16 +107,17 @@ security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. -Affected APIs SHOULD be designed to allow library and application code to -perform the following actions after detecting the presence of a relevant +The backported APIs allow library and application code to perform the +following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced - security features in older maintenance releases of Python) + security features in older maintenance releases of Python with less + secure default behaviour) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit - Python version checks to determine the Python feature release, but MUST + Python version checks to determine the Python feature release, but DOES NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking @@ -187,12 +171,12 @@ Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. -This policy represents an explicit invitation to contribute some of those -changes back to the upstream Python community in cases where they are likely -to have a broad impact that helps improve the security of the internet as a -whole, with the assurance that the existing core development team not only -won't object to such contributions, but will actively encourage their -incorporation into the next Python 2.7 maintenance release. +This policy represents an explicit invitation to implement those changes +in the core development tree in cases where they are likely to have a broad +impact that helps improve the security of the internet as a whole, with the +assurance that the existing core development team not only won't object to +such contributions, but will actively encourage their incorporation into +the next Python 2.7 maintenance release. Documentation @@ -201,7 +185,7 @@ All modules covered by this policy MUST include a "Security Considerations" section in their documentation in order to take advantage of this policy. -In addition to any other module specific contents, this section MUST +In addition to any other module specific contents, this section SHOULD enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. @@ -232,8 +216,8 @@ libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even -after the latest Python 2.7 maintenance release has been synchronised with -a new Python 3 feature release for the modules covered by this policy. +after the features covered in this policy have been backported to the +Python 2.7 series. Handling lower security environments with low risk tolerance @@ -258,16 +242,16 @@ Evolution of this Policy ======================== -The key requirement for a module to be considered for inclusion in this -policy (whether under a blanket or conditional exemption) is that it must -have security implications *beyond* the specific application that is written -in Python and the system that application is running on. Thus the focus on -network security protocols and related cryptographic infrastructure - Python -is a popular choice for the development of web services and clients, and -thus the capabilities of widely used Python versions have implications for -the security design of other services that may themselves be using newer -versions of Python or other development languages, but need to interoperate -with clients or servers written using older versions of Python. +The key requirement for a feature to be considered for inclusion in this +policy is that it must have security implications *beyond* the specific +application that is written in Python and the system that application is +running on. Thus the focus on network security protocols, password storage +and related cryptographic infrastructure - Python is a popular choice for +the development of web services and clients, and thus the capabilities of +widely used Python versions have implications for the security design of +other services that may themselves be using newer versions of Python or +other development languages, but need to interoperate with clients or +servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of @@ -280,15 +264,6 @@ Motivation and Rationale ======================== -This PEP can be seen as a more targeted version of the "faster standard -library release cycle" proposals discussed in PEP 407 and PEP 413, -focusing specifically on those areas which have implications beyond the -Python community. - - -Background ----------- - The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular @@ -315,8 +290,8 @@ shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy -aims to acknowledge the fact that Python's popularity and adoption has now -reached a level where some of our design and policy decisions have +aims to acknowledge the fact that Python's popularity and adoption is at a +sufficiently high level that some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server @@ -371,11 +346,10 @@ *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing -software. Subscribers *are* able to continue using a given point release -after the next point release has become available, but a corresponding -add-on subscription for `Extended Update Support -`__ -is needed to cover the additional backporting work involved. +software. The policy described in this PEP brings our approach to long term +maintenance more into line with this precedent - we retain our strict +backwards compatibility requirements, but slightly relax the restrictions +against adding new features. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly @@ -385,8 +359,8 @@ Enterprise Linux and its downstream derivatives. -Alternative: advise developers of networked software to migrate to Python 3 ---------------------------------------------------------------------------- +Rejected alternative: just advise developers to migrate to Python 3 +------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications @@ -428,7 +402,7 @@ The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time -to migrate. +to migrate fully to a Python 2/3 compatible code base. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of @@ -455,8 +429,8 @@ my preferred approach to addressing the situation. -Alternative: create and release Python 2.8 ------------------------------------------- +Rejected alternative: create and release Python 2.8 +--------------------------------------------------- With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner @@ -485,8 +459,8 @@ addressed. -Alternative: distribute the security enhancements via PyPI ----------------------------------------------------------- +Rejected alternative: distribute the security enhancements via PyPI +------------------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. @@ -537,8 +511,8 @@ security infrastructure, which most of them are unlikely to do. -Alternative: provide a "legacy SSL infrastructure" branch ---------------------------------------------------------- +Rejected variant: provide a "legacy SSL infrastructure" branch +-------------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network @@ -559,28 +533,14 @@ refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". -Alternative: selectively backport particular APIs -------------------------------------------------- +Rejected variant: synchronise particular modules entirely with Python 3 +----------------------------------------------------------------------- -An instinctively minimalist reaction to this proposal is to only backport -particular APIs in the affected modules that are judged to be "security -critical". However, this ends up providing a worse end user experience, -as well as a worse developer experience. +Earlier versions of this PEP suggested synchronising the ``hmac``, +``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. -For end users, the selective backporting approach means learning not only -the legacy Python 2.7 API and the current Python 3 APIs, but also the -hybrid API created by the selective backporting process. It is much -simpler to just learn the APIs of the original Python 2.7 feature release -and the relevant Python 3 features releases, without trying to define a new -hybrid API that only exists in later Python 2.7 maintenance branches. - -For developers, the main benefit of the "just align the available -functionality with Python 3" approach is that it reduces the amount of -design work needing when updating Python 2.7 with new security features. -The "feature or fix?" and "security related or not?" debates are both -handled in advance by this policy, leaving only the matter of ensuring -that backwards compatibility is preserved as needed by adjusting the -default behaviour in the Python 2.7 backport when appropriate. +This approach proved too vague to build a compelling case for the exception, +and has thus been replaced by the current more explicit proposal. Open Questions @@ -601,10 +561,8 @@ as support for handling any more specific security issues affecting these modules. -* I believe I've addressed all the technical and scope questions I had, or - others raised. That just leaves the question of "Do we agree this plan - actually makes sense, given the constraints on downstream redistributors - that would also like to see this problem solved?" :) +* Did I miss anything important in the switch to a more restrictive + proposal? Disclosure of Interest @@ -620,7 +578,7 @@ Acknowledgements ================ -Thanks to Christian Heimes for his recent efforts on greatly improving +Thanks to Christian Heimes and other for their efforts in greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that @@ -628,20 +586,44 @@ (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. -Christian and Donald Stufft also provided valuable feedback on a preliminary +Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set +of essential security features that allowed the proposal to be made more +fine-grained than backporting entire modules from Python 3.4 [7,8]_. + +Christian and Donald also provided valuable feedback on a preliminary draft of this proposal. -Thanks also to participants in the python-dev mailing list threads [1,2,5]_ +Thanks also to participants in the python-dev mailing list threads +[1,2,5,6]_ References ========== -.. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html -.. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html -.. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html -.. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html -.. [5] https://mail.python.org/pipermail/python-dev/2014-March/133442.html +.. [1] PEP 466 discussion (round 1) + (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) + +.. [2] PEP 466 discussion (round 2) + (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) + +.. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows + (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) + +.. [4] Ned Deily's OpenSSL feedback for Mac OS X + (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) + +.. [5] PEP 466 discussion (round 3) + (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) + +.. [6] PEP 466 discussion (round 4) + (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) + +.. [7] Donald Stufft's recommended set of backported features + (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) + +.. [8] Alex Gaynor's recommended set of backported features + (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) + Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Mar 26 13:03:16 2014 From: python-checkins at python.org (nick.coghlan) Date: Wed, 26 Mar 2014 13:03:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_Fix_copy_=26_paste?= =?utf-8?q?_error?= Message-ID: <3fv7JN6SSbz7LkD@mail.python.org> http://hg.python.org/peps/rev/1f1ea7a7798b changeset: 5431:1f1ea7a7798b user: Nick Coghlan date: Wed Mar 26 22:03:06 2014 +1000 summary: PEP 466: Fix copy & paste error files: pep-0466.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -62,7 +62,7 @@ * password hashing function (``hashlib.pbkdf2_hmac()``). * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` - and ``hashlib.algorithms_guaranteed``). + and ``hashlib.algorithms_available``). * in the ``ssl`` module: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Mar 26 13:05:48 2014 From: python-checkins at python.org (nick.coghlan) Date: Wed, 26 Mar 2014 13:05:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_finish_incomplete_?= =?utf-8?q?edit?= Message-ID: <3fv7MJ1SH6z7LlB@mail.python.org> http://hg.python.org/peps/rev/3118239a142a changeset: 5432:3118239a142a user: Nick Coghlan date: Wed Mar 26 22:05:39 2014 +1000 summary: PEP 466: finish incomplete edit files: pep-0466.txt | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -86,9 +86,12 @@ As in the Python 3 series, the backported ``ssl.create_default_context()`` API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to -be made +be updated in maintenance releases to use higher default security settings. +This allows them to appropriately balance compatibility and security at the +time of the maintenance release, rather than at the time of the original +feature release. -This PEP does *not* grant any exemptions to the usual backwards +This PEP does *not* grant any other exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Mar 26 15:46:40 2014 From: python-checkins at python.org (ethan.furman) Date: Wed, 26 Mar 2014 15:46:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_added_list_of_currently_suppo?= =?utf-8?q?rted_codes=3B_modified_description_for_=25a=3B_added?= Message-ID: <3fvBww3sGxz7Ljv@mail.python.org> http://hg.python.org/peps/rev/bede07793448 changeset: 5433:bede07793448 user: Ethan Furman date: Wed Mar 26 07:46:34 2014 -0700 summary: added list of currently supported codes; modified description for %a; added reference to competing PEP 460 files: pep-0461.txt | 46 ++++++++++++++++++++++++++------------- 1 files changed, 31 insertions(+), 15 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -50,11 +50,17 @@ %-interpolation --------------- -All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, -``%g``, etc.) will be supported, and will work as they do for str, including -the padding, justification and other related modifiers. The only difference -will be that the results from these codes will be ASCII-encoded text, not -unicode. In other words, for any numeric formatting code `%x`:: +All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, +``e``, ``E'', ``f``, ``F``, ``g``, ``G``, and any that are subsequently added +to Python 3) will be supported, and will work as they do for str, including +the padding, justification and other related modifiers (currently ``#``, ``0``, +``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only +non-numeric codes allowed are ``c``, ``s``, and ``a``. + +For the numeric codes, the only difference between ``str`` and ``bytes`` (or +``bytearray``) interpolation is that the results from these codes will be +ASCII-encoded text, not unicode. In other words, for any numeric formatting +code `%x`:: b"%x" % val @@ -116,18 +122,24 @@ TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` -``%a`` will call ``ascii()`` on the interpolated value. This is intended -as a debugging aid, rather than something that should be used in production. -Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` -representation. Use cases include developing a new protocol and writing -landmarks into the stream; debugging data going into an existing protocol -to see if the problem is the protocol itself or bad data; a fall-back for a -serialization format; or even a rudimentary serialization format when -defining ``__bytes__`` would not be appropriate [8]. +``%a`` will give the equivalent of +``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated +value. Use cases include developing a new protocol and writing landmarks +into the stream; debugging data going into an existing protocol to see if +the problem is the protocol itself or bad data; a fall-back for a serialization +format; or any situation where defining ``__bytes__`` would not be appropriate +but a readable/informative representation is needed [8]. -.. note:: +Examples:: - If a ``str`` is passed into ``%a``, it will be surrounded by quotes. + >>> b'%a' % 3.14 + b'3.14' + + >>> b'%a' % b'abc' + b'abc' + + >>> b'%a' % 'def' + b"'def'" Unsupported codes @@ -166,6 +178,9 @@ ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. +A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9], also +exists. + Objections ========== @@ -204,6 +219,7 @@ examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ .. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html +.. [9] http://python.org/dev/peps/pep-0460/ Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Mar 26 23:47:19 2014 From: python-checkins at python.org (ethan.furman) Date: Wed, 26 Mar 2014 23:47:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_fix_=25a_bytes_example?= Message-ID: <3fvPbW6p91z7LjV@mail.python.org> http://hg.python.org/peps/rev/887045b096c8 changeset: 5434:887045b096c8 user: Ethan Furman date: Wed Mar 26 15:47:12 2014 -0700 summary: fix %a bytes example files: pep-0461.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -136,7 +136,7 @@ b'3.14' >>> b'%a' % b'abc' - b'abc' + b"b'abc'" >>> b'%a' % 'def' b"'def'" -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 07:34:28 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:34:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_Use_www=2Eexample=2Ecom_instead_of_www=2Epython=2Eorg_to_av?= =?utf-8?q?oid_test?= Message-ID: <3fvbyX1HwNz7LjY@mail.python.org> http://hg.python.org/cpython/rev/426a7046cdb0 changeset: 89983:426a7046cdb0 branch: 2.7 parent: 89979:3f8b801e7e76 user: Ned Deily date: Wed Mar 26 23:25:02 2014 -0700 summary: Issue #20939: Use www.example.com instead of www.python.org to avoid test failures when ssl is not present. files: Lib/test/test_robotparser.py | 7 +++++++ Lib/test/test_urllib2net.py | 12 ++++++------ Lib/test/test_urllibnet.py | 24 ++++++++++++------------ Misc/NEWS | 5 +++-- 4 files changed, 28 insertions(+), 20 deletions(-) diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -2,6 +2,12 @@ from test import test_support from urllib2 import urlopen, HTTPError +HAVE_HTTPS = True +try: + from urllib2 import HTTPSHandler +except ImportError: + HAVE_HTTPS = False + class RobotTestCase(unittest.TestCase): def __init__(self, index, parser, url, good, agent): unittest.TestCase.__init__(self) @@ -269,6 +275,7 @@ self.skipTest('%s is unavailable' % url) self.assertEqual(parser.can_fetch("*", robots_url), False) + @unittest.skipUnless(HAVE_HTTPS, 'need SSL support to download license') def testPythonOrg(self): test_support.requires('network') with test_support.transient_internet('www.python.org'): diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -78,7 +78,7 @@ # underlying socket # delve deep into response to fetch socket._socketobject - response = _urlopen_with_retry("http://www.python.org/") + response = _urlopen_with_retry("http://www.example.com/") abused_fileobject = response.fp self.assertIs(abused_fileobject.__class__, socket._fileobject) httpresponse = abused_fileobject._sock @@ -163,7 +163,7 @@ "http://docs.python.org/2/glossary.html#glossary") def test_fileno(self): - req = urllib2.Request("http://www.python.org") + req = urllib2.Request("http://www.example.com") opener = urllib2.build_opener() res = opener.open(req) try: @@ -251,14 +251,14 @@ class TimeoutTest(unittest.TestCase): def test_http_basic(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with test_support.transient_internet(url, timeout=None): u = _urlopen_with_retry(url) self.assertIsNone(u.fp._sock.fp._sock.gettimeout()) def test_http_default_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with test_support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -269,7 +269,7 @@ def test_http_no_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with test_support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -279,7 +279,7 @@ self.assertIsNone(u.fp._sock.fp._sock.gettimeout()) def test_http_timeout(self): - url = "http://www.python.org" + url = "http://www.example.com" with test_support.transient_internet(url): u = _urlopen_with_retry(url, timeout=120) self.assertEqual(u.fp._sock.fp._sock.gettimeout(), 120) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -34,7 +34,7 @@ socket.setdefaulttimeout(None) def testURLread(self): - f = _open_with_retry(urllib.urlopen, "http://www.python.org/") + f = _open_with_retry(urllib.urlopen, "http://www.example.com/") x = f.read() class urlopenNetworkTests(unittest.TestCase): @@ -46,7 +46,7 @@ for transparent redirection have been written. setUp is not used for always constructing a connection to - http://www.python.org/ since there a few tests that don't use that address + http://www.example.com/ since there a few tests that don't use that address and making a connection is expensive enough to warrant minimizing unneeded connections. @@ -57,7 +57,7 @@ def test_basic(self): # Simple test expected to pass. - open_url = self.urlopen("http://www.python.org/") + open_url = self.urlopen("http://www.example.com/") for attr in ("read", "readline", "readlines", "fileno", "close", "info", "geturl"): self.assertTrue(hasattr(open_url, attr), "object returned from " @@ -69,7 +69,7 @@ def test_readlines(self): # Test both readline and readlines. - open_url = self.urlopen("http://www.python.org/") + open_url = self.urlopen("http://www.example.com/") try: self.assertIsInstance(open_url.readline(), basestring, "readline did not return a string") @@ -80,7 +80,7 @@ def test_info(self): # Test 'info'. - open_url = self.urlopen("http://www.python.org/") + open_url = self.urlopen("http://www.example.com/") try: info_obj = open_url.info() finally: @@ -92,7 +92,7 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "https://www.python.org/" + URL = "http://www.example.com/" open_url = self.urlopen(URL) try: gotten_url = open_url.geturl() @@ -102,7 +102,7 @@ def test_getcode(self): # test getcode() with the fancy opener to get 404 error codes - URL = "http://www.python.org/XXXinvalidXXX" + URL = "http://www.example.com/XXXinvalidXXX" open_url = urllib.FancyURLopener().open(URL) try: code = open_url.getcode() @@ -114,7 +114,7 @@ @unittest.skipUnless(hasattr(os, 'fdopen'), 'os.fdopen not available') def test_fileno(self): # Make sure fd returned by fileno is valid. - open_url = self.urlopen("http://www.python.org/") + open_url = self.urlopen("http://www.example.com/") fd = open_url.fileno() FILE = os.fdopen(fd) try: @@ -152,7 +152,7 @@ def test_basic(self): # Test basic functionality. - file_location,info = self.urlretrieve("http://www.python.org/") + file_location,info = self.urlretrieve("http://www.example.com/") self.assertTrue(os.path.exists(file_location), "file location returned by" " urlretrieve is not a valid path") FILE = file(file_location) @@ -165,7 +165,7 @@ def test_specified_path(self): # Make sure that specifying the location of the file to write to works. - file_location,info = self.urlretrieve("http://www.python.org/", + file_location,info = self.urlretrieve("http://www.example.com/", test_support.TESTFN) self.assertEqual(file_location, test_support.TESTFN) self.assertTrue(os.path.exists(file_location)) @@ -178,13 +178,13 @@ def test_header(self): # Make sure header returned as 2nd value from urlretrieve is good. - file_location, header = self.urlretrieve("http://www.python.org/") + file_location, header = self.urlretrieve("http://www.example.com/") os.unlink(file_location) self.assertIsInstance(header, mimetools.Message, "header is not an instance of mimetools.Message") def test_data_header(self): - logo = "http://python.org/static/community_logos/python-logo-master-v3-TM.png" + logo = "http://www.example.com/" file_location, fileheaders = self.urlretrieve(logo) os.unlink(file_location) datevalue = fileheaders.getheader('Date') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -324,8 +324,9 @@ - Issue #20605: Make test_socket getaddrinfo OS X segfault test more robust. -- Issue #20939: Fix test_geturl failure in test_urllibnet due to - new redirect of http://www.python.org/ to https://www.python.org. +- Issue #20939: Avoid various network test failures due to new + redirect of http://www.python.org/ to https://www.python.org: + use http://www.example.com instead. Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 07:34:29 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:34:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_Backout_test=5Furllib2=2Etest=5Fissue16464_disables=3A?= Message-ID: <3fvbyY33Mbz7LjR@mail.python.org> http://hg.python.org/cpython/rev/31e42208eb99 changeset: 89984:31e42208eb99 branch: 3.4 parent: 89980:32407a677215 user: Ned Deily date: Wed Mar 26 23:26:03 2014 -0700 summary: Issue #20939: Backout test_urllib2.test_issue16464 disables: 68335b8afb1f 3.4 ad0c75b7bd7d default files: Lib/test/test_urllib2.py | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -6,10 +6,6 @@ import socket import array import sys -try: - import ssl -except ImportError: - ssl = None import urllib.request # The proxy bypass method imported below has logic specific to the OSX @@ -1442,8 +1438,6 @@ @unittest.skipUnless(support.is_resource_enabled('network'), 'test requires network access') - @unittest.skipIf(ssl is None, - 'test requires the ssl module') def test_issue16464(self): opener = urllib.request.build_opener() request = urllib.request.Request("http://www.python.org/~jeremy/") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 07:34:30 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:34:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_Use_www=2Eexample=2Ecom_instead_of_www=2Epython=2Eorg_to_av?= =?utf-8?q?oid_test?= Message-ID: <3fvbyZ5ZQrz7LjR@mail.python.org> http://hg.python.org/cpython/rev/6134684ba222 changeset: 89985:6134684ba222 branch: 3.4 user: Ned Deily date: Wed Mar 26 23:31:39 2014 -0700 summary: Issue #20939: Use www.example.com instead of www.python.org to avoid test failures when ssl is not present. files: Lib/test/test_site.py | 4 ++- Lib/test/test_urllib2.py | 2 +- Lib/test/test_urllib2net.py | 10 ++++---- Lib/test/test_urllibnet.py | 26 ++++++++++++------------ Misc/NEWS | 4 +++ 5 files changed, 26 insertions(+), 20 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -414,8 +414,10 @@ @test.support.requires_resource('network') @unittest.skipUnless(sys.version_info[3] == 'final', 'only for released versions') + @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"), + 'need SSL support to download license') def test_license_exists_at_url(self): - # This test is a bit fragile since it depends on the format of the + # This test `is a bit fragile since it depends on the format of the # string displayed by license in the absence of a LICENSE file. url = license._Printer__data.split()[1] req = urllib.request.Request(url, method='HEAD') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1440,7 +1440,7 @@ 'test requires network access') def test_issue16464(self): opener = urllib.request.build_opener() - request = urllib.request.Request("http://www.python.org/~jeremy/") + request = urllib.request.Request("http://www.example.com/") self.assertEqual(None, request.data) opener.open(request, "1".encode("us-ascii")) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -83,7 +83,7 @@ def test_close(self): # calling .close() on urllib2's response objects should close the # underlying socket - url = "http://www.python.org/" + url = "http://www.example.com/" with support.transient_internet(url): response = _urlopen_with_retry(url) sock = response.fp @@ -263,7 +263,7 @@ class TimeoutTest(unittest.TestCase): def test_http_basic(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url, timeout=None): u = _urlopen_with_retry(url) self.addCleanup(u.close) @@ -271,7 +271,7 @@ def test_http_default_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -283,7 +283,7 @@ def test_http_no_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -294,7 +294,7 @@ self.assertIsNone(u.fp.raw._sock.gettimeout()) def test_http_timeout(self): - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): u = _urlopen_with_retry(url, timeout=120) self.addCleanup(u.close) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -22,8 +22,8 @@ socket.setdefaulttimeout(None) def testURLread(self): - with support.transient_internet("www.python.org"): - f = urllib.request.urlopen("http://www.python.org/") + with support.transient_internet("www.example.com"): + f = urllib.request.urlopen("http://www.example.com/") x = f.read() @@ -36,7 +36,7 @@ for transparent redirection have been written. setUp is not used for always constructing a connection to - http://www.python.org/ since there a few tests that don't use that address + http://www.example.com/ since there a few tests that don't use that address and making a connection is expensive enough to warrant minimizing unneeded connections. @@ -54,7 +54,7 @@ def test_basic(self): # Simple test expected to pass. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: for attr in ("read", "readline", "readlines", "fileno", "close", "info", "geturl"): self.assertTrue(hasattr(open_url, attr), "object returned from " @@ -63,7 +63,7 @@ def test_readlines(self): # Test both readline and readlines. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: self.assertIsInstance(open_url.readline(), bytes, "readline did not return a string") self.assertIsInstance(open_url.readlines(), list, @@ -71,7 +71,7 @@ def test_info(self): # Test 'info'. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: info_obj = open_url.info() self.assertIsInstance(info_obj, email.message.Message, "object returned by 'info' is not an " @@ -80,14 +80,14 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "https://www.python.org/" + URL = "http://www.example.com/" with self.urlopen(URL) as open_url: gotten_url = open_url.geturl() self.assertEqual(gotten_url, URL) def test_getcode(self): # test getcode() with the fancy opener to get 404 error codes - URL = "http://www.python.org/XXXinvalidXXX" + URL = "http://www.example.com/XXXinvalidXXX" with support.transient_internet(URL): open_url = urllib.request.FancyURLopener().open(URL) try: @@ -101,7 +101,7 @@ @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): # Make sure fd returned by fileno is valid. - with self.urlopen("http://www.python.org/", timeout=None) as open_url: + with self.urlopen("http://www.example.com/", timeout=None) as open_url: fd = open_url.fileno() with os.fdopen(fd, 'rb') as f: self.assertTrue(f.read(), "reading from file created using fd " @@ -147,7 +147,7 @@ def test_basic(self): # Test basic functionality. - with self.urlretrieve("http://www.python.org/") as (file_location, info): + with self.urlretrieve("http://www.example.com/") as (file_location, info): self.assertTrue(os.path.exists(file_location), "file location returned by" " urlretrieve is not a valid path") with open(file_location, 'rb') as f: @@ -156,7 +156,7 @@ def test_specified_path(self): # Make sure that specifying the location of the file to write to works. - with self.urlretrieve("http://www.python.org/", + with self.urlretrieve("http://www.example.com/", support.TESTFN) as (file_location, info): self.assertEqual(file_location, support.TESTFN) self.assertTrue(os.path.exists(file_location)) @@ -165,11 +165,11 @@ def test_header(self): # Make sure header returned as 2nd value from urlretrieve is good. - with self.urlretrieve("http://www.python.org/") as (file_location, info): + with self.urlretrieve("http://www.example.com/") as (file_location, info): self.assertIsInstance(info, email.message.Message, "info is not an instance of email.message.Message") - logo = "http://www.python.org/static/community_logos/python-logo-master-v3-TM.png" + logo = "http://www.example.com/" def test_data_header(self): with self.urlretrieve(self.logo) as (file_location, fileheaders): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -103,6 +103,10 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #20939: Avoid various network test failures due to new + redirect of http://www.python.org/ to https://www.python.org: + use http://www.example.com instead. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 07:34:32 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:34:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320939=3A_merge_from_3=2E4?= Message-ID: <3fvbyc13Dsz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/67dcb2d19dc4 changeset: 89986:67dcb2d19dc4 parent: 89982:0f6c518183ef parent: 89985:6134684ba222 user: Ned Deily date: Wed Mar 26 23:33:28 2014 -0700 summary: Issue #20939: merge from 3.4 files: Lib/test/test_site.py | 4 ++- Lib/test/test_urllib2.py | 8 +------ Lib/test/test_urllib2net.py | 10 ++++---- Lib/test/test_urllibnet.py | 26 ++++++++++++------------ Misc/NEWS | 4 +++ 5 files changed, 26 insertions(+), 26 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -414,8 +414,10 @@ @test.support.requires_resource('network') @unittest.skipUnless(sys.version_info[3] == 'final', 'only for released versions') + @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"), + 'need SSL support to download license') def test_license_exists_at_url(self): - # This test is a bit fragile since it depends on the format of the + # This test `is a bit fragile since it depends on the format of the # string displayed by license in the absence of a LICENSE file. url = license._Printer__data.split()[1] req = urllib.request.Request(url, method='HEAD') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -6,10 +6,6 @@ import socket import array import sys -try: - import ssl -except ImportError: - ssl = None import urllib.request # The proxy bypass method imported below has logic specific to the OSX @@ -1442,11 +1438,9 @@ @unittest.skipUnless(support.is_resource_enabled('network'), 'test requires network access') - @unittest.skipIf(ssl is None, - 'test requires the ssl module') def test_issue16464(self): opener = urllib.request.build_opener() - request = urllib.request.Request("http://www.python.org/~jeremy/") + request = urllib.request.Request("http://www.example.com/") self.assertEqual(None, request.data) opener.open(request, "1".encode("us-ascii")) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -83,7 +83,7 @@ def test_close(self): # calling .close() on urllib2's response objects should close the # underlying socket - url = "http://www.python.org/" + url = "http://www.example.com/" with support.transient_internet(url): response = _urlopen_with_retry(url) sock = response.fp @@ -263,7 +263,7 @@ class TimeoutTest(unittest.TestCase): def test_http_basic(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url, timeout=None): u = _urlopen_with_retry(url) self.addCleanup(u.close) @@ -271,7 +271,7 @@ def test_http_default_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -283,7 +283,7 @@ def test_http_no_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): socket.setdefaulttimeout(60) try: @@ -294,7 +294,7 @@ self.assertIsNone(u.fp.raw._sock.gettimeout()) def test_http_timeout(self): - url = "http://www.python.org" + url = "http://www.example.com" with support.transient_internet(url): u = _urlopen_with_retry(url, timeout=120) self.addCleanup(u.close) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -22,8 +22,8 @@ socket.setdefaulttimeout(None) def testURLread(self): - with support.transient_internet("www.python.org"): - f = urllib.request.urlopen("http://www.python.org/") + with support.transient_internet("www.example.com"): + f = urllib.request.urlopen("http://www.example.com/") x = f.read() @@ -36,7 +36,7 @@ for transparent redirection have been written. setUp is not used for always constructing a connection to - http://www.python.org/ since there a few tests that don't use that address + http://www.example.com/ since there a few tests that don't use that address and making a connection is expensive enough to warrant minimizing unneeded connections. @@ -54,7 +54,7 @@ def test_basic(self): # Simple test expected to pass. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: for attr in ("read", "readline", "readlines", "fileno", "close", "info", "geturl"): self.assertTrue(hasattr(open_url, attr), "object returned from " @@ -63,7 +63,7 @@ def test_readlines(self): # Test both readline and readlines. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: self.assertIsInstance(open_url.readline(), bytes, "readline did not return a string") self.assertIsInstance(open_url.readlines(), list, @@ -71,7 +71,7 @@ def test_info(self): # Test 'info'. - with self.urlopen("http://www.python.org/") as open_url: + with self.urlopen("http://www.example.com/") as open_url: info_obj = open_url.info() self.assertIsInstance(info_obj, email.message.Message, "object returned by 'info' is not an " @@ -80,14 +80,14 @@ def test_geturl(self): # Make sure same URL as opened is returned by geturl. - URL = "https://www.python.org/" + URL = "http://www.example.com/" with self.urlopen(URL) as open_url: gotten_url = open_url.geturl() self.assertEqual(gotten_url, URL) def test_getcode(self): # test getcode() with the fancy opener to get 404 error codes - URL = "http://www.python.org/XXXinvalidXXX" + URL = "http://www.example.com/XXXinvalidXXX" with support.transient_internet(URL): open_url = urllib.request.FancyURLopener().open(URL) try: @@ -101,7 +101,7 @@ @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): # Make sure fd returned by fileno is valid. - with self.urlopen("http://www.python.org/", timeout=None) as open_url: + with self.urlopen("http://www.example.com/", timeout=None) as open_url: fd = open_url.fileno() with os.fdopen(fd, 'rb') as f: self.assertTrue(f.read(), "reading from file created using fd " @@ -147,7 +147,7 @@ def test_basic(self): # Test basic functionality. - with self.urlretrieve("http://www.python.org/") as (file_location, info): + with self.urlretrieve("http://www.example.com/") as (file_location, info): self.assertTrue(os.path.exists(file_location), "file location returned by" " urlretrieve is not a valid path") with open(file_location, 'rb') as f: @@ -156,7 +156,7 @@ def test_specified_path(self): # Make sure that specifying the location of the file to write to works. - with self.urlretrieve("http://www.python.org/", + with self.urlretrieve("http://www.example.com/", support.TESTFN) as (file_location, info): self.assertEqual(file_location, support.TESTFN) self.assertTrue(os.path.exists(file_location)) @@ -165,11 +165,11 @@ def test_header(self): # Make sure header returned as 2nd value from urlretrieve is good. - with self.urlretrieve("http://www.python.org/") as (file_location, info): + with self.urlretrieve("http://www.example.com/") as (file_location, info): self.assertIsInstance(info, email.message.Message, "info is not an instance of email.message.Message") - logo = "http://www.python.org/static/community_logos/python-logo-master-v3-TM.png" + logo = "http://www.example.com/" def test_data_header(self): with self.urlretrieve(self.logo) as (file_location, fileheaders): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -125,6 +125,10 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #20939: Avoid various network test failures due to new + redirect of http://www.python.org/ to https://www.python.org: + use http://www.example.com instead. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 07:44:50 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:44:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTM5?= =?utf-8?q?=3A_remove_stray_character_from_comment?= Message-ID: <3fvcBV1jSWz7LjN@mail.python.org> http://hg.python.org/cpython/rev/b533cc11d114 changeset: 89987:b533cc11d114 branch: 3.4 parent: 89985:6134684ba222 user: Ned Deily date: Wed Mar 26 23:43:26 2014 -0700 summary: Issue #20939: remove stray character from comment files: Lib/test/test_site.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -417,7 +417,7 @@ @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"), 'need SSL support to download license') def test_license_exists_at_url(self): - # This test `is a bit fragile since it depends on the format of the + # This test is a bit fragile since it depends on the format of the # string displayed by license in the absence of a LICENSE file. url = license._Printer__data.split()[1] req = urllib.request.Request(url, method='HEAD') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 07:44:51 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 07:44:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320939=3A_merge_from_3=2E4?= Message-ID: <3fvcBW3QGgz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/ff27cb871b16 changeset: 89988:ff27cb871b16 parent: 89986:67dcb2d19dc4 parent: 89987:b533cc11d114 user: Ned Deily date: Wed Mar 26 23:44:18 2014 -0700 summary: Issue #20939: merge from 3.4 files: Lib/test/test_site.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -417,7 +417,7 @@ @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"), 'need SSL support to download license') def test_license_exists_at_url(self): - # This test `is a bit fragile since it depends on the format of the + # This test is a bit fragile since it depends on the format of the # string displayed by license in the absence of a LICENSE file. url = license._Printer__data.split()[1] req = urllib.request.Request(url, method='HEAD') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 09:40:40 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 09:40:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDY5?= =?utf-8?q?=3A_Temporarily_use_www=2Egoogle=2Ecom_while_investigating?= Message-ID: <3fvfm82KF3z7LkS@mail.python.org> http://hg.python.org/cpython/rev/adf1e04478b4 changeset: 89989:adf1e04478b4 branch: 3.4 parent: 89987:b533cc11d114 user: Ned Deily date: Thu Mar 27 01:39:28 2014 -0700 summary: Issue #21069: Temporarily use www.google.com while investigating test_urllibnet.test_fileno intermittent failures with www.example.com. files: Lib/test/test_urllibnet.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -101,7 +101,7 @@ @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): # Make sure fd returned by fileno is valid. - with self.urlopen("http://www.example.com/", timeout=None) as open_url: + with self.urlopen("http://www.google.com/", timeout=None) as open_url: fd = open_url.fileno() with os.fdopen(fd, 'rb') as f: self.assertTrue(f.read(), "reading from file created using fd " -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 09:40:41 2014 From: python-checkins at python.org (ned.deily) Date: Thu, 27 Mar 2014 09:40:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_21069=3A_merge_from_3=2E4?= Message-ID: <3fvfm93ZDTz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/8d4cace71113 changeset: 89990:8d4cace71113 parent: 89988:ff27cb871b16 parent: 89989:adf1e04478b4 user: Ned Deily date: Thu Mar 27 01:40:16 2014 -0700 summary: Issue 21069: merge from 3.4 files: Lib/test/test_urllibnet.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -101,7 +101,7 @@ @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): # Make sure fd returned by fileno is valid. - with self.urlopen("http://www.example.com/", timeout=None) as open_url: + with self.urlopen("http://www.google.com/", timeout=None) as open_url: fd = open_url.fileno() with os.fdopen(fd, 'rb') as f: self.assertTrue(f.read(), "reading from file created using fd " -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Mar 27 10:06:56 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 27 Mar 2014 10:06:56 +0100 Subject: [Python-checkins] Daily reference leaks (0f6c518183ef): sum=-4 Message-ID: results for 0f6c518183ef on branch "default" -------------------------------------------- test_site leaked [0, 0, -2] references, sum=-2 test_site leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogg1wzgb', '-x'] From python-checkins at python.org Thu Mar 27 11:34:24 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 27 Mar 2014 11:34:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461=3A_fix_reST_syntax?= Message-ID: <3fvjHN4Cfhz7LjQ@mail.python.org> http://hg.python.org/peps/rev/ce49a9c37d25 changeset: 5435:ce49a9c37d25 user: Victor Stinner date: Thu Mar 27 11:34:18 2014 +0100 summary: PEP 461: fix reST syntax files: pep-0461.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -51,7 +51,7 @@ --------------- All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, -``e``, ``E'', ``f``, ``F``, ``g``, ``G``, and any that are subsequently added +``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently ``#``, ``0``, ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 11:56:38 2014 From: python-checkins at python.org (nick.coghlan) Date: Thu, 27 Mar 2014 11:56:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_reword_maintainabi?= =?utf-8?q?lity_section_at_Alex=27s_request?= Message-ID: <3fvjn26Wc4z7Ljm@mail.python.org> http://hg.python.org/peps/rev/3ff9b547be48 changeset: 5436:3ff9b547be48 user: Nick Coghlan date: Thu Mar 27 20:56:27 2014 +1000 summary: PEP 466: reword maintainability section at Alex's request files: pep-0466.txt | 24 +++++++++--------------- 1 files changed, 9 insertions(+), 15 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -164,22 +164,16 @@ Maintainability --------------- -This policy does NOT represent a commitment by volunteer contributors to -actually backport network security related changes from the Python 3 series -to the Python 2 series. Rather, it is intended to send a clear signal to -potential corporate contributors that the core development team are willing -to accept offers of corporate assistance in putting this policy into -effect and handling the resulting increase in the Python 2 maintenance -load. +A number of developers, including Alex Gaynor and Donald Stufft, have +expressed interest in carrying out the feature backports covered by this +policy, and assisting with any additional maintenance burdens that arise +in the Python 2 series as a result. -Backporting security related fixes and enhancements to earlier versions is -a common service for commercial redistributors to offer to their customers. -This policy represents an explicit invitation to implement those changes -in the core development tree in cases where they are likely to have a broad -impact that helps improve the security of the internet as a whole, with the -assurance that the existing core development team not only won't object to -such contributions, but will actively encourage their incorporation into -the next Python 2.7 maintenance release. +This PEP is primarily about establishing the consensus needed to allow them +to carry out this work. For other core developers, this policy change +shouldn't impose any additional effort beyond potentially reviewing the +resulting patches for those developers specifically interested in the +affected modules. Documentation -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 12:22:48 2014 From: python-checkins at python.org (nick.coghlan) Date: Thu, 27 Mar 2014 12:22:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461=3A_markup_tweaks?= Message-ID: <3fvkMD1Dcsz7LjQ@mail.python.org> http://hg.python.org/peps/rev/2894f0acc316 changeset: 5437:2894f0acc316 user: Nick Coghlan date: Thu Mar 27 21:22:38 2014 +1000 summary: PEP 461: markup tweaks files: pep-0461.txt | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -64,7 +64,7 @@ b"%x" % val -is equivalent to +is equivalent to:: ("%x" % val).encode("ascii") @@ -92,7 +92,7 @@ ``%s`` is included for two reasons: 1) `b` is already a format code for ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x -code uses ``%s``; however, it is restricted in what it will accept:: +code uses ``%s``; however, it is restricted in what it will accept: - input type supports ``Py_buffer`` [6]_? use it to collect the necessary bytes @@ -128,7 +128,7 @@ into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining ``__bytes__`` would not be appropriate -but a readable/informative representation is needed [8]. +but a readable/informative representation is needed [8]_. Examples:: @@ -178,8 +178,8 @@ ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. -A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9], also -exists. +A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9]_, +also exists. Objections -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 14:14:39 2014 From: python-checkins at python.org (giampaolo.rodola) Date: Thu, 27 Mar 2014 14:14:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_cosmetic_enhancement?= =?utf-8?q?_to_provide_a_more_readable_repr=28=29esentation_of?= Message-ID: <3fvmrH69hbz7LjX@mail.python.org> http://hg.python.org/cpython/rev/978a08a5856f changeset: 89991:978a08a5856f user: Giampaolo Rodola' date: Thu Mar 27 14:14:16 2014 +0100 summary: Minor cosmetic enhancement to provide a more readable repr()esentation of Extension instances: - + files: Lib/distutils/extension.py | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/distutils/extension.py b/Lib/distutils/extension.py --- a/Lib/distutils/extension.py +++ b/Lib/distutils/extension.py @@ -131,6 +131,14 @@ msg = "Unknown Extension options: %s" % options warnings.warn(msg) + def __repr__(self): + return '<%s.%s(%r) at %#x>' % ( + self.__class__.__module__, + self.__class__.__name__, + self.name, + id(self)) + + def read_setup_file(filename): """Reads a Setup file and returns Extension instances.""" from distutils.sysconfig import (parse_makefile, expand_makefile_vars, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 16:32:30 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 16:32:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2Esignature=3A_Use?= =?utf-8?q?_enum_for_parameter_kind_constants=2E_Closes_=2319573?= Message-ID: <3fvqvL3Rmsz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/c2b94f891c88 changeset: 89992:c2b94f891c88 user: Yury Selivanov date: Thu Mar 27 11:23:13 2014 -0400 summary: inspect.signature: Use enum for parameter kind constants. Closes #19573 Patch by Antony Lee. files: Lib/inspect.py | 31 +++++++++++++++---------------- Misc/NEWS | 2 ++ 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -32,6 +32,7 @@ 'Yury Selivanov ') import ast +import enum import importlib.machinery import itertools import linecache @@ -2027,24 +2028,22 @@ pass -class _ParameterKind(int): - def __new__(self, *args, name): - obj = int.__new__(self, *args) - obj._name = name - return obj +class _ParameterKind(enum.IntEnum): + POSITIONAL_ONLY = 0 + POSITIONAL_OR_KEYWORD = 1 + VAR_POSITIONAL = 2 + KEYWORD_ONLY = 3 + VAR_KEYWORD = 4 def __str__(self): - return self._name - - def __repr__(self): - return '<_ParameterKind: {!r}>'.format(self._name) - - -_POSITIONAL_ONLY = _ParameterKind(0, name='POSITIONAL_ONLY') -_POSITIONAL_OR_KEYWORD = _ParameterKind(1, name='POSITIONAL_OR_KEYWORD') -_VAR_POSITIONAL = _ParameterKind(2, name='VAR_POSITIONAL') -_KEYWORD_ONLY = _ParameterKind(3, name='KEYWORD_ONLY') -_VAR_KEYWORD = _ParameterKind(4, name='VAR_KEYWORD') + return self._name_ + + +_POSITIONAL_ONLY = _ParameterKind.POSITIONAL_ONLY +_POSITIONAL_OR_KEYWORD = _ParameterKind.POSITIONAL_OR_KEYWORD +_VAR_POSITIONAL = _ParameterKind.VAR_POSITIONAL +_KEYWORD_ONLY = _ParameterKind.KEYWORD_ONLY +_VAR_KEYWORD = _ParameterKind.VAR_KEYWORD class Parameter: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -105,6 +105,8 @@ - Issue #19748: On AIX, time.mktime() now raises an OverflowError for year outsize range [1902; 2037]. +- Issue #19573: inspect.signature: Use enum for parameter kind constants. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 16:32:32 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 16:32:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2Esignature=3A_Mak?= =?utf-8?q?e_Signature_and_Parameter_picklable=2E_Closes_=2320726?= Message-ID: <3fvqvN2xpHz7LkC@mail.python.org> http://hg.python.org/cpython/rev/0a8e3c910c0a changeset: 89993:0a8e3c910c0a user: Yury Selivanov date: Thu Mar 27 11:31:43 2014 -0400 summary: inspect.signature: Make Signature and Parameter picklable. Closes #20726 files: Doc/whatsnew/3.5.rst | 3 + Lib/inspect.py | 20 ++++++++++ Lib/test/test_inspect.py | 53 ++++++++++++++++++++++++++++ Misc/NEWS | 2 + 4 files changed, 78 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -137,6 +137,9 @@ * :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager` (contributed by Claudiu Popa in :issue:`20627`). +* :class:`inspect.Signature` and :class:`inspect.Parameter` are now + picklable (contributed by Yury Selivanov in :issue:`20726`). + Optimizations ============= diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -2106,6 +2106,18 @@ self._partial_kwarg = _partial_kwarg + def __reduce__(self): + return (type(self), + (self._name, self._kind), + {'_partial_kwarg': self._partial_kwarg, + '_default': self._default, + '_annotation': self._annotation}) + + def __setstate__(self, state): + self._partial_kwarg = state['_partial_kwarg'] + self._default = state['_default'] + self._annotation = state['_annotation'] + @property def name(self): return self._name @@ -2658,6 +2670,14 @@ ''' return args[0]._bind(args[1:], kwargs, partial=True) + def __reduce__(self): + return (type(self), + (tuple(self._parameters.values()),), + {'_return_annotation': self._return_annotation}) + + def __setstate__(self, state): + self._return_annotation = state['_return_annotation'] + def __str__(self): result = [] render_pos_only_separator = False diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -8,6 +8,7 @@ import os from os.path import normcase import _pickle +import pickle import re import shutil import sys @@ -73,6 +74,7 @@ for i in range(2): yield i + class TestPredicates(IsTestBase): def test_sixteen(self): count = len([x for x in dir(inspect) if x.startswith('is')]) @@ -1597,6 +1599,17 @@ self.assertRaises(TypeError, inspect.getgeneratorlocals, (2,3)) +class MySignature(inspect.Signature): + # Top-level to make it picklable; + # used in test_signature_object_pickle + pass + +class MyParameter(inspect.Parameter): + # Top-level to make it picklable; + # used in test_signature_object_pickle + pass + + class TestSignatureObject(unittest.TestCase): @staticmethod def signature(func): @@ -1654,6 +1667,36 @@ with self.assertRaisesRegex(ValueError, 'follows default argument'): S((pkd, pk)) + def test_signature_object_pickle(self): + def foo(a, b, *, c:1={}, **kw) -> {42:'ham'}: pass + foo_partial = functools.partial(foo, a=1) + + sig = inspect.signature(foo_partial) + self.assertTrue(sig.parameters['a']._partial_kwarg) + + for ver in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_ver=ver, subclass=False): + sig_pickled = pickle.loads(pickle.dumps(sig, ver)) + self.assertEqual(sig, sig_pickled) + self.assertTrue(sig_pickled.parameters['a']._partial_kwarg) + + # Test that basic sub-classing works + sig = inspect.signature(foo) + myparam = MyParameter(name='z', kind=inspect.Parameter.POSITIONAL_ONLY) + myparams = collections.OrderedDict(sig.parameters, a=myparam) + mysig = MySignature().replace(parameters=myparams.values(), + return_annotation=sig.return_annotation) + self.assertTrue(isinstance(mysig, MySignature)) + self.assertTrue(isinstance(mysig.parameters['z'], MyParameter)) + + for ver in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_ver=ver, subclass=True): + sig_pickled = pickle.loads(pickle.dumps(mysig, ver)) + self.assertEqual(mysig, sig_pickled) + self.assertTrue(isinstance(sig_pickled, MySignature)) + self.assertTrue(isinstance(sig_pickled.parameters['z'], + MyParameter)) + def test_signature_immutability(self): def test(a): pass @@ -2845,6 +2888,16 @@ ba4 = inspect.signature(bar).bind(1) self.assertNotEqual(ba, ba4) + def test_signature_bound_arguments_pickle(self): + def foo(a, b, *, c:1={}, **kw) -> {42:'ham'}: pass + sig = inspect.signature(foo) + ba = sig.bind(20, 30, z={}) + + for ver in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_ver=ver): + ba_pickled = pickle.loads(pickle.dumps(ba, ver)) + self.assertEqual(ba, ba_pickled) + class TestSignaturePrivateHelpers(unittest.TestCase): def test_signature_get_bound_param(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -107,6 +107,8 @@ - Issue #19573: inspect.signature: Use enum for parameter kind constants. +- Issue #20726: inspect.signature: Make Signature and Parameter picklable. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 17:12:02 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 17:12:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2ESignature=3A_Add?= =?utf-8?q?_=27Signature=2Efrom=5Fcallable=27_classmethod=2E_Closes_=23173?= =?utf-8?q?73?= Message-ID: <3fvrmy6cvPz7Ll5@mail.python.org> http://hg.python.org/cpython/rev/8a4e44473fdd changeset: 89994:8a4e44473fdd user: Yury Selivanov date: Thu Mar 27 12:09:24 2014 -0400 summary: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373 files: Doc/library/inspect.rst | 12 +++ Doc/whatsnew/3.5.rst | 4 + Lib/inspect.py | 86 ++++++++++++++++++--------- Lib/test/test_inspect.py | 13 ++++ Misc/NEWS | 2 + 5 files changed, 87 insertions(+), 30 deletions(-) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -506,6 +506,18 @@ >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. This method simplifies subclassing of :class:`Signature`: + + :: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -140,6 +140,10 @@ * :class:`inspect.Signature` and :class:`inspect.Parameter` are now picklable (contributed by Yury Selivanov in :issue:`20726`). +* New class method :meth:`inspect.Signature.from_callable`, which makes + subclassing of :class:`~inspect.Signature` easier (contributed + by Yury Selivanov and Eric Snow in :issue:`17373`). + Optimizations ============= diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -969,7 +969,8 @@ sig = _signature_internal(func, follow_wrapper_chains=False, - skip_bound_arg=False) + skip_bound_arg=False, + sigcls=Signature) except Exception as ex: # Most of the times 'signature' will raise ValueError. # But, it can also raise AttributeError, and, maybe something @@ -1861,7 +1862,10 @@ return _signature_fromstr(cls, func, s, skip_bound_arg) -def _signature_internal(obj, follow_wrapper_chains=True, skip_bound_arg=True): +def _signature_internal(obj, *, + follow_wrapper_chains=True, + skip_bound_arg=True, + sigcls): if not callable(obj): raise TypeError('{!r} is not a callable object'.format(obj)) @@ -1869,9 +1873,12 @@ if isinstance(obj, types.MethodType): # In this case we skip the first parameter of the underlying # function (usually `self` or `cls`). - sig = _signature_internal(obj.__func__, - follow_wrapper_chains, - skip_bound_arg) + sig = _signature_internal( + obj.__func__, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) + if skip_bound_arg: return _signature_bound_method(sig) else: @@ -1902,9 +1909,12 @@ # (usually `self`, or `cls`) will not be passed # automatically (as for boundmethods) - wrapped_sig = _signature_internal(partialmethod.func, - follow_wrapper_chains, - skip_bound_arg) + wrapped_sig = _signature_internal( + partialmethod.func, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) + sig = _signature_get_partial(wrapped_sig, partialmethod, (None,)) first_wrapped_param = tuple(wrapped_sig.parameters.values())[0] @@ -1915,16 +1925,18 @@ if isfunction(obj) or _signature_is_functionlike(obj): # If it's a pure Python function, or an object that is duck type # of a Python function (Cython functions, for instance), then: - return Signature.from_function(obj) + return sigcls.from_function(obj) if _signature_is_builtin(obj): - return _signature_from_builtin(Signature, obj, + return _signature_from_builtin(sigcls, obj, skip_bound_arg=skip_bound_arg) if isinstance(obj, functools.partial): - wrapped_sig = _signature_internal(obj.func, - follow_wrapper_chains, - skip_bound_arg) + wrapped_sig = _signature_internal( + obj.func, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) return _signature_get_partial(wrapped_sig, obj) sig = None @@ -1935,23 +1947,29 @@ # in its metaclass call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: - sig = _signature_internal(call, - follow_wrapper_chains, - skip_bound_arg) + sig = _signature_internal( + call, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) else: # Now we check if the 'obj' class has a '__new__' method new = _signature_get_user_defined_method(obj, '__new__') if new is not None: - sig = _signature_internal(new, - follow_wrapper_chains, - skip_bound_arg) + sig = _signature_internal( + new, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) else: # Finally, we should have at least __init__ implemented init = _signature_get_user_defined_method(obj, '__init__') if init is not None: - sig = _signature_internal(init, - follow_wrapper_chains, - skip_bound_arg) + sig = _signature_internal( + init, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) if sig is None: # At this point we know, that `obj` is a class, with no user- @@ -1973,7 +1991,7 @@ if text_sig: # If 'obj' class has a __text_signature__ attribute: # return a signature based on it - return _signature_fromstr(Signature, obj, text_sig) + return _signature_fromstr(sigcls, obj, text_sig) # No '__text_signature__' was found for the 'obj' class. # Last option is to check if its '__init__' is @@ -1993,9 +2011,11 @@ call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: try: - sig = _signature_internal(call, - follow_wrapper_chains, - skip_bound_arg) + sig = _signature_internal( + call, + follow_wrapper_chains=follow_wrapper_chains, + skip_bound_arg=skip_bound_arg, + sigcls=sigcls) except ValueError as ex: msg = 'no signature found for {!r}'.format(obj) raise ValueError(msg) from ex @@ -2015,10 +2035,6 @@ raise ValueError('callable {!r} is not supported by signature'.format(obj)) -def signature(obj): - '''Get a signature object for the passed callable.''' - return _signature_internal(obj) - class _void: '''A private marker - used in Parameter & Signature''' @@ -2464,6 +2480,10 @@ def from_builtin(cls, func): return _signature_from_builtin(cls, func) + @classmethod + def from_callable(cls, obj): + return _signature_internal(obj, sigcls=cls) + @property def parameters(self): return self._parameters @@ -2723,6 +2743,12 @@ return rendered + +def signature(obj): + '''Get a signature object for the passed callable.''' + return Signature.from_callable(obj) + + def _main(): """ Logic for inspecting an object given at command line """ import argparse diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -2517,6 +2517,19 @@ self.assertEqual(self.signature(Spam.foo), self.signature(Ham.foo)) + def test_signature_from_callable_python_obj(self): + class MySignature(inspect.Signature): pass + def foo(a, *, b:1): pass + foo_sig = MySignature.from_callable(foo) + self.assertTrue(isinstance(foo_sig, MySignature)) + + @unittest.skipIf(MISSING_C_DOCSTRINGS, + "Signature information for builtins requires docstrings") + def test_signature_from_callable_builtin_obj(self): + class MySignature(inspect.Signature): pass + sig = MySignature.from_callable(_pickle.Pickler) + self.assertTrue(isinstance(sig, MySignature)) + class TestParameterObject(unittest.TestCase): def test_signature_parameter_kinds(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -109,6 +109,8 @@ - Issue #20726: inspect.signature: Make Signature and Parameter picklable. +- Issue #17373: Add inspect.Signature.from_callable method. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 17:22:01 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 17:22:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=2Etests=3A_Autodis?= =?utf-8?q?cover_asyncio_tests=2E_Patch_by_Vajrasky_Kok=2E_Closes_=2320668?= Message-ID: <3fvs0T1bpLz7LlP@mail.python.org> http://hg.python.org/cpython/rev/bcc77493249c changeset: 89995:bcc77493249c user: Yury Selivanov date: Thu Mar 27 12:21:20 2014 -0400 summary: asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668 files: Lib/test/test_asyncio/__init__.py | 22 ++++++++---------- Misc/NEWS | 3 ++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_asyncio/__init__.py b/Lib/test/test_asyncio/__init__.py --- a/Lib/test/test_asyncio/__init__.py +++ b/Lib/test/test_asyncio/__init__.py @@ -10,20 +10,18 @@ def suite(): - tests_file = os.path.join(os.path.dirname(__file__), 'tests.txt') - with open(tests_file) as fp: - test_names = fp.read().splitlines() tests = unittest.TestSuite() loader = unittest.TestLoader() - for test_name in test_names: - mod_name = 'test.' + test_name - try: - __import__(mod_name) - except unittest.SkipTest: - pass - else: - mod = sys.modules[mod_name] - tests.addTests(loader.loadTestsFromModule(mod)) + for fn in os.listdir(os.path.dirname(__file__)): + if fn.startswith("test") and fn.endswith(".py"): + mod_name = 'test.test_asyncio.' + fn[:-3] + try: + __import__(mod_name) + except unittest.SkipTest: + pass + else: + mod = sys.modules[mod_name] + tests.addTests(loader.loadTestsFromModule(mod)) return tests diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -135,6 +135,9 @@ redirect of http://www.python.org/ to https://www.python.org: use http://www.example.com instead. +- Issue #20668: asyncio tests no longer rely on tests.txt file. + (Patch by Vajrasky Kok) + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 17:42:22 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 17:42:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2Esignature=3A_Imp?= =?utf-8?q?rove_repr_of_Signature_and_Parameter=2E_Closes_=2320378?= Message-ID: <3fvsRy6Rm4z7LlT@mail.python.org> http://hg.python.org/cpython/rev/3f9a81297b39 changeset: 89996:3f9a81297b39 user: Yury Selivanov date: Thu Mar 27 12:41:53 2014 -0400 summary: inspect.signature: Improve repr of Signature and Parameter. Closes #20378 files: Lib/inspect.py | 8 ++++++-- Lib/test/test_inspect.py | 4 ++++ Misc/NEWS | 2 ++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -2192,8 +2192,8 @@ return formatted def __repr__(self): - return '<{} at {:#x} {!r}>'.format(self.__class__.__name__, - id(self), self.name) + return '<{} at {:#x} "{}">'.format(self.__class__.__name__, + id(self), self) def __eq__(self, other): # NB: We deliberately do not compare '_partial_kwarg' attributes @@ -2698,6 +2698,10 @@ def __setstate__(self, state): self._return_annotation = state['_return_annotation'] + def __repr__(self): + return '<{} at {:#x} "{}">'.format(self.__class__.__name__, + id(self), self) + def __str__(self): result = [] render_pos_only_separator = False diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1667,6 +1667,9 @@ with self.assertRaisesRegex(ValueError, 'follows default argument'): S((pkd, pk)) + self.assertTrue(repr(sig).startswith(' {42:'ham'}: pass foo_partial = functools.partial(foo, a=1) @@ -2575,6 +2578,7 @@ p.replace(kind=inspect.Parameter.VAR_POSITIONAL) self.assertTrue(repr(p).startswith(' http://hg.python.org/peps/rev/c73784a49b33 changeset: 5438:c73784a49b33 user: Ethan Furman date: Thu Mar 27 11:06:08 2014 -0700 summary: fixed foot-notes; added %b and clarified %s as a synonym for %b files: pep-0461.txt | 33 ++++++++++++++++++++------------- 1 files changed, 20 insertions(+), 13 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -94,14 +94,22 @@ ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x code uses ``%s``; however, it is restricted in what it will accept: - - input type supports ``Py_buffer`` [6]_? + - input type supports ``Py_buffer`` [4]_? use it to collect the necessary bytes - input type is something else? - use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError`` + use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError`` -In particular, ``%s`` will not accept numbers (use a numeric format code for -that), nor ``str`` (encode it to ``bytes``). +In particular, ``%s`` will not accept numbers nor ``str``. ``str`` is rejected +as the string to bytes conversion requires an encoding, and we are refusing to +guess; numbers are rejected because: + + - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) + + - allowing numbers would lead to ambiguity between numbers and textual + representations of numbers (3.14 vs '3.14') + + - given the nature of wire formats, explicit is definitely better than implicit Examples:: @@ -128,7 +136,7 @@ into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining ``__bytes__`` would not be appropriate -but a readable/informative representation is needed [8]_. +but a readable/informative representation is needed [6]_. Examples:: @@ -178,17 +186,18 @@ ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. -A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9]_, +A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [7]_, also exists. Objections ========== -The objections raised against this PEP were mainly variations on two themes:: +The objections raised against this PEP were mainly variations on two themes: - the ``bytes`` and ``bytearray`` types are for pure binary data, with no assumptions about encodings + - offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 ``str``/``unicode`` text model @@ -213,13 +222,11 @@ .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html -.. [4] to use a str object in a bytes interpolation, encode it first -.. [5] %c is not an exception as neither of its possible arguments are str -.. [6] http://docs.python.org/3/c-api/buffer.html +.. [4] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` -.. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ -.. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html -.. [9] http://python.org/dev/peps/pep-0460/ +.. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ +.. [6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html +.. [7] http://python.org/dev/peps/pep-0460/ Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 19:15:14 2014 From: python-checkins at python.org (ethan.furman) Date: Thu, 27 Mar 2014 19:15:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_remembered_to_save_changes_be?= =?utf-8?q?fore_committing_=3A/?= Message-ID: <3fvvW647qcz7LjQ@mail.python.org> http://hg.python.org/peps/rev/8ae8bf3e49c7 changeset: 5439:8ae8bf3e49c7 user: Ethan Furman date: Thu Mar 27 11:15:06 2014 -0700 summary: remembered to save changes before committing :/ files: pep-0461.txt | 37 +++++++++++++++++-------------------- 1 files changed, 17 insertions(+), 20 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -55,7 +55,8 @@ to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently ``#``, ``0``, ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only -non-numeric codes allowed are ``c``, ``s``, and ``a``. +non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a +synonym for b). For the numeric codes, the only difference between ``str`` and ``bytes`` (or ``bytearray``) interpolation is that the results from these codes will be @@ -90,9 +91,8 @@ >>> b'%c' % b'a' b'a' -``%s`` is included for two reasons: 1) `b` is already a format code for -``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x -code uses ``%s``; however, it is restricted in what it will accept: +``%b`` will insert a series of bytes. These bytes are collected in one of two +ways: - input type supports ``Py_buffer`` [4]_? use it to collect the necessary bytes @@ -100,7 +100,7 @@ - input type is something else? use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError`` -In particular, ``%s`` will not accept numbers nor ``str``. ``str`` is rejected +In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: @@ -111,23 +111,26 @@ - given the nature of wire formats, explicit is definitely better than implicit +``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code +bases easier to maintain. Python 3 only code should use ``%b``. + Examples:: - >>> b'%s' % b'abc' + >>> b'%b' % b'abc' b'abc' - >>> b'%s' % 'some string'.encode('utf8') + >>> b'%b' % 'some string'.encode('utf8') b'some string' - >>> b'%s' % 3.14 + >>> b'%b' % 3.14 Traceback (most recent call last): ... - TypeError: b'%s' does not accept numbers, use a numeric code instead + TypeError: b'%b' does not accept 'float' - >>> b'%s' % 'hello world!' + >>> b'%b' % 'hello world!' Traceback (most recent call last): ... - TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` + TypeError: b'%b' does not accept 'str' ``%a`` will give the equivalent of @@ -159,20 +162,14 @@ Proposed variations =================== -It was suggested to let ``%s`` accept numbers, but since numbers have their own -format codes this idea was discarded. - -It has been suggested to use ``%b`` for bytes as well as ``%s``. This was -rejected as not adding any value either in clarity or simplicity. - It has been proposed to automatically use ``.encode('ascii','strict')`` for -``str`` arguments to ``%s``. +``str`` arguments to ``%b``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. -It has been proposed to have ``%s`` return the ascii-encoded repr when the -value is a ``str`` (b'%s' % 'abc' --> b"'abc'"). +It has been proposed to have ``%b`` return the ascii-encoded repr when the +value is a ``str`` (b'%b' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 20:46:14 2014 From: python-checkins at python.org (ethan.furman) Date: Thu, 27 Mar 2014 20:46:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_add_section_on_Python_2_compa?= =?utf-8?q?tibility?= Message-ID: <3fvxX61jbyz7LjQ@mail.python.org> http://hg.python.org/peps/rev/b913b974028e changeset: 5440:b913b974028e user: Ethan Furman date: Thu Mar 27 12:46:04 2014 -0700 summary: add section on Python 2 compatibility files: pep-0461.txt | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -159,6 +159,20 @@ ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. +Compatibility with Python 2 +=========================== + +As noted above, ``%s`` is being included solely to help ease migration from, +and/or have a single code base with, Python 2. This is important as there +are modules both in the wild and behind closed doors that currently use the +Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` +as a bytes interpolator. + +However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will +immediately be deprecated, but not removed until the next major Python +release. + + Proposed variations =================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 21:42:44 2014 From: python-checkins at python.org (ethan.furman) Date: Thu, 27 Mar 2014 21:42:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461=3A_add_latest_post_da?= =?utf-8?q?te?= Message-ID: <3fvynJ086Jz7LjQ@mail.python.org> http://hg.python.org/peps/rev/a86c63cfecda changeset: 5441:a86c63cfecda user: Ethan Furman date: Thu Mar 27 13:42:38 2014 -0700 summary: PEP 461: add latest post date files: pep-0461.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -8,7 +8,8 @@ Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 -Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25 +Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, + 2014-03-27 Resolution: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 22:20:28 2014 From: python-checkins at python.org (ethan.furman) Date: Thu, 27 Mar 2014 22:20:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461=3A_marked_as_accepted?= =?utf-8?q?=3B_added_comment_about_using_=25a_instead_of_=25r?= Message-ID: <3fvzcr4kJZz7LjX@mail.python.org> http://hg.python.org/peps/rev/d7d4949e2cd1 changeset: 5442:d7d4949e2cd1 user: Ethan Furman date: Thu Mar 27 14:20:20 2014 -0700 summary: PEP 461: marked as accepted; added comment about using %a instead of %r files: pep-0461.txt | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -3,14 +3,14 @@ Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, 2014-03-27 -Resolution: +Resolution: http://mail.python.org/pipermail/python-dev/2014-March/133621.html Abstract @@ -157,7 +157,8 @@ Unsupported codes ----------------- -``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. +``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported +(``%a`` is a good alternative, though). Compatibility with Python 2 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 22:27:00 2014 From: python-checkins at python.org (ethan.furman) Date: Thu, 27 Mar 2014 22:27:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461=3A_remove_ambiguity_a?= =?utf-8?q?round_=27major_release=27?= Message-ID: <3fvzmN1fqrz7Ll5@mail.python.org> http://hg.python.org/peps/rev/ca86bbf44482 changeset: 5443:ca86bbf44482 user: Ethan Furman date: Thu Mar 27 14:26:53 2014 -0700 summary: PEP 461: remove ambiguity around 'major release' files: pep-0461.txt | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/pep-0461.txt b/pep-0461.txt --- a/pep-0461.txt +++ b/pep-0461.txt @@ -171,8 +171,7 @@ as a bytes interpolator. However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will -immediately be deprecated, but not removed until the next major Python -release. +immediately be deprecated, but not removed from the 3.x series. Proposed variations -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Mar 27 23:05:40 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 23:05:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpby50ZXN0?= =?utf-8?q?s=3A_Autodiscover_asyncio_tests=2E_Patch_by_Vajrasky_Kok=2E_Clo?= =?utf-8?q?ses_=2320668?= Message-ID: <3fw0d04jdWz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/07984815003f changeset: 89997:07984815003f branch: 3.4 parent: 89989:adf1e04478b4 user: Yury Selivanov date: Thu Mar 27 12:21:20 2014 -0400 summary: asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668 files: Lib/test/test_asyncio/__init__.py | 22 ++++++++---------- Misc/NEWS | 3 ++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_asyncio/__init__.py b/Lib/test/test_asyncio/__init__.py --- a/Lib/test/test_asyncio/__init__.py +++ b/Lib/test/test_asyncio/__init__.py @@ -10,20 +10,18 @@ def suite(): - tests_file = os.path.join(os.path.dirname(__file__), 'tests.txt') - with open(tests_file) as fp: - test_names = fp.read().splitlines() tests = unittest.TestSuite() loader = unittest.TestLoader() - for test_name in test_names: - mod_name = 'test.' + test_name - try: - __import__(mod_name) - except unittest.SkipTest: - pass - else: - mod = sys.modules[mod_name] - tests.addTests(loader.loadTestsFromModule(mod)) + for fn in os.listdir(os.path.dirname(__file__)): + if fn.startswith("test") and fn.endswith(".py"): + mod_name = 'test.test_asyncio.' + fn[:-3] + try: + __import__(mod_name) + except unittest.SkipTest: + pass + else: + mod = sys.modules[mod_name] + tests.addTests(loader.loadTestsFromModule(mod)) return tests diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -107,6 +107,9 @@ redirect of http://www.python.org/ to https://www.python.org: use http://www.example.com instead. +- Issue #20668: asyncio tests no longer rely on tests.txt file. + (Patch by Vajrasky Kok) + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 23:28:06 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 23:28:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=3A_Fix_getcallargs?= =?utf-8?q?=28=29_to_raise_correct_TypeError?= Message-ID: <3fw16t4dqnz7LjS@mail.python.org> http://hg.python.org/cpython/rev/3de2e729d0fb changeset: 89998:3de2e729d0fb parent: 89996:3f9a81297b39 user: Yury Selivanov date: Thu Mar 27 18:23:03 2014 -0400 summary: inspect: Fix getcallargs() to raise correct TypeError ... for missing keyword-only arguments. Patch by Jeremiah Lowin. Closes #20816. files: Lib/inspect.py | 2 +- Lib/test/test_inspect.py | 8 ++++++++ Misc/NEWS | 3 +++ 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1210,7 +1210,7 @@ missing = 0 for kwarg in kwonlyargs: if kwarg not in arg2value: - if kwarg in kwonlydefaults: + if kwonlydefaults and kwarg in kwonlydefaults: arg2value[kwarg] = kwonlydefaults[kwarg] else: missing += 1 diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1208,6 +1208,14 @@ self.assertEqualException(f3, '1, 2') self.assertEqualException(f3, '1, 2, a=1, b=2') + # issue #20816: getcallargs() fails to iterate over non-existent + # kwonlydefaults and raises a wrong TypeError + def f5(*, a): pass + with self.assertRaisesRegex(TypeError, + 'missing 1 required keyword-only'): + inspect.getcallargs(f5) + + class TestGetcallargsMethods(TestGetcallargsFunctions): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -113,6 +113,9 @@ - Issue #20378: Improve repr of inspect.Signature and inspect.Parameter. +- Issue #20816: Fix inspect.getcallargs() to raise correct TypeError for + missing keyword-only arguments. Patch by Jeremiah Lowin. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 23:28:07 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 23:28:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdDogRml4?= =?utf-8?q?_getcallargs=28=29_to_raise_correct_TypeError?= Message-ID: <3fw16v6WR2z7LjS@mail.python.org> http://hg.python.org/cpython/rev/070dfca74610 changeset: 89999:070dfca74610 branch: 3.4 parent: 89997:07984815003f user: Yury Selivanov date: Thu Mar 27 18:23:03 2014 -0400 summary: inspect: Fix getcallargs() to raise correct TypeError ... for missing keyword-only arguments. Patch by Jeremiah Lowin. Closes #20816. files: Lib/inspect.py | 2 +- Lib/test/test_inspect.py | 8 ++++++++ Misc/NEWS | 3 +++ 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1208,7 +1208,7 @@ missing = 0 for kwarg in kwonlyargs: if kwarg not in arg2value: - if kwarg in kwonlydefaults: + if kwonlydefaults and kwarg in kwonlydefaults: arg2value[kwarg] = kwonlydefaults[kwarg] else: missing += 1 diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1206,6 +1206,14 @@ self.assertEqualException(f3, '1, 2') self.assertEqualException(f3, '1, 2, a=1, b=2') + # issue #20816: getcallargs() fails to iterate over non-existent + # kwonlydefaults and raises a wrong TypeError + def f5(*, a): pass + with self.assertRaisesRegex(TypeError, + 'missing 1 required keyword-only'): + inspect.getcallargs(f5) + + class TestGetcallargsMethods(TestGetcallargsFunctions): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -83,6 +83,9 @@ - Issue #19748: On AIX, time.mktime() now raises an OverflowError for year outsize range [1902; 2037]. +- Issue #20816: Fix inspect.getcallargs() to raise correct TypeError for + missing keyword-only arguments. Patch by Jeremiah Lowin. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 23:45:43 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 23:45:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=3A_Fix_getcallargs?= =?utf-8?q?=28=29_to_fail_correctly_if_more_than_3_args_are_missing=2E?= Message-ID: <3fw1WC03lxz7LjX@mail.python.org> http://hg.python.org/cpython/rev/35302cc4fc93 changeset: 90000:35302cc4fc93 parent: 89998:3de2e729d0fb user: Yury Selivanov date: Thu Mar 27 18:42:52 2014 -0400 summary: inspect: Fix getcallargs() to fail correctly if more than 3 args are missing. Patch by Jeremiah Lowin. Closes #20817. files: Lib/inspect.py | 2 +- Lib/test/test_inspect.py | 6 ++++++ Misc/NEWS | 3 +++ 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1127,7 +1127,7 @@ elif missing == 2: s = "{} and {}".format(*names) else: - tail = ", {} and {}".format(names[-2:]) + tail = ", {} and {}".format(*names[-2:]) del names[-2:] s = ", ".join(names) + tail raise TypeError("%s() missing %i required %s argument%s: %s" % diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1216,6 +1216,12 @@ inspect.getcallargs(f5) + # issue20817: + def f6(a, b, c): + pass + with self.assertRaisesRegex(TypeError, "'a', 'b' and 'c'"): + inspect.getcallargs(f6) + class TestGetcallargsMethods(TestGetcallargsFunctions): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -116,6 +116,9 @@ - Issue #20816: Fix inspect.getcallargs() to raise correct TypeError for missing keyword-only arguments. Patch by Jeremiah Lowin. +- Issue #20817: Fix inspect.getcallargs() to fail correctly if more + than 3 arguments are missing. Patch by Jeremiah Lowin. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Mar 27 23:45:44 2014 From: python-checkins at python.org (yury.selivanov) Date: Thu, 27 Mar 2014 23:45:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdDogRml4?= =?utf-8?q?_getcallargs=28=29_to_fail_correctly_if_more_than_3_args_are_mi?= =?utf-8?q?ssing=2E?= Message-ID: <3fw1WD26Fdz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/9f06cbb7962b changeset: 90001:9f06cbb7962b branch: 3.4 parent: 89999:070dfca74610 user: Yury Selivanov date: Thu Mar 27 18:42:52 2014 -0400 summary: inspect: Fix getcallargs() to fail correctly if more than 3 args are missing. Patch by Jeremiah Lowin. Closes #20817. files: Lib/inspect.py | 2 +- Lib/test/test_inspect.py | 6 ++++++ Misc/NEWS | 3 +++ 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1125,7 +1125,7 @@ elif missing == 2: s = "{} and {}".format(*names) else: - tail = ", {} and {}".format(names[-2:]) + tail = ", {} and {}".format(*names[-2:]) del names[-2:] s = ", ".join(names) + tail raise TypeError("%s() missing %i required %s argument%s: %s" % diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1214,6 +1214,12 @@ inspect.getcallargs(f5) + # issue20817: + def f6(a, b, c): + pass + with self.assertRaisesRegex(TypeError, "'a', 'b' and 'c'"): + inspect.getcallargs(f6) + class TestGetcallargsMethods(TestGetcallargsFunctions): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -86,6 +86,9 @@ - Issue #20816: Fix inspect.getcallargs() to raise correct TypeError for missing keyword-only arguments. Patch by Jeremiah Lowin. +- Issue #20817: Fix inspect.getcallargs() to fail correctly if more + than 3 arguments are missing. Patch by Jeremiah Lowin. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 00:44:37 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 00:44:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzY2NzY6?= =?utf-8?q?_Ensure_a_meaningful_exception_is_raised_when_attempting?= Message-ID: <3fw2q94G2dz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/74faca1ac59c changeset: 90002:74faca1ac59c branch: 2.7 parent: 89983:426a7046cdb0 user: Ned Deily date: Thu Mar 27 16:38:32 2014 -0700 summary: Issue #6676: Ensure a meaningful exception is raised when attempting to parse more than one XML document per pyexpat xmlparser instance. (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with suggested wording by David Gutteridge) files: Doc/library/pyexpat.rst | 8 +++++++- Lib/test/test_pyexpat.py | 11 +++++++++++ Misc/NEWS | 5 +++++ Modules/pyexpat.c | 2 +- 4 files changed, 24 insertions(+), 2 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -103,6 +103,10 @@ http://www.python.org/ns/ elem1 elem2 + Due to limitations in the ``Expat`` library used by :mod:`pyexpat`, + the :class:`xmlparser` instance returned can only be used to parse a single + XML document. Call ``ParserCreate`` for each document to provide unique + parser instances. .. seealso:: @@ -122,7 +126,9 @@ Parses the contents of the string *data*, calling the appropriate handler functions to process the parsed data. *isfinal* must be true on the final call - to this method. *data* can be the empty string at any time. + to this method; it allows the parsing of a single file in fragments, + not the submission of multiple files. + *data* can be the empty string at any time. .. method:: xmlparser.ParseFile(file) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -228,6 +228,17 @@ finally: test_support.unlink(test_support.TESTFN) + def test_parse_again(self): + parser = expat.ParserCreate() + file = StringIO.StringIO(data) + parser.ParseFile(file) + # Issue 6676: ensure a meaningful exception is raised when attempting + # to parse more than one XML document per xmlparser instance, + # a limitation of the Expat library. + with self.assertRaises(expat.error) as cm: + parser.ParseFile(file) + self.assertEqual(expat.ErrorString(cm.exception.code), + expat.errors.XML_ERROR_FINISHED) class NamespaceSeparatorTest(unittest.TestCase): def test_legal(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -246,6 +246,11 @@ - Issue #19286: Directories in ``package_data`` are no longer added to the filelist, preventing failure outlined in the ticket. +- Issue #6676: Ensure a meaningful exception is raised when attempting + to parse more than one XML document per pyexpat xmlparser instance. + (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with + suggested wording by David Gutteridge) + Tools/Demos ----------- diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -976,7 +976,7 @@ void *buf = XML_GetBuffer(self->itself, BUF_SIZE); if (buf == NULL) { Py_XDECREF(readmethod); - return PyErr_NoMemory(); + return get_parse_result(self, 0); } bytes_read = readinst(buf, BUF_SIZE, readmethod); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 00:44:38 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 00:44:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzY2NzY6?= =?utf-8?q?_Ensure_a_meaningful_exception_is_raised_when_attempting?= Message-ID: <3fw2qB6JNpz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/9e3fc66ee0b8 changeset: 90003:9e3fc66ee0b8 branch: 3.4 parent: 90001:9f06cbb7962b user: Ned Deily date: Thu Mar 27 16:39:58 2014 -0700 summary: Issue #6676: Ensure a meaningful exception is raised when attempting to parse more than one XML document per pyexpat xmlparser instance. (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with suggested wording by David Gutteridge) files: Doc/library/pyexpat.rst | 9 ++++++++- Lib/test/test_pyexpat.py | 12 ++++++++++++ Misc/NEWS | 5 +++++ Modules/pyexpat.c | 2 +- 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -100,6 +100,11 @@ http://www.python.org/ns/ elem1 elem2 + Due to limitations in the ``Expat`` library used by :mod:`pyexpat`, + the :class:`xmlparser` instance returned can only be used to parse a single + XML document. Call ``ParserCreate`` for each document to provide unique + parser instances. + .. seealso:: @@ -119,7 +124,9 @@ Parses the contents of the string *data*, calling the appropriate handler functions to process the parsed data. *isfinal* must be true on the final call - to this method. *data* can be the empty string at any time. + to this method; it allows the parsing of a single file in fragments, + not the submission of multiple files. + *data* can be the empty string at any time. .. method:: xmlparser.ParseFile(file) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -236,6 +236,18 @@ operations = out.out self._verify_parse_output(operations) + def test_parse_again(self): + parser = expat.ParserCreate() + file = BytesIO(data) + parser.ParseFile(file) + # Issue 6676: ensure a meaningful exception is raised when attempting + # to parse more than one XML document per xmlparser instance, + # a limitation of the Expat library. + with self.assertRaises(expat.error) as cm: + parser.ParseFile(file) + self.assertEqual(expat.ErrorString(cm.exception.code), + expat.errors.XML_ERROR_FINISHED) + class NamespaceSeparatorTest(unittest.TestCase): def test_legal(self): # Tests that make sure we get errors when the namespace_separator value diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -89,6 +89,11 @@ - Issue #20817: Fix inspect.getcallargs() to fail correctly if more than 3 arguments are missing. Patch by Jeremiah Lowin. +- Issue #6676: Ensure a meaningful exception is raised when attempting + to parse more than one XML document per pyexpat xmlparser instance. + (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with + suggested wording by David Gutteridge) + Documentation ------------- diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -908,7 +908,7 @@ void *buf = XML_GetBuffer(self->itself, BUF_SIZE); if (buf == NULL) { Py_XDECREF(readmethod); - return PyErr_NoMemory(); + return get_parse_result(self, 0); } bytes_read = readinst(buf, BUF_SIZE, readmethod); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 00:44:40 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 00:44:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=236676=3A_merge_from_3=2E4?= Message-ID: <3fw2qD13nGz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/ee0034434e65 changeset: 90004:ee0034434e65 parent: 90000:35302cc4fc93 parent: 90003:9e3fc66ee0b8 user: Ned Deily date: Thu Mar 27 16:44:06 2014 -0700 summary: Issue #6676: merge from 3.4 files: Doc/library/pyexpat.rst | 9 ++++++++- Lib/test/test_pyexpat.py | 12 ++++++++++++ Misc/NEWS | 5 +++++ Modules/pyexpat.c | 2 +- 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -100,6 +100,11 @@ http://www.python.org/ns/ elem1 elem2 + Due to limitations in the ``Expat`` library used by :mod:`pyexpat`, + the :class:`xmlparser` instance returned can only be used to parse a single + XML document. Call ``ParserCreate`` for each document to provide unique + parser instances. + .. seealso:: @@ -119,7 +124,9 @@ Parses the contents of the string *data*, calling the appropriate handler functions to process the parsed data. *isfinal* must be true on the final call - to this method. *data* can be the empty string at any time. + to this method; it allows the parsing of a single file in fragments, + not the submission of multiple files. + *data* can be the empty string at any time. .. method:: xmlparser.ParseFile(file) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -236,6 +236,18 @@ operations = out.out self._verify_parse_output(operations) + def test_parse_again(self): + parser = expat.ParserCreate() + file = BytesIO(data) + parser.ParseFile(file) + # Issue 6676: ensure a meaningful exception is raised when attempting + # to parse more than one XML document per xmlparser instance, + # a limitation of the Expat library. + with self.assertRaises(expat.error) as cm: + parser.ParseFile(file) + self.assertEqual(expat.ErrorString(cm.exception.code), + expat.errors.XML_ERROR_FINISHED) + class NamespaceSeparatorTest(unittest.TestCase): def test_legal(self): # Tests that make sure we get errors when the namespace_separator value diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -119,6 +119,11 @@ - Issue #20817: Fix inspect.getcallargs() to fail correctly if more than 3 arguments are missing. Patch by Jeremiah Lowin. +- Issue #6676: Ensure a meaningful exception is raised when attempting + to parse more than one XML document per pyexpat xmlparser instance. + (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with + suggested wording by David Gutteridge) + Documentation ------------- diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -908,7 +908,7 @@ void *buf = XML_GetBuffer(self->itself, BUF_SIZE); if (buf == NULL) { Py_XDECREF(readmethod); - return PyErr_NoMemory(); + return get_parse_result(self, 0); } bytes_read = readinst(buf, BUF_SIZE, readmethod); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 02:12:40 2014 From: python-checkins at python.org (giampaolo.rodola) Date: Fri, 28 Mar 2014 02:12:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_add_myself_to_some_=27exp?= =?utf-8?q?erts=27?= Message-ID: <3fw4mm3SW4z7LjY@mail.python.org> http://hg.python.org/devguide/rev/02cfd563e710 changeset: 684:02cfd563e710 user: Giampaolo Rodola' date: Fri Mar 28 02:11:30 2014 +0100 summary: add myself to some 'experts' files: experts.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -55,9 +55,9 @@ argparse bethard array ast benjamin.peterson -asynchat josiahcarlson, giampaolo.rodola, stutzbach -asyncio gvanrossum, haypo, pitrou, yselivanov -asyncore josiahcarlson, giampaolo.rodola, stutzbach +asynchat josiahcarlson, giampaolo.rodola*, stutzbach +asyncio gvanrossum, haypo, pitrou, yselivanov, giampaolo.rodola +asyncore josiahcarlson, giampaolo.rodola*, stutzbach atexit audioop base64 @@ -111,7 +111,7 @@ formatter fpectl fractions mark.dickinson, rhettinger -ftplib giampaolo.rodola +ftplib giampaolo.rodola* functools ncoghlan, rhettinger gc pitrou getopt @@ -191,7 +191,7 @@ runpy ncoghlan sched giampaolo.rodola select -selectors +selectors neologix, giampaolo.rodola shelve shlex shutil tarek, hynek -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Fri Mar 28 04:52:04 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 04:52:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3NjU0?= =?utf-8?q?=3A_Ensure_IDLE_menus_are_customized_properly_on_OS_X_for?= Message-ID: <3fw8Jh3Q7vz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/f551740c26b6 changeset: 90005:f551740c26b6 branch: 2.7 parent: 90002:74faca1ac59c user: Ned Deily date: Thu Mar 27 20:47:04 2014 -0700 summary: Issue #17654: Ensure IDLE menus are customized properly on OS X for non-framework builds and for all variants of Tk. files: Lib/idlelib/Bindings.py | 32 +--- Lib/idlelib/Debugger.py | 2 +- Lib/idlelib/EditorWindow.py | 14 +- Lib/idlelib/MultiCall.py | 3 +- Lib/idlelib/PyShell.py | 4 +- Lib/idlelib/ScriptBinding.py | 4 +- Lib/idlelib/ZoomHeight.py | 2 +- Lib/idlelib/configDialog.py | 2 +- Lib/idlelib/configHandler.py | 12 +- Lib/idlelib/keybindingDialog.py | 4 +- Lib/idlelib/macosxSupport.py | 126 ++++++++++++++----- Misc/NEWS | 3 + 12 files changed, 127 insertions(+), 81 deletions(-) diff --git a/Lib/idlelib/Bindings.py b/Lib/idlelib/Bindings.py --- a/Lib/idlelib/Bindings.py +++ b/Lib/idlelib/Bindings.py @@ -8,9 +8,14 @@ windows. """ -import sys from idlelib.configHandler import idleConf -from idlelib import macosxSupport + +# Warning: menudefs is altered in macosxSupport.overrideRootMenu() +# after it is determined that an OS X Aqua Tk is in use, +# which cannot be done until after Tk() is first called. +# Do not alter the 'file', 'options', or 'help' cascades here +# without altering overrideRootMenu() as well. +# TODO: Make this more robust menudefs = [ # underscore prefixes character to underscore @@ -81,27 +86,4 @@ ]), ] -if macosxSupport.runningAsOSXApp(): - # Running as a proper MacOS application bundle. This block restructures - # the menus a little to make them conform better to the HIG. - - quitItem = menudefs[0][1][-1] - closeItem = menudefs[0][1][-2] - - # Remove the last 3 items of the file menu: a separator, close window and - # quit. Close window will be reinserted just above the save item, where - # it should be according to the HIG. Quit is in the application menu. - del menudefs[0][1][-3:] - menudefs[0][1].insert(6, closeItem) - - # Remove the 'About' entry from the help menu, it is in the application - # menu - del menudefs[-1][1][0:2] - - # Remove the 'Configure' entry from the options menu, it is in the - # application menu as 'Preferences' - del menudefs[-2][1][0:2] - default_keydefs = idleConf.GetCurrentKeySet() - -del sys diff --git a/Lib/idlelib/Debugger.py b/Lib/idlelib/Debugger.py --- a/Lib/idlelib/Debugger.py +++ b/Lib/idlelib/Debugger.py @@ -323,7 +323,7 @@ class StackViewer(ScrolledList): def __init__(self, master, flist, gui): - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # At least on with the stock AquaTk version on OSX 10.4 you'll # get an shaking GUI that eventually kills IDLE if the width # argument is specified. diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -138,8 +138,8 @@ 'Python%s.chm' % _sphinx_version()) if os.path.isfile(chmfile): dochome = chmfile - elif macosxSupport.runningAsOSXApp(): - # documentation is stored inside the python framework + elif sys.platform == 'darwin': + # documentation may be stored inside a python framework dochome = os.path.join(sys.prefix, 'Resources/English.lproj/Documentation/index.html') dochome = os.path.normpath(dochome) @@ -193,7 +193,7 @@ self.top.protocol("WM_DELETE_WINDOW", self.close) self.top.bind("<>", self.close_event) - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Command-W on editorwindows doesn't work without this. text.bind('<>', self.close_event) # Some OS X systems have only one mouse button, @@ -440,7 +440,7 @@ def set_status_bar(self): self.status_bar = self.MultiStatusBar(self.top) - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": # Insert some padding to avoid obscuring some of the statusbar # by the resize widget. self.status_bar.set_label('_padding1', ' ', side=RIGHT) @@ -467,7 +467,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -479,7 +479,7 @@ menudict[name] = menu = Menu(mbar, name=name) mbar.add_cascade(label=label, menu=menu, underline=underline) - if macosxSupport.isCarbonAquaTk(self.root): + if macosxSupport.isCarbonTk(): # Insert the application menu menudict['application'] = menu = Menu(mbar, name='apple') mbar.add_cascade(label='IDLE', menu=menu) @@ -1682,7 +1682,7 @@ keylist = keydefs.get(eventname) # issue10940: temporary workaround to prevent hang with OS X Cocoa Tk 8.5 # if not keylist: - if (not keylist) or (macosxSupport.runningAsOSXApp() and eventname in { + if (not keylist) or (macosxSupport.isCocoaTk() and eventname in { "<>", "<>", "<>"}): diff --git a/Lib/idlelib/MultiCall.py b/Lib/idlelib/MultiCall.py --- a/Lib/idlelib/MultiCall.py +++ b/Lib/idlelib/MultiCall.py @@ -33,7 +33,6 @@ import string import re import Tkinter -from idlelib import macosxSupport # the event type constants, which define the meaning of mc_type MC_KEYPRESS=0; MC_KEYRELEASE=1; MC_BUTTONPRESS=2; MC_BUTTONRELEASE=3; @@ -46,7 +45,7 @@ MC_OPTION = 1<<6; MC_COMMAND = 1<<7 # define the list of modifiers, to be used in complex event types. -if macosxSupport.runningAsOSXApp(): +if sys.platform == "darwin": _modifiers = (("Shift",), ("Control",), ("Option",), ("Command",)) _modifier_masks = (MC_SHIFT, MC_CONTROL, MC_OPTION, MC_COMMAND) else: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -866,7 +866,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -1566,7 +1566,7 @@ shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: + if macosxSupport.isAquaTk() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py --- a/Lib/idlelib/ScriptBinding.py +++ b/Lib/idlelib/ScriptBinding.py @@ -54,7 +54,7 @@ self.flist = self.editwin.flist self.root = self.editwin.root - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): self.editwin.text_frame.bind('<>', self._run_module_event) def check_module_event(self, event): @@ -168,7 +168,7 @@ interp.runcode(code) return 'break' - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): # Tk-Cocoa in MacOSX is broken until at least # Tk 8.5.9, and without this rather # crude workaround IDLE would hang when a user diff --git a/Lib/idlelib/ZoomHeight.py b/Lib/idlelib/ZoomHeight.py --- a/Lib/idlelib/ZoomHeight.py +++ b/Lib/idlelib/ZoomHeight.py @@ -32,7 +32,7 @@ newy = 0 newheight = newheight - 72 - elif macosxSupport.runningAsOSXApp(): + elif macosxSupport.isAquaTk(): # The '88' below is a magic number that avoids placing the bottom # of the window below the panel on my machine. I don't know how # to calculate the correct value for this with tkinter. diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -71,7 +71,7 @@ page_names=['Fonts/Tabs','Highlighting','Keys','General']) frameActionButtons = Frame(self,pady=2) #action buttons - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Changing the default padding on OSX results in unreadable # text in the buttons paddingArgs={} diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -20,7 +20,6 @@ import os import sys import string -from idlelib import macosxSupport from ConfigParser import ConfigParser, NoOptionError, NoSectionError class InvalidConfigType(Exception): pass @@ -526,10 +525,13 @@ def GetCurrentKeySet(self): result = self.GetKeySet(self.CurrentKeys()) - if macosxSupport.runningAsOSXApp(): - # We're using AquaTk, replace all keybingings that use the - # Alt key by ones that use the Option key because the former - # don't work reliably. + if sys.platform == "darwin": + # OS X Tk variants do not support the "Alt" keyboard modifier. + # So replace all keybingings that use "Alt" with ones that + # use the "Option" keyboard modifier. + # TO DO: the "Option" modifier does not work properly for + # Cocoa Tk and XQuartz Tk so we should not use it + # in default OS X KeySets. for k, v in result.items(): v2 = [ x.replace('>'), ) - else: - # assume Cocoa AquaTk + if isCocoaTk(): # replace default About dialog with About IDLE one root.createcommand('tkAboutDialog', about_dialog) # replace default "Help" item in Help menu @@ -166,10 +214,22 @@ def setupApp(root, flist): """ - Perform setup for the OSX application bundle. + Perform initial OS X customizations if needed. + Called from PyShell.main() after initial calls to Tk() + + There are currently three major versions of Tk in use on OS X: + 1. Aqua Cocoa Tk (native default since OS X 10.6) + 2. Aqua Carbon Tk (original native, 32-bit only, deprecated) + 3. X11 (supported by some third-party distributors, deprecated) + There are various differences among the three that affect IDLE + behavior, primarily with menus, mouse key events, and accelerators. + Some one-time customizations are performed here. + Others are dynamically tested throughout idlelib by calls to the + isAquaTk(), isCarbonTk(), isCocoaTk(), isXQuartz() functions which + are initialized here as well. """ - if not runningAsOSXApp(): return - - hideTkConsole(root) - overrideRootMenu(root, flist) - addOpenEventSupport(root, flist) + _initializeTkVariantTests(root) + if isAquaTk(): + hideTkConsole(root) + overrideRootMenu(root, flist) + addOpenEventSupport(root, flist) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -279,6 +279,9 @@ - Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial shell window is present. +- Issue #17654: Ensure IDLE menus are customized properly on OS X for + non-framework builds and for all variants of Tk. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 04:52:06 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 04:52:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE3NjU0?= =?utf-8?q?=3A_Ensure_IDLE_menus_are_customized_properly_on_OS_X_for?= Message-ID: <3fw8Jk0blTz7LlT@mail.python.org> http://hg.python.org/cpython/rev/67a7a49e7b78 changeset: 90006:67a7a49e7b78 branch: 3.4 parent: 90003:9e3fc66ee0b8 user: Ned Deily date: Thu Mar 27 20:49:14 2014 -0700 summary: Issue #17654: Ensure IDLE menus are customized properly on OS X for non-framework builds and for all variants of Tk. files: Lib/idlelib/Bindings.py | 32 +--- Lib/idlelib/Debugger.py | 2 +- Lib/idlelib/EditorWindow.py | 14 +- Lib/idlelib/MultiCall.py | 3 +- Lib/idlelib/PyShell.py | 4 +- Lib/idlelib/ScriptBinding.py | 4 +- Lib/idlelib/ZoomHeight.py | 2 +- Lib/idlelib/configDialog.py | 2 +- Lib/idlelib/configHandler.py | 12 +- Lib/idlelib/keybindingDialog.py | 5 +- Lib/idlelib/macosxSupport.py | 138 +++++++++++++------ Misc/NEWS | 6 + 12 files changed, 130 insertions(+), 94 deletions(-) diff --git a/Lib/idlelib/Bindings.py b/Lib/idlelib/Bindings.py --- a/Lib/idlelib/Bindings.py +++ b/Lib/idlelib/Bindings.py @@ -8,9 +8,14 @@ windows. """ -import sys from idlelib.configHandler import idleConf -from idlelib import macosxSupport + +# Warning: menudefs is altered in macosxSupport.overrideRootMenu() +# after it is determined that an OS X Aqua Tk is in use, +# which cannot be done until after Tk() is first called. +# Do not alter the 'file', 'options', or 'help' cascades here +# without altering overrideRootMenu() as well. +# TODO: Make this more robust menudefs = [ # underscore prefixes character to underscore @@ -81,27 +86,4 @@ ]), ] -if macosxSupport.runningAsOSXApp(): - # Running as a proper MacOS application bundle. This block restructures - # the menus a little to make them conform better to the HIG. - - quitItem = menudefs[0][1][-1] - closeItem = menudefs[0][1][-2] - - # Remove the last 3 items of the file menu: a separator, close window and - # quit. Close window will be reinserted just above the save item, where - # it should be according to the HIG. Quit is in the application menu. - del menudefs[0][1][-3:] - menudefs[0][1].insert(6, closeItem) - - # Remove the 'About' entry from the help menu, it is in the application - # menu - del menudefs[-1][1][0:2] - - # Remove the 'Configure' entry from the options menu, it is in the - # application menu as 'Preferences' - del menudefs[-2][1][0:2] - default_keydefs = idleConf.GetCurrentKeySet() - -del sys diff --git a/Lib/idlelib/Debugger.py b/Lib/idlelib/Debugger.py --- a/Lib/idlelib/Debugger.py +++ b/Lib/idlelib/Debugger.py @@ -322,7 +322,7 @@ class StackViewer(ScrolledList): def __init__(self, master, flist, gui): - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # At least on with the stock AquaTk version on OSX 10.4 you'll # get an shaking GUI that eventually kills IDLE if the width # argument is specified. diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -109,8 +109,8 @@ 'Python%s.chm' % _sphinx_version()) if os.path.isfile(chmfile): dochome = chmfile - elif macosxSupport.runningAsOSXApp(): - # documentation is stored inside the python framework + elif sys.platform == 'darwin': + # documentation may be stored inside a python framework dochome = os.path.join(sys.base_prefix, 'Resources/English.lproj/Documentation/index.html') dochome = os.path.normpath(dochome) @@ -166,7 +166,7 @@ self.top.protocol("WM_DELETE_WINDOW", self.close) self.top.bind("<>", self.close_event) - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Command-W on editorwindows doesn't work without this. text.bind('<>', self.close_event) # Some OS X systems have only one mouse button, @@ -409,7 +409,7 @@ def set_status_bar(self): self.status_bar = self.MultiStatusBar(self.top) - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": # Insert some padding to avoid obscuring some of the statusbar # by the resize widget. self.status_bar.set_label('_padding1', ' ', side=RIGHT) @@ -436,7 +436,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -447,7 +447,7 @@ underline, label = prepstr(label) menudict[name] = menu = Menu(mbar, name=name) mbar.add_cascade(label=label, menu=menu, underline=underline) - if macosxSupport.isCarbonAquaTk(self.root): + if macosxSupport.isCarbonTk(): # Insert the application menu menudict['application'] = menu = Menu(mbar, name='apple') mbar.add_cascade(label='IDLE', menu=menu) @@ -1673,7 +1673,7 @@ keylist = keydefs.get(eventname) # issue10940: temporary workaround to prevent hang with OS X Cocoa Tk 8.5 # if not keylist: - if (not keylist) or (macosxSupport.runningAsOSXApp() and eventname in { + if (not keylist) or (macosxSupport.isCocoaTk() and eventname in { "<>", "<>", "<>"}): diff --git a/Lib/idlelib/MultiCall.py b/Lib/idlelib/MultiCall.py --- a/Lib/idlelib/MultiCall.py +++ b/Lib/idlelib/MultiCall.py @@ -32,7 +32,6 @@ import sys import re import tkinter -from idlelib import macosxSupport # the event type constants, which define the meaning of mc_type MC_KEYPRESS=0; MC_KEYRELEASE=1; MC_BUTTONPRESS=2; MC_BUTTONRELEASE=3; @@ -45,7 +44,7 @@ MC_OPTION = 1<<6; MC_COMMAND = 1<<7 # define the list of modifiers, to be used in complex event types. -if macosxSupport.runningAsOSXApp(): +if sys.platform == "darwin": _modifiers = (("Shift",), ("Control",), ("Option",), ("Command",)) _modifier_masks = (MC_SHIFT, MC_CONTROL, MC_OPTION, MC_COMMAND) else: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -844,7 +844,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -1560,7 +1560,7 @@ shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: + if macosxSupport.isAquaTk() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py --- a/Lib/idlelib/ScriptBinding.py +++ b/Lib/idlelib/ScriptBinding.py @@ -53,7 +53,7 @@ self.flist = self.editwin.flist self.root = self.editwin.root - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): self.editwin.text_frame.bind('<>', self._run_module_event) def check_module_event(self, event): @@ -114,7 +114,7 @@ shell.set_warning_stream(saved_stream) def run_module_event(self, event): - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): # Tk-Cocoa in MacOSX is broken until at least # Tk 8.5.9, and without this rather # crude workaround IDLE would hang when a user diff --git a/Lib/idlelib/ZoomHeight.py b/Lib/idlelib/ZoomHeight.py --- a/Lib/idlelib/ZoomHeight.py +++ b/Lib/idlelib/ZoomHeight.py @@ -32,7 +32,7 @@ newy = 0 newheight = newheight - 72 - elif macosxSupport.runningAsOSXApp(): + elif macosxSupport.isAquaTk(): # The '88' below is a magic number that avoids placing the bottom # of the window below the panel on my machine. I don't know how # to calculate the correct value for this with tkinter. diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -74,7 +74,7 @@ frameActionButtons = Frame(self,pady=2) #action buttons - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Surpress the padx and pady arguments when # running as IDLE.app, otherwise the text # on these buttons will not be readable. diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -20,7 +20,6 @@ import os import sys -from idlelib import macosxSupport from configparser import ConfigParser, NoOptionError, NoSectionError class InvalidConfigType(Exception): pass @@ -527,10 +526,13 @@ def GetCurrentKeySet(self): result = self.GetKeySet(self.CurrentKeys()) - if macosxSupport.runningAsOSXApp(): - # We're using AquaTk, replace all keybingings that use the - # Alt key by ones that use the Option key because the former - # don't work reliably. + if sys.platform == "darwin": + # OS X Tk variants do not support the "Alt" keyboard modifier. + # So replace all keybingings that use "Alt" with ones that + # use the "Option" keyboard modifier. + # TO DO: the "Option" modifier does not work properly for + # Cocoa Tk and XQuartz Tk so we should not use it + # in default OS X KeySets. for k, v in result.items(): v2 = [ x.replace('>'), ) - else: - # assume Cocoa AquaTk + if isCocoaTk(): # replace default About dialog with About IDLE one root.createcommand('tkAboutDialog', about_dialog) # replace default "Help" item in Help menu @@ -182,10 +218,22 @@ def setupApp(root, flist): """ - Perform setup for the OSX application bundle. + Perform initial OS X customizations if needed. + Called from PyShell.main() after initial calls to Tk() + + There are currently three major versions of Tk in use on OS X: + 1. Aqua Cocoa Tk (native default since OS X 10.6) + 2. Aqua Carbon Tk (original native, 32-bit only, deprecated) + 3. X11 (supported by some third-party distributors, deprecated) + There are various differences among the three that affect IDLE + behavior, primarily with menus, mouse key events, and accelerators. + Some one-time customizations are performed here. + Others are dynamically tested throughout idlelib by calls to the + isAquaTk(), isCarbonTk(), isCocoaTk(), isXQuartz() functions which + are initialized here as well. """ - if not runningAsOSXApp(): return - - hideTkConsole(root) - overrideRootMenu(root, flist) - addOpenEventSupport(root, flist) + _initializeTkVariantTests(root) + if isAquaTk(): + hideTkConsole(root) + overrideRootMenu(root, flist) + addOpenEventSupport(root, flist) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -94,6 +94,12 @@ (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with suggested wording by David Gutteridge) +IDLE +---- + +- Issue #17654: Ensure IDLE menus are customized properly on OS X for + non-framework builds and for all variants of Tk. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 04:52:07 2014 From: python-checkins at python.org (ned.deily) Date: Fri, 28 Mar 2014 04:52:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317654=3A_merge_from_3=2E4?= Message-ID: <3fw8Jl4mMNz7Llg@mail.python.org> http://hg.python.org/cpython/rev/d8659dbebfd1 changeset: 90007:d8659dbebfd1 parent: 90004:ee0034434e65 parent: 90006:67a7a49e7b78 user: Ned Deily date: Thu Mar 27 20:51:15 2014 -0700 summary: Issue #17654: merge from 3.4 files: Lib/idlelib/Bindings.py | 32 +--- Lib/idlelib/Debugger.py | 2 +- Lib/idlelib/EditorWindow.py | 14 +- Lib/idlelib/MultiCall.py | 3 +- Lib/idlelib/PyShell.py | 4 +- Lib/idlelib/ScriptBinding.py | 4 +- Lib/idlelib/ZoomHeight.py | 2 +- Lib/idlelib/configDialog.py | 2 +- Lib/idlelib/configHandler.py | 12 +- Lib/idlelib/keybindingDialog.py | 5 +- Lib/idlelib/macosxSupport.py | 138 +++++++++++++------ Misc/NEWS | 6 + 12 files changed, 130 insertions(+), 94 deletions(-) diff --git a/Lib/idlelib/Bindings.py b/Lib/idlelib/Bindings.py --- a/Lib/idlelib/Bindings.py +++ b/Lib/idlelib/Bindings.py @@ -8,9 +8,14 @@ windows. """ -import sys from idlelib.configHandler import idleConf -from idlelib import macosxSupport + +# Warning: menudefs is altered in macosxSupport.overrideRootMenu() +# after it is determined that an OS X Aqua Tk is in use, +# which cannot be done until after Tk() is first called. +# Do not alter the 'file', 'options', or 'help' cascades here +# without altering overrideRootMenu() as well. +# TODO: Make this more robust menudefs = [ # underscore prefixes character to underscore @@ -81,27 +86,4 @@ ]), ] -if macosxSupport.runningAsOSXApp(): - # Running as a proper MacOS application bundle. This block restructures - # the menus a little to make them conform better to the HIG. - - quitItem = menudefs[0][1][-1] - closeItem = menudefs[0][1][-2] - - # Remove the last 3 items of the file menu: a separator, close window and - # quit. Close window will be reinserted just above the save item, where - # it should be according to the HIG. Quit is in the application menu. - del menudefs[0][1][-3:] - menudefs[0][1].insert(6, closeItem) - - # Remove the 'About' entry from the help menu, it is in the application - # menu - del menudefs[-1][1][0:2] - - # Remove the 'Configure' entry from the options menu, it is in the - # application menu as 'Preferences' - del menudefs[-2][1][0:2] - default_keydefs = idleConf.GetCurrentKeySet() - -del sys diff --git a/Lib/idlelib/Debugger.py b/Lib/idlelib/Debugger.py --- a/Lib/idlelib/Debugger.py +++ b/Lib/idlelib/Debugger.py @@ -322,7 +322,7 @@ class StackViewer(ScrolledList): def __init__(self, master, flist, gui): - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # At least on with the stock AquaTk version on OSX 10.4 you'll # get an shaking GUI that eventually kills IDLE if the width # argument is specified. diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -109,8 +109,8 @@ 'Python%s.chm' % _sphinx_version()) if os.path.isfile(chmfile): dochome = chmfile - elif macosxSupport.runningAsOSXApp(): - # documentation is stored inside the python framework + elif sys.platform == 'darwin': + # documentation may be stored inside a python framework dochome = os.path.join(sys.base_prefix, 'Resources/English.lproj/Documentation/index.html') dochome = os.path.normpath(dochome) @@ -166,7 +166,7 @@ self.top.protocol("WM_DELETE_WINDOW", self.close) self.top.bind("<>", self.close_event) - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Command-W on editorwindows doesn't work without this. text.bind('<>', self.close_event) # Some OS X systems have only one mouse button, @@ -409,7 +409,7 @@ def set_status_bar(self): self.status_bar = self.MultiStatusBar(self.top) - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": # Insert some padding to avoid obscuring some of the statusbar # by the resize widget. self.status_bar.set_label('_padding1', ' ', side=RIGHT) @@ -436,7 +436,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -447,7 +447,7 @@ underline, label = prepstr(label) menudict[name] = menu = Menu(mbar, name=name) mbar.add_cascade(label=label, menu=menu, underline=underline) - if macosxSupport.isCarbonAquaTk(self.root): + if macosxSupport.isCarbonTk(): # Insert the application menu menudict['application'] = menu = Menu(mbar, name='apple') mbar.add_cascade(label='IDLE', menu=menu) @@ -1673,7 +1673,7 @@ keylist = keydefs.get(eventname) # issue10940: temporary workaround to prevent hang with OS X Cocoa Tk 8.5 # if not keylist: - if (not keylist) or (macosxSupport.runningAsOSXApp() and eventname in { + if (not keylist) or (macosxSupport.isCocoaTk() and eventname in { "<>", "<>", "<>"}): diff --git a/Lib/idlelib/MultiCall.py b/Lib/idlelib/MultiCall.py --- a/Lib/idlelib/MultiCall.py +++ b/Lib/idlelib/MultiCall.py @@ -32,7 +32,6 @@ import sys import re import tkinter -from idlelib import macosxSupport # the event type constants, which define the meaning of mc_type MC_KEYPRESS=0; MC_KEYRELEASE=1; MC_BUTTONPRESS=2; MC_BUTTONRELEASE=3; @@ -45,7 +44,7 @@ MC_OPTION = 1<<6; MC_COMMAND = 1<<7 # define the list of modifiers, to be used in complex event types. -if macosxSupport.runningAsOSXApp(): +if sys.platform == "darwin": _modifiers = (("Shift",), ("Control",), ("Option",), ("Command",)) _modifier_masks = (MC_SHIFT, MC_CONTROL, MC_OPTION, MC_COMMAND) else: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -844,7 +844,7 @@ ("help", "_Help"), ] - if macosxSupport.runningAsOSXApp(): + if sys.platform == "darwin": menu_specs[-2] = ("windows", "_Window") @@ -1560,7 +1560,7 @@ shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: + if macosxSupport.isAquaTk() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py --- a/Lib/idlelib/ScriptBinding.py +++ b/Lib/idlelib/ScriptBinding.py @@ -53,7 +53,7 @@ self.flist = self.editwin.flist self.root = self.editwin.root - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): self.editwin.text_frame.bind('<>', self._run_module_event) def check_module_event(self, event): @@ -114,7 +114,7 @@ shell.set_warning_stream(saved_stream) def run_module_event(self, event): - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isCocoaTk(): # Tk-Cocoa in MacOSX is broken until at least # Tk 8.5.9, and without this rather # crude workaround IDLE would hang when a user diff --git a/Lib/idlelib/ZoomHeight.py b/Lib/idlelib/ZoomHeight.py --- a/Lib/idlelib/ZoomHeight.py +++ b/Lib/idlelib/ZoomHeight.py @@ -32,7 +32,7 @@ newy = 0 newheight = newheight - 72 - elif macosxSupport.runningAsOSXApp(): + elif macosxSupport.isAquaTk(): # The '88' below is a magic number that avoids placing the bottom # of the window below the panel on my machine. I don't know how # to calculate the correct value for this with tkinter. diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -74,7 +74,7 @@ frameActionButtons = Frame(self,pady=2) #action buttons - if macosxSupport.runningAsOSXApp(): + if macosxSupport.isAquaTk(): # Surpress the padx and pady arguments when # running as IDLE.app, otherwise the text # on these buttons will not be readable. diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -20,7 +20,6 @@ import os import sys -from idlelib import macosxSupport from configparser import ConfigParser, NoOptionError, NoSectionError class InvalidConfigType(Exception): pass @@ -527,10 +526,13 @@ def GetCurrentKeySet(self): result = self.GetKeySet(self.CurrentKeys()) - if macosxSupport.runningAsOSXApp(): - # We're using AquaTk, replace all keybingings that use the - # Alt key by ones that use the Option key because the former - # don't work reliably. + if sys.platform == "darwin": + # OS X Tk variants do not support the "Alt" keyboard modifier. + # So replace all keybingings that use "Alt" with ones that + # use the "Option" keyboard modifier. + # TO DO: the "Option" modifier does not work properly for + # Cocoa Tk and XQuartz Tk so we should not use it + # in default OS X KeySets. for k, v in result.items(): v2 = [ x.replace('>'), ) - else: - # assume Cocoa AquaTk + if isCocoaTk(): # replace default About dialog with About IDLE one root.createcommand('tkAboutDialog', about_dialog) # replace default "Help" item in Help menu @@ -182,10 +218,22 @@ def setupApp(root, flist): """ - Perform setup for the OSX application bundle. + Perform initial OS X customizations if needed. + Called from PyShell.main() after initial calls to Tk() + + There are currently three major versions of Tk in use on OS X: + 1. Aqua Cocoa Tk (native default since OS X 10.6) + 2. Aqua Carbon Tk (original native, 32-bit only, deprecated) + 3. X11 (supported by some third-party distributors, deprecated) + There are various differences among the three that affect IDLE + behavior, primarily with menus, mouse key events, and accelerators. + Some one-time customizations are performed here. + Others are dynamically tested throughout idlelib by calls to the + isAquaTk(), isCarbonTk(), isCocoaTk(), isXQuartz() functions which + are initialized here as well. """ - if not runningAsOSXApp(): return - - hideTkConsole(root) - overrideRootMenu(root, flist) - addOpenEventSupport(root, flist) + _initializeTkVariantTests(root) + if isAquaTk(): + hideTkConsole(root) + overrideRootMenu(root, flist) + addOpenEventSupport(root, flist) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -124,6 +124,12 @@ (Original patches by Hirokazu Yamamoto and Amaury Forgeot d'Arc, with suggested wording by David Gutteridge) +IDLE +---- + +- Issue #17654: Ensure IDLE menus are customized properly on OS X for + non-framework builds and for all variants of Tk. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Mar 28 09:40:02 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 28 Mar 2014 09:40:02 +0100 Subject: [Python-checkins] Daily reference leaks (ee0034434e65): sum=4 Message-ID: results for ee0034434e65 on branch "default" -------------------------------------------- test_site leaked [2, -2, 2] references, sum=2 test_site leaked [2, -2, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog47pW9p', '-x'] From python-checkins at python.org Fri Mar 28 16:41:31 2014 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 28 Mar 2014 16:41:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_tlsv1_-=3E_tlsv1=2E2?= Message-ID: <3fwS3H3vMTz7LjQ@mail.python.org> http://hg.python.org/peps/rev/d641c2c751eb changeset: 5444:d641c2c751eb user: Benjamin Peterson date: Fri Mar 28 11:41:27 2014 -0400 summary: tlsv1 -> tlsv1.2 files: pep-0466.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -67,7 +67,7 @@ * in the ``ssl`` module: * this module is almost entirely synchronised with its Python 3 - counterpart, bringing TLSv2, SSLContext manipulation, Server Name + counterpart, bringing TLSv1.2, SSLContext manipulation, Server Name Identification, access to platform certificate stores, standard library support for peer hostname validation and more to the Python 2 series. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Mar 28 19:49:12 2014 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Mar 2014 19:49:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogRG9jOiBkbyBub3Qg?= =?utf-8?q?compress_=2Eepub_for_dist=2C_it_is_already_zipped=2E?= Message-ID: <3fwXCr6z9Qz7LjN@mail.python.org> http://hg.python.org/cpython/rev/1f4daa805922 changeset: 90008:1f4daa805922 branch: 3.4 parent: 90006:67a7a49e7b78 user: Georg Brandl date: Fri Mar 28 19:48:55 2014 +0100 summary: Doc: do not compress .epub for dist, it is already zipped. files: Doc/Makefile | 10 ++-------- Doc/tools/sphinxext/download.html | 4 ++-- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -136,16 +136,10 @@ cp build/latex/docs-pdf.zip dist/python-$(DISTVERSION)-docs-pdf-letter.zip cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-letter.tar.bz2 - # archive the epub build + # copy the epub build rm -rf build/epub make epub - mkdir -p dist/python-$(DISTVERSION)-docs-epub - cp -pPR build/epub/*.epub dist/python-$(DISTVERSION)-docs-epub/ - tar -C dist -cf dist/python-$(DISTVERSION)-docs-epub.tar python-$(DISTVERSION)-docs-epub - bzip2 -9 -k dist/python-$(DISTVERSION)-docs-epub.tar - (cd dist; zip -q -r -9 python-$(DISTVERSION)-docs-epub.zip python-$(DISTVERSION)-docs-epub) - rm -r dist/python-$(DISTVERSION)-docs-epub - rm dist/python-$(DISTVERSION)-docs-epub.tar + cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub check: $(PYTHON) tools/rstlint.py -i tools diff --git a/Doc/tools/sphinxext/download.html b/Doc/tools/sphinxext/download.html --- a/Doc/tools/sphinxext/download.html +++ b/Doc/tools/sphinxext/download.html @@ -34,8 +34,8 @@ Download (ca. 1.5 MB) EPUB - Download (ca. 3.5 MB) - Download (ca. 3.5 MB) + Download (ca. 4.5 MB) + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 19:49:14 2014 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Mar 2014 19:49:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <3fwXCt1XqGz7LjN@mail.python.org> http://hg.python.org/cpython/rev/19b374217d07 changeset: 90009:19b374217d07 parent: 90007:d8659dbebfd1 parent: 90008:1f4daa805922 user: Georg Brandl date: Fri Mar 28 19:49:02 2014 +0100 summary: merge with 3.4 files: Doc/Makefile | 10 ++-------- Doc/tools/sphinxext/download.html | 4 ++-- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -136,16 +136,10 @@ cp build/latex/docs-pdf.zip dist/python-$(DISTVERSION)-docs-pdf-letter.zip cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-letter.tar.bz2 - # archive the epub build + # copy the epub build rm -rf build/epub make epub - mkdir -p dist/python-$(DISTVERSION)-docs-epub - cp -pPR build/epub/*.epub dist/python-$(DISTVERSION)-docs-epub/ - tar -C dist -cf dist/python-$(DISTVERSION)-docs-epub.tar python-$(DISTVERSION)-docs-epub - bzip2 -9 -k dist/python-$(DISTVERSION)-docs-epub.tar - (cd dist; zip -q -r -9 python-$(DISTVERSION)-docs-epub.zip python-$(DISTVERSION)-docs-epub) - rm -r dist/python-$(DISTVERSION)-docs-epub - rm dist/python-$(DISTVERSION)-docs-epub.tar + cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub check: $(PYTHON) tools/rstlint.py -i tools diff --git a/Doc/tools/sphinxext/download.html b/Doc/tools/sphinxext/download.html --- a/Doc/tools/sphinxext/download.html +++ b/Doc/tools/sphinxext/download.html @@ -34,8 +34,8 @@ Download (ca. 1.5 MB) EPUB - Download (ca. 3.5 MB) - Download (ca. 3.5 MB) + Download (ca. 4.5 MB) + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 20:07:45 2014 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Mar 2014 20:07:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_suspici?= =?utf-8?q?ous_ignore_file=2E?= Message-ID: <3fwXdF3cYvz7LjS@mail.python.org> http://hg.python.org/cpython/rev/3567d7ebd382 changeset: 90010:3567d7ebd382 branch: 2.7 parent: 90005:f551740c26b6 user: Georg Brandl date: Fri Mar 28 20:07:35 2014 +0100 summary: Update suspicious ignore file. files: Doc/tools/sphinxext/susp-ignored.csv | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -43,9 +43,8 @@ howto/logging,,:logger,severity:logger name:message howto/logging,,:message,severity:logger name:message howto/logging,,:This,DEBUG:root:This message should go to the log file -howto/pyporting,75,::,# make sure to use :: Python *and* :: Python :: 3 so -howto/pyporting,75,::,"'Programming Language :: Python'," -howto/pyporting,75,::,'Programming Language :: Python :: 3' +howto/pyporting,,::,Programming Language :: Python :: 2 +howto/pyporting,,::,Programming Language :: Python :: 3 howto/regex,,::, howto/regex,,:foo,(?:foo) howto/urllib2,,:example,"for example ""joe at password:example.com""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 21:15:57 2014 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 28 Mar 2014 21:15:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Mark_PEP_3145_as_withdrawn=2E?= Message-ID: <3fwZ7x3d5kz7Ljh@mail.python.org> http://hg.python.org/peps/rev/8440d642f647 changeset: 5445:8440d642f647 user: Guido van Rossum date: Fri Mar 28 13:15:54 2014 -0700 summary: Mark PEP 3145 as withdrawn. files: pep-3145.txt | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/pep-3145.txt b/pep-3145.txt --- a/pep-3145.txt +++ b/pep-3145.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson -Status: Deferred +Status: Withdrawn Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 @@ -25,6 +25,12 @@ at least until after PEP 3156 has been resolved. +PEP Withdrawal: + + This can be dealt with in the bug tracker. + A specific proposal is attached to http://bugs.python.org/issue18823 + + Motivation: A search for "python asynchronous subprocess" will turn up numerous -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Mar 28 23:55:30 2014 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 28 Mar 2014 23:55:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_undefine_SET?= =?utf-8?q?=5FSYS=5FFROM=5FSTRING=5FBORROW_after_its_done_being_used_=28cl?= =?utf-8?q?oses_=2321089=29?= Message-ID: <3fwdh20vyPz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/c88a7c38eb0d changeset: 90011:c88a7c38eb0d branch: 3.4 parent: 90008:1f4daa805922 user: Benjamin Peterson date: Fri Mar 28 18:52:45 2014 -0400 summary: undefine SET_SYS_FROM_STRING_BORROW after its done being used (closes #21089) files: Python/sysmodule.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1799,6 +1799,7 @@ #endif #undef SET_SYS_FROM_STRING +#undef SET_SYS_FROM_STRING_BORROW if (PyErr_Occurred()) return NULL; return m; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Mar 28 23:55:31 2014 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 28 Mar 2014 23:55:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjEwODkp?= Message-ID: <3fwdh32Hc7z7Ljk@mail.python.org> http://hg.python.org/cpython/rev/c73b71363d4c changeset: 90012:c73b71363d4c parent: 90009:19b374217d07 parent: 90011:c88a7c38eb0d user: Benjamin Peterson date: Fri Mar 28 18:55:04 2014 -0400 summary: merge 3.4 (#21089) files: Python/sysmodule.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1799,6 +1799,7 @@ #endif #undef SET_SYS_FROM_STRING +#undef SET_SYS_FROM_STRING_BORROW if (PyErr_Occurred()) return NULL; return m; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 00:39:58 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 29 Mar 2014 00:39:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgMjEwMTQ6?= =?utf-8?q?__Use_booleans_instead_of_0_and_1_in_examples=2E?= Message-ID: <3fwfgL2mF6z7Ll6@mail.python.org> http://hg.python.org/cpython/rev/deb71529aad1 changeset: 90013:deb71529aad1 branch: 3.4 parent: 90011:c88a7c38eb0d user: Raymond Hettinger date: Fri Mar 28 16:39:25 2014 -0700 summary: Issue 21014: Use booleans instead of 0 and 1 in examples. files: Doc/howto/logging-cookbook.rst | 2 +- Doc/tutorial/interpreter.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -416,7 +416,7 @@ Simple TCP socket-based logging receiver suitable for testing. """ - allow_reuse_address = 1 + allow_reuse_address = True def __init__(self, host='localhost', port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -106,7 +106,7 @@ Continuation lines are needed when entering a multi-line construct. As an example, take a look at this :keyword:`if` statement:: - >>> the_world_is_flat = 1 + >>> the_world_is_flat = True >>> if the_world_is_flat: ... print("Be careful not to fall off!") ... -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 00:39:59 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 29 Mar 2014 00:39:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3fwfgM4CkWz7LlM@mail.python.org> http://hg.python.org/cpython/rev/c5336d30fd99 changeset: 90014:c5336d30fd99 parent: 90012:c73b71363d4c parent: 90013:deb71529aad1 user: Raymond Hettinger date: Fri Mar 28 16:39:45 2014 -0700 summary: merge files: Doc/howto/logging-cookbook.rst | 2 +- Doc/tutorial/interpreter.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -416,7 +416,7 @@ Simple TCP socket-based logging receiver suitable for testing. """ - allow_reuse_address = 1 + allow_reuse_address = True def __init__(self, host='localhost', port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -106,7 +106,7 @@ Continuation lines are needed when entering a multi-line construct. As an example, take a look at this :keyword:`if` statement:: - >>> the_world_is_flat = 1 + >>> the_world_is_flat = True >>> if the_world_is_flat: ... print("Be careful not to fall off!") ... -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 01:11:55 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 01:11:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319697=3A_Document?= =?utf-8?b?IGNhc2VzIHdoZXJlIF9fbWFpbl9fLl9fc3BlY19fIGlzIE5vbmUu?= Message-ID: <3fwgNC0pz5z7Ljk@mail.python.org> http://hg.python.org/cpython/rev/21d8222b667f changeset: 90015:21d8222b667f user: Eric Snow date: Fri Mar 28 18:10:33 2014 -0600 summary: Issue #19697: Document cases where __main__.__spec__ is None. files: Doc/reference/import.rst | 42 +++++++++++++++++++++++++++- Misc/NEWS | 2 + 2 files changed, 43 insertions(+), 1 deletions(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -519,7 +519,10 @@ The ``__spec__`` attribute must be set to the module spec that was used when importing the module. This is used primarily for - introspection and during reloading. + introspection and during reloading. Setting ``__spec__`` + appropriately applies equally to :ref:`modules initialized during + interpreter startup `. The one exception is ``__main__``, + where ``__spec__`` is :ref:`set to None in some cases `. .. versionadded:: 3.4 @@ -829,6 +832,37 @@ while raising an exception terminates it immediately. +Special considerations for __main__ +=================================== + +The :mod:`__main__` module is a special case relative to Python's import +system. As noted :ref:`elsewhere `, the ``__main__`` module +is directly initialized at interpreter startup, much like :mod:`sys` and +:mod:`builtins`. However, unlike those two, it doesn't strictly +qualify as a built-in module. This is because the manner in which +``__main__`` is initialized depends on the flags and other options with +which the interpreter is invoked. + +.. _main_spec: + +__main__.__spec__ +----------------- + +Depending on how :mod:`__main__` is initialized, ``__main__.__spec__`` +gets set appropriately or to ``None``. + +When Python is started with the :option:`-m` option, ``__spec__`` is set +to the module spec of the corresponding module. + +In :ref:`the remaining cases ` +``__main__.__spec__`` is set to ``None``: + +- interactive prompt +- -c switch +- running from stdin +- running directly from a source or bytecode file + + Open issues =========== @@ -841,6 +875,12 @@ XXX runpy, pkgutil, et al in the library manual should all get "See Also" links at the top pointing to the new import system section. +XXX Add more explanation regarding the different ways in which +``__main__`` is initialized? + +XXX Add more info on ``__main__`` quirks/pitfalls (i.e. copy from +:pep:`395`). + References ========== diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -145,6 +145,8 @@ required reference material for tool developers that isn't recorded anywhere else. +- Issue #19697: Document cases where __main__.__spec__ is None. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 01:30:05 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 01:30:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5Njk3?= =?utf-8?b?OiBEb2N1bWVudCBjYXNlcyB3aGVyZSBfX21haW5fXy5fX3NwZWNfXyBpcyBO?= =?utf-8?q?one=2E?= Message-ID: <3fwgn96VCVz7LjN@mail.python.org> http://hg.python.org/cpython/rev/4972475ae2e7 changeset: 90016:4972475ae2e7 branch: 3.4 parent: 90013:deb71529aad1 user: Eric Snow date: Fri Mar 28 18:10:33 2014 -0600 summary: Issue #19697: Document cases where __main__.__spec__ is None. files: Doc/reference/import.rst | 42 +++++++++++++++++++++++++++- Misc/NEWS | 2 + 2 files changed, 43 insertions(+), 1 deletions(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -519,7 +519,10 @@ The ``__spec__`` attribute must be set to the module spec that was used when importing the module. This is used primarily for - introspection and during reloading. + introspection and during reloading. Setting ``__spec__`` + appropriately applies equally to :ref:`modules initialized during + interpreter startup `. The one exception is ``__main__``, + where ``__spec__`` is :ref:`set to None in some cases `. .. versionadded:: 3.4 @@ -829,6 +832,37 @@ while raising an exception terminates it immediately. +Special considerations for __main__ +=================================== + +The :mod:`__main__` module is a special case relative to Python's import +system. As noted :ref:`elsewhere `, the ``__main__`` module +is directly initialized at interpreter startup, much like :mod:`sys` and +:mod:`builtins`. However, unlike those two, it doesn't strictly +qualify as a built-in module. This is because the manner in which +``__main__`` is initialized depends on the flags and other options with +which the interpreter is invoked. + +.. _main_spec: + +__main__.__spec__ +----------------- + +Depending on how :mod:`__main__` is initialized, ``__main__.__spec__`` +gets set appropriately or to ``None``. + +When Python is started with the :option:`-m` option, ``__spec__`` is set +to the module spec of the corresponding module. + +In :ref:`the remaining cases ` +``__main__.__spec__`` is set to ``None``: + +- interactive prompt +- -c switch +- running from stdin +- running directly from a source or bytecode file + + Open issues =========== @@ -841,6 +875,12 @@ XXX runpy, pkgutil, et al in the library manual should all get "See Also" links at the top pointing to the new import system section. +XXX Add more explanation regarding the different ways in which +``__main__`` is initialized? + +XXX Add more info on ``__main__`` quirks/pitfalls (i.e. copy from +:pep:`395`). + References ========== diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -115,6 +115,8 @@ required reference material for tool developers that isn't recorded anywhere else. +- Issue #19697: Document cases where __main__.__spec__ is None. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 01:30:07 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 01:30:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_into_default=2E?= Message-ID: <3fwgnC1Dmfz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/ae5169297eb9 changeset: 90017:ae5169297eb9 parent: 90015:21d8222b667f parent: 90016:4972475ae2e7 user: Eric Snow date: Fri Mar 28 18:26:46 2014 -0600 summary: Merge 3.4 into default. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 05:37:25 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 29 Mar 2014 05:37:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_another_ol?= =?utf-8?q?d_mention_of_=22Standard_Edition=22_VS_in_PCbuild/readme=2Etxt?= =?utf-8?q?=2E?= Message-ID: <3fwnGY1HfFz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/3036985cf895 changeset: 90018:3036985cf895 branch: 3.4 parent: 90016:4972475ae2e7 user: Zachary Ware date: Fri Mar 28 23:31:04 2014 -0500 summary: Fix another old mention of "Standard Edition" VS in PCbuild/readme.txt. Also removes a mention of using "Express Edition with the Windows SDK 64-bit compilers" to build 64-bit Python. If that can be made to work, it will need some extra instructions somewhere. files: PCbuild/readme.txt | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -278,8 +278,7 @@ you just have to set x64 as platform. In addition, the HOST_PYTHON environment variable must point to a Python interpreter (at least 2.4), to support cross-compilation from Win32. Note that Visual Studio -requires either Standard Edition or better, or Express Edition with the -Windows SDK 64-bit compilers to be available in order to build 64-bit +requires Professional Edition or better in order to build 64-bit binaries. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 05:37:26 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 29 Mar 2014 05:37:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <3fwnGZ35jgz7Ljc@mail.python.org> http://hg.python.org/cpython/rev/60b7bc5ec885 changeset: 90019:60b7bc5ec885 parent: 90017:ae5169297eb9 parent: 90018:3036985cf895 user: Zachary Ware date: Fri Mar 28 23:31:53 2014 -0500 summary: Merge with 3.4 files: PCbuild/readme.txt | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -266,8 +266,7 @@ you just have to set x64 as platform. In addition, the HOST_PYTHON environment variable must point to a Python interpreter (at least 2.4), to support cross-compilation from Win32. Note that Visual Studio -requires either Standard Edition or better, or Express Edition with the -Windows SDK 64-bit compilers to be available in order to build 64-bit +requires Professional Edition or better in order to build 64-bit binaries. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 05:37:27 2014 From: python-checkins at python.org (zach.ware) Date: Sat, 29 Mar 2014 05:37:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_tcltk=2Eprops_to_=22Vi?= =?utf-8?q?sual_Studio_properties=22_section_of_PCbuild/readme=2Etxt=2E?= Message-ID: <3fwnGb4lg1z7Ljt@mail.python.org> http://hg.python.org/cpython/rev/bca58d3cf035 changeset: 90020:bca58d3cf035 user: Zachary Ware date: Fri Mar 28 23:36:43 2014 -0500 summary: Add tcltk.props to "Visual Studio properties" section of PCbuild/readme.txt. files: PCbuild/readme.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -320,6 +320,7 @@ * pyproject (base settings for all projects, user macros like PyDllName) * release (release macro: NDEBUG) * sqlite3 (used only by sqlite3.vcxproj) + * tcltk (used by _tkinter, tcl, tk and tix projects) * x64 (AMD64 / x64 platform specific settings) The pyproject property file defines _WIN32 and x64 defines _WIN64 and -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 06:41:35 2014 From: python-checkins at python.org (nick.coghlan) Date: Sat, 29 Mar 2014 06:41:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE5Njk3?= =?utf-8?b?OiBkb2N1bWVudCBtb3JlIF9fbWFpbl9fLl9fc3BlY19fIHF1aXJrcw==?= Message-ID: <3fwphb1sc8z7LjY@mail.python.org> http://hg.python.org/cpython/rev/0427c2ff653d changeset: 90021:0427c2ff653d branch: 3.4 parent: 90018:3036985cf895 user: Nick Coghlan date: Sat Mar 29 15:39:42 2014 +1000 summary: Issue #19697: document more __main__.__spec__ quirks files: Doc/reference/import.rst | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -852,16 +852,30 @@ gets set appropriately or to ``None``. When Python is started with the :option:`-m` option, ``__spec__`` is set -to the module spec of the corresponding module. +to the module spec of the corresponding module or package. ``__spec__`` is +also populated when the ``__main__`` module is loaded as part of executing a +directory, zipfile or other :data:`sys.path` entry. In :ref:`the remaining cases ` -``__main__.__spec__`` is set to ``None``: +``__main__.__spec__`` is set to ``None``, as the code used to populate the +:mod:`__main__` does not correspond directly with an importable module: - interactive prompt - -c switch - running from stdin - running directly from a source or bytecode file +Note that ``__main__.__spec__`` is always ``None`` in the last case, +*even if* the file could technically be imported directly as a module +instead. Use the :option:`-m` switch if valid module metadata is desired +in :mod:`__main__`. + +Note also that even when ``__main__`` corresponds with an importable module +and ``__main__.__spec__`` is set accordingly, they're still considered +*distinct* modules. This is due to the fact that blocks guarded by +``if __name__ == "__main__":`` checks only execute when the module is used +to populate the ``__main__`` namespace, and not during normal import. + Open issues =========== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 06:41:36 2014 From: python-checkins at python.org (nick.coghlan) Date: Sat, 29 Mar 2014 06:41:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319697=3A_merge_docs_from_3=2E4?= Message-ID: <3fwphc39Cvz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/a90254be2da2 changeset: 90022:a90254be2da2 parent: 90020:bca58d3cf035 parent: 90021:0427c2ff653d user: Nick Coghlan date: Sat Mar 29 15:41:18 2014 +1000 summary: Issue #19697: merge docs from 3.4 files: Doc/reference/import.rst | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -852,16 +852,30 @@ gets set appropriately or to ``None``. When Python is started with the :option:`-m` option, ``__spec__`` is set -to the module spec of the corresponding module. +to the module spec of the corresponding module or package. ``__spec__`` is +also populated when the ``__main__`` module is loaded as part of executing a +directory, zipfile or other :data:`sys.path` entry. In :ref:`the remaining cases ` -``__main__.__spec__`` is set to ``None``: +``__main__.__spec__`` is set to ``None``, as the code used to populate the +:mod:`__main__` does not correspond directly with an importable module: - interactive prompt - -c switch - running from stdin - running directly from a source or bytecode file +Note that ``__main__.__spec__`` is always ``None`` in the last case, +*even if* the file could technically be imported directly as a module +instead. Use the :option:`-m` switch if valid module metadata is desired +in :mod:`__main__`. + +Note also that even when ``__main__`` corresponds with an importable module +and ``__main__.__spec__`` is set accordingly, they're still considered +*distinct* modules. This is due to the fact that blocks guarded by +``if __name__ == "__main__":`` checks only execute when the module is used +to populate the ``__main__`` namespace, and not during normal import. + Open issues =========== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 08:10:24 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 29 Mar 2014 08:10:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDkz?= =?utf-8?q?=3A_Prevent_failures_of_ctypes_test=5Fmacholib_on_OS_X_if_a?= Message-ID: <3fwrg41CCfz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/e30142fde075 changeset: 90023:e30142fde075 branch: 2.7 parent: 90010:3567d7ebd382 user: Ned Deily date: Sat Mar 29 00:07:42 2014 -0700 summary: Issue #21093: Prevent failures of ctypes test_macholib on OS X if a copy of libz exists in $HOME/lib or /usr/local/lib. files: Lib/ctypes/test/test_macholib.py | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/ctypes/test/test_macholib.py b/Lib/ctypes/test/test_macholib.py --- a/Lib/ctypes/test/test_macholib.py +++ b/Lib/ctypes/test/test_macholib.py @@ -52,8 +52,11 @@ '/usr/lib/libSystem.B.dylib') result = find_lib('z') - self.assertTrue(result.startswith('/usr/lib/libz.1')) - self.assertTrue(result.endswith('.dylib')) + # Issue #21093: dyld default search path includes $HOME/lib and + # /usr/local/lib before /usr/lib, which caused test failures if + # a local copy of libz exists in one of them. Now ignore the head + # of the path. + self.assertRegexpMatches(result, r".*/lib/libz\..*.*\.dylib") self.assertEqual(find_lib('IOKit'), '/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -336,6 +336,9 @@ redirect of http://www.python.org/ to https://www.python.org: use http://www.example.com instead. +- Issue #21093: Prevent failures of ctypes test_macholib on OS X if a + copy of libz exists in $HOME/lib or /usr/local/lib. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 08:10:25 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 29 Mar 2014 08:10:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDkz?= =?utf-8?q?=3A_Prevent_failures_of_ctypes_test=5Fmacholib_on_OS_X_if_a?= Message-ID: <3fwrg53DYwz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/831bd1a1cf6c changeset: 90024:831bd1a1cf6c branch: 3.4 parent: 90021:0427c2ff653d user: Ned Deily date: Sat Mar 29 00:08:44 2014 -0700 summary: Issue #21093: Prevent failures of ctypes test_macholib on OS X if a copy of libz exists in $HOME/lib or /usr/local/lib. files: Lib/ctypes/test/test_macholib.py | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/ctypes/test/test_macholib.py b/Lib/ctypes/test/test_macholib.py --- a/Lib/ctypes/test/test_macholib.py +++ b/Lib/ctypes/test/test_macholib.py @@ -50,8 +50,11 @@ '/usr/lib/libSystem.B.dylib') result = find_lib('z') - self.assertTrue(result.startswith('/usr/lib/libz.1')) - self.assertTrue(result.endswith('.dylib')) + # Issue #21093: dyld default search path includes $HOME/lib and + # /usr/local/lib before /usr/lib, which caused test failures if + # a local copy of libz exists in one of them. Now ignore the head + # of the path. + self.assertRegex(result, r".*/lib/libz\..*.*\.dylib") self.assertEqual(find_lib('IOKit'), '/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -129,6 +129,9 @@ - Issue #20668: asyncio tests no longer rely on tests.txt file. (Patch by Vajrasky Kok) +- Issue #21093: Prevent failures of ctypes test_macholib on OS X if a + copy of libz exists in $HOME/lib or /usr/local/lib. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 08:10:26 2014 From: python-checkins at python.org (ned.deily) Date: Sat, 29 Mar 2014 08:10:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2321093=3A_merge_from_3=2E4?= Message-ID: <3fwrg65RFMz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/33c4c01404cd changeset: 90025:33c4c01404cd parent: 90022:a90254be2da2 parent: 90024:831bd1a1cf6c user: Ned Deily date: Sat Mar 29 00:09:56 2014 -0700 summary: Issue #21093: merge from 3.4 files: Lib/ctypes/test/test_macholib.py | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/ctypes/test/test_macholib.py b/Lib/ctypes/test/test_macholib.py --- a/Lib/ctypes/test/test_macholib.py +++ b/Lib/ctypes/test/test_macholib.py @@ -50,8 +50,11 @@ '/usr/lib/libSystem.B.dylib') result = find_lib('z') - self.assertTrue(result.startswith('/usr/lib/libz.1')) - self.assertTrue(result.endswith('.dylib')) + # Issue #21093: dyld default search path includes $HOME/lib and + # /usr/local/lib before /usr/lib, which caused test failures if + # a local copy of libz exists in one of them. Now ignore the head + # of the path. + self.assertRegex(result, r".*/lib/libz\..*.*\.dylib") self.assertEqual(find_lib('IOKit'), '/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -159,6 +159,9 @@ - Issue #20668: asyncio tests no longer rely on tests.txt file. (Patch by Vajrasky Kok) +- Issue #21093: Prevent failures of ctypes test_macholib on OS X if a + copy of libz exists in $HOME/lib or /usr/local/lib. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Mar 29 09:54:30 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 29 Mar 2014 09:54:30 +0100 Subject: [Python-checkins] Daily reference leaks (ae5169297eb9): sum=0 Message-ID: results for ae5169297eb9 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogasCAEV', '-x'] From python-checkins at python.org Sat Mar 29 16:17:41 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 29 Mar 2014 16:17:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Withdraw_PEP_460_=28mostly_ob?= =?utf-8?q?soleted_by_PEP_461=29?= Message-ID: <3fx3TK3X54z7Ljl@mail.python.org> http://hg.python.org/peps/rev/9eb15a7aa3d6 changeset: 5446:9eb15a7aa3d6 user: Antoine Pitrou date: Sat Mar 29 16:17:37 2014 +0100 summary: Withdraw PEP 460 (mostly obsoleted by PEP 461) files: pep-0460.txt | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/pep-0460.txt b/pep-0460.txt --- a/pep-0460.txt +++ b/pep-0460.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou -Status: Draft +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 6-Jan-2014 @@ -137,6 +137,16 @@ allow for hashing), and one mutable type (as efficient accumulation is often necessary when working with network messages). + +Resolution +========== + +This PEP is made obsolete by the `acceptance +`_ +of :pep:`461`, which introduces a more extended formatting language for +bytes objects in conjunction with the modulo operator. + + References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Mar 29 18:18:44 2014 From: python-checkins at python.org (yury.selivanov) Date: Sat, 29 Mar 2014 18:18:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2Edocs=3A_Fix_inde?= =?utf-8?q?ntation_and_version-added_for_Signature=2Efrom=5Fcallable?= Message-ID: <3fx6900MF4z7LjW@mail.python.org> http://hg.python.org/cpython/rev/399ad1c31d91 changeset: 90026:399ad1c31d91 user: Yury Selivanov date: Sat Mar 29 13:18:30 2014 -0400 summary: inspect.docs: Fix indentation and version-added for Signature.from_callable files: Doc/library/inspect.rst | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -506,7 +506,7 @@ >>> str(new_sig) "(a, b) -> 'new return anno'" - .. classmethod:: Signature.from_callable(obj) + .. classmethod:: Signature.from_callable(obj) Return a :class:`Signature` (or its subclass) object for a given callable ``obj``. This method simplifies subclassing of :class:`Signature`: @@ -518,6 +518,8 @@ sig = MySignature.from_callable(min) assert isinstance(sig, MySignature) + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 18:24:36 2014 From: python-checkins at python.org (yury.selivanov) Date: Sat, 29 Mar 2014 18:24:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=2Edocs=3A_Document?= =?utf-8?q?_that_Signature_and_Parameter_are_now_picklable_=28issue?= Message-ID: <3fx6Hm4ZtRz7LjW@mail.python.org> http://hg.python.org/cpython/rev/f2bae047f6d0 changeset: 90027:f2bae047f6d0 user: Yury Selivanov date: Sat Mar 29 13:24:14 2014 -0400 summary: inspect.docs: Document that Signature and Parameter are now picklable (issue #20726) files: Doc/library/inspect.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -462,6 +462,9 @@ Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -526,6 +529,9 @@ Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 18:47:21 2014 From: python-checkins at python.org (yury.selivanov) Date: Sat, 29 Mar 2014 18:47:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=3A_Update_docstrin?= =?utf-8?q?gs=3B_Rename_=5Fsignature=5Finternal_to?= Message-ID: <3fx6p102Lkz7LkM@mail.python.org> http://hg.python.org/cpython/rev/bc4186df43ad changeset: 90028:bc4186df43ad user: Yury Selivanov date: Sat Mar 29 13:47:11 2014 -0400 summary: inspect: Update docstrings; Rename _signature_internal to _signature_from_callable. files: Lib/inspect.py | 135 +++++++++++++++++++++--------------- 1 files changed, 78 insertions(+), 57 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -967,10 +967,10 @@ # getfullargspec() historically ignored __wrapped__ attributes, # so we ensure that remains the case in 3.3+ - sig = _signature_internal(func, - follow_wrapper_chains=False, - skip_bound_arg=False, - sigcls=Signature) + sig = _signature_from_callable(func, + follow_wrapper_chains=False, + skip_bound_arg=False, + sigcls=Signature) except Exception as ex: # Most of the times 'signature' will raise ValueError. # But, it can also raise AttributeError, and, maybe something @@ -1497,6 +1497,10 @@ def _signature_get_user_defined_method(cls, method_name): + """Private helper. Checks if ``cls`` has an attribute + named ``method_name`` and returns it only if it is a + pure python function. + """ try: meth = getattr(cls, method_name) except AttributeError: @@ -1509,9 +1513,10 @@ def _signature_get_partial(wrapped_sig, partial, extra_args=()): - # Internal helper to calculate how 'wrapped_sig' signature will - # look like after applying a 'functools.partial' object (or alike) - # on it. + """Private helper to calculate how 'wrapped_sig' signature will + look like after applying a 'functools.partial' object (or alike) + on it. + """ new_params = OrderedDict(wrapped_sig.parameters.items()) @@ -1558,8 +1563,9 @@ def _signature_bound_method(sig): - # Internal helper to transform signatures for unbound - # functions to bound methods + """Private helper to transform signatures for unbound + functions to bound methods. + """ params = tuple(sig.parameters.values()) @@ -1583,8 +1589,9 @@ def _signature_is_builtin(obj): - # Internal helper to test if `obj` is a callable that might - # support Argument Clinic's __text_signature__ protocol. + """Private helper to test if `obj` is a callable that might + support Argument Clinic's __text_signature__ protocol. + """ return (isbuiltin(obj) or ismethoddescriptor(obj) or isinstance(obj, _NonUserDefinedCallables) or @@ -1594,10 +1601,11 @@ def _signature_is_functionlike(obj): - # Internal helper to test if `obj` is a duck type of FunctionType. - # A good example of such objects are functions compiled with - # Cython, which have all attributes that a pure Python function - # would have, but have their code statically compiled. + """Private helper to test if `obj` is a duck type of FunctionType. + A good example of such objects are functions compiled with + Cython, which have all attributes that a pure Python function + would have, but have their code statically compiled. + """ if not callable(obj) or isclass(obj): # All function-like objects are obviously callables, @@ -1618,11 +1626,12 @@ def _signature_get_bound_param(spec): - # Internal helper to get first parameter name from a - # __text_signature__ of a builtin method, which should - # be in the following format: '($param1, ...)'. - # Assumptions are that the first argument won't have - # a default value or an annotation. + """ Private helper to get first parameter name from a + __text_signature__ of a builtin method, which should + be in the following format: '($param1, ...)'. + Assumptions are that the first argument won't have + a default value or an annotation. + """ assert spec.startswith('($') @@ -1641,7 +1650,9 @@ def _signature_strip_non_python_syntax(signature): """ - Takes a signature in Argument Clinic's extended signature format. + Private helper function. Takes a signature in Argument Clinic's + extended signature format. + Returns a tuple of three things: * that signature re-rendered in standard Python syntax, * the index of the "self" parameter (generally 0), or None if @@ -1710,8 +1721,10 @@ def _signature_fromstr(cls, obj, s, skip_bound_arg=True): - # Internal helper to parse content of '__text_signature__' - # and return a Signature based on it + """Private helper to parse content of '__text_signature__' + and return a Signature based on it. + """ + Parameter = cls._parameter_cls clean_signature, self_parameter, last_positional_only = \ @@ -1849,8 +1862,10 @@ def _signature_from_builtin(cls, func, skip_bound_arg=True): - # Internal helper function to get signature for - # builtin callables + """Private helper function to get signature for + builtin callables. + """ + if not _signature_is_builtin(func): raise TypeError("{!r} is not a Python builtin " "function".format(func)) @@ -1862,10 +1877,14 @@ return _signature_fromstr(cls, func, s, skip_bound_arg) -def _signature_internal(obj, *, - follow_wrapper_chains=True, - skip_bound_arg=True, - sigcls): +def _signature_from_callable(obj, *, + follow_wrapper_chains=True, + skip_bound_arg=True, + sigcls): + + """Private helper function to get signature for arbitrary + callable objects. + """ if not callable(obj): raise TypeError('{!r} is not a callable object'.format(obj)) @@ -1873,7 +1892,7 @@ if isinstance(obj, types.MethodType): # In this case we skip the first parameter of the underlying # function (usually `self` or `cls`). - sig = _signature_internal( + sig = _signature_from_callable( obj.__func__, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -1909,7 +1928,7 @@ # (usually `self`, or `cls`) will not be passed # automatically (as for boundmethods) - wrapped_sig = _signature_internal( + wrapped_sig = _signature_from_callable( partialmethod.func, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -1932,7 +1951,7 @@ skip_bound_arg=skip_bound_arg) if isinstance(obj, functools.partial): - wrapped_sig = _signature_internal( + wrapped_sig = _signature_from_callable( obj.func, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -1947,7 +1966,7 @@ # in its metaclass call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: - sig = _signature_internal( + sig = _signature_from_callable( call, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -1956,7 +1975,7 @@ # Now we check if the 'obj' class has a '__new__' method new = _signature_get_user_defined_method(obj, '__new__') if new is not None: - sig = _signature_internal( + sig = _signature_from_callable( new, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -1965,7 +1984,7 @@ # Finally, we should have at least __init__ implemented init = _signature_get_user_defined_method(obj, '__init__') if init is not None: - sig = _signature_internal( + sig = _signature_from_callable( init, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -2011,7 +2030,7 @@ call = _signature_get_user_defined_method(type(obj), '__call__') if call is not None: try: - sig = _signature_internal( + sig = _signature_from_callable( call, follow_wrapper_chains=follow_wrapper_chains, skip_bound_arg=skip_bound_arg, @@ -2037,11 +2056,11 @@ class _void: - '''A private marker - used in Parameter & Signature''' + """A private marker - used in Parameter & Signature.""" class _empty: - pass + """Marker object for Signature.empty and Parameter.empty.""" class _ParameterKind(enum.IntEnum): @@ -2063,7 +2082,7 @@ class Parameter: - '''Represents a parameter in a function signature. + """Represents a parameter in a function signature. Has the following public attributes: @@ -2082,7 +2101,7 @@ Possible values: `Parameter.POSITIONAL_ONLY`, `Parameter.POSITIONAL_OR_KEYWORD`, `Parameter.VAR_POSITIONAL`, `Parameter.KEYWORD_ONLY`, `Parameter.VAR_KEYWORD`. - ''' + """ __slots__ = ('_name', '_kind', '_default', '_annotation', '_partial_kwarg') @@ -2152,7 +2171,7 @@ def replace(self, *, name=_void, kind=_void, annotation=_void, default=_void, _partial_kwarg=_void): - '''Creates a customized copy of the Parameter.''' + """Creates a customized copy of the Parameter.""" if name is _void: name = self._name @@ -2218,7 +2237,7 @@ class BoundArguments: - '''Result of `Signature.bind` call. Holds the mapping of arguments + """Result of `Signature.bind` call. Holds the mapping of arguments to the function's parameters. Has the following public attributes: @@ -2232,7 +2251,7 @@ Tuple of positional arguments values. * kwargs : dict Dict of keyword arguments values. - ''' + """ def __init__(self, signature, arguments): self.arguments = arguments @@ -2311,7 +2330,7 @@ class Signature: - '''A Signature object represents the overall signature of a function. + """A Signature object represents the overall signature of a function. It stores a Parameter object for each parameter accepted by the function, as well as information specific to the function itself. @@ -2331,7 +2350,7 @@ * bind_partial(*args, **kwargs) -> BoundArguments Creates a partial mapping from positional and keyword arguments to parameters (simulating 'functools.partial' behavior.) - ''' + """ __slots__ = ('_return_annotation', '_parameters') @@ -2342,9 +2361,9 @@ def __init__(self, parameters=None, *, return_annotation=_empty, __validate_parameters__=True): - '''Constructs Signature from the given list of Parameter + """Constructs Signature from the given list of Parameter objects and 'return_annotation'. All arguments are optional. - ''' + """ if parameters is None: params = OrderedDict() @@ -2397,7 +2416,7 @@ @classmethod def from_function(cls, func): - '''Constructs Signature for the given python function''' + """Constructs Signature for the given python function.""" is_duck_function = False if not isfunction(func): @@ -2478,11 +2497,13 @@ @classmethod def from_builtin(cls, func): + """Constructs Signature for the given builtin function.""" return _signature_from_builtin(cls, func) @classmethod def from_callable(cls, obj): - return _signature_internal(obj, sigcls=cls) + """Constructs Signature for the given callable object.""" + return _signature_from_callable(obj, sigcls=cls) @property def parameters(self): @@ -2493,10 +2514,10 @@ return self._return_annotation def replace(self, *, parameters=_void, return_annotation=_void): - '''Creates a customized copy of the Signature. + """Creates a customized copy of the Signature. Pass 'parameters' and/or 'return_annotation' arguments to override them in the new copy. - ''' + """ if parameters is _void: parameters = self.parameters.values() @@ -2541,7 +2562,7 @@ return not self.__eq__(other) def _bind(self, args, kwargs, *, partial=False): - '''Private method. Don't use directly.''' + """Private method. Don't use directly.""" arguments = OrderedDict() @@ -2677,17 +2698,17 @@ return self._bound_arguments_cls(self, arguments) def bind(*args, **kwargs): - '''Get a BoundArguments object, that maps the passed `args` + """Get a BoundArguments object, that maps the passed `args` and `kwargs` to the function's signature. Raises `TypeError` if the passed arguments can not be bound. - ''' + """ return args[0]._bind(args[1:], kwargs) def bind_partial(*args, **kwargs): - '''Get a BoundArguments object, that partially maps the + """Get a BoundArguments object, that partially maps the passed `args` and `kwargs` to the function's signature. Raises `TypeError` if the passed arguments can not be bound. - ''' + """ return args[0]._bind(args[1:], kwargs, partial=True) def __reduce__(self): @@ -2749,7 +2770,7 @@ def signature(obj): - '''Get a signature object for the passed callable.''' + """Get a signature object for the passed callable.""" return Signature.from_callable(obj) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 19:49:17 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 29 Mar 2014 19:49:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Minor_readabil?= =?utf-8?q?ity_improvement=2E?= Message-ID: <3fx89T2Qwqz7LjY@mail.python.org> http://hg.python.org/cpython/rev/3c577e68859e changeset: 90029:3c577e68859e branch: 2.7 parent: 90023:e30142fde075 user: Raymond Hettinger date: Sat Mar 29 11:49:11 2014 -0700 summary: Minor readability improvement. files: Doc/library/xml.etree.elementtree.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -319,7 +319,7 @@ +=======================+======================================================+ | ``tag`` | Selects all child elements with the given tag. | | | For example, ``spam`` selects all child elements | -| | named ``spam``, ``spam/egg`` selects all | +| | named ``spam``, and ``spam/egg`` selects all | | | grandchildren named ``egg`` in all children named | | | ``spam``. | +-----------------------+------------------------------------------------------+ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 19:50:40 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 29 Mar 2014 19:50:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Minor_readabil?= =?utf-8?q?ity_improvement=2E?= Message-ID: <3fx8C41yrwz7LjY@mail.python.org> http://hg.python.org/cpython/rev/59f57d729d9a changeset: 90030:59f57d729d9a branch: 3.4 parent: 90024:831bd1a1cf6c user: Raymond Hettinger date: Sat Mar 29 11:50:08 2014 -0700 summary: Minor readability improvement. files: Doc/library/xml.etree.elementtree.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -325,7 +325,7 @@ +=======================+======================================================+ | ``tag`` | Selects all child elements with the given tag. | | | For example, ``spam`` selects all child elements | -| | named ``spam``, ``spam/egg`` selects all | +| | named ``spam``, and ``spam/egg`` selects all | | | grandchildren named ``egg`` in all children named | | | ``spam``. | +-----------------------+------------------------------------------------------+ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 19:50:41 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 29 Mar 2014 19:50:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3fx8C53lq4z7LkZ@mail.python.org> http://hg.python.org/cpython/rev/36492f927a40 changeset: 90031:36492f927a40 parent: 90028:bc4186df43ad parent: 90030:59f57d729d9a user: Raymond Hettinger date: Sat Mar 29 11:50:29 2014 -0700 summary: merge files: Doc/library/xml.etree.elementtree.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -325,7 +325,7 @@ +=======================+======================================================+ | ``tag`` | Selects all child elements with the given tag. | | | For example, ``spam`` selects all child elements | -| | named ``spam``, ``spam/egg`` selects all | +| | named ``spam``, and ``spam/egg`` selects all | | | grandchildren named ``egg`` in all children named | | | ``spam``. | +-----------------------+------------------------------------------------------+ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 23:08:25 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 23:08:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321097=3A_Move_tes?= =?utf-8?q?t=5Fnamespace=5Fpkgs_into_test=5Fimportlib=2E?= Message-ID: <3fxDbF0wq7z7LlQ@mail.python.org> http://hg.python.org/cpython/rev/6d44bd8066ee changeset: 90032:6d44bd8066ee user: Eric Snow date: Sat Mar 29 15:57:44 2014 -0600 summary: Issue #21097: Move test_namespace_pkgs into test_importlib. files: Lib/test/namespace_pkgs/both_portions/foo/one.py | 0 Lib/test/namespace_pkgs/both_portions/foo/two.py | 0 Lib/test/namespace_pkgs/missing_directory.zip | 0 Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py | 0 Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty | 0 Lib/test/namespace_pkgs/nested_portion1.zip | 0 Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py | 0 Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py | 0 Lib/test/namespace_pkgs/portion1/foo/one.py | 0 Lib/test/namespace_pkgs/portion2/foo/two.py | 0 Lib/test/namespace_pkgs/project1/parent/child/one.py | 0 Lib/test/namespace_pkgs/project2/parent/child/two.py | 0 Lib/test/namespace_pkgs/project3/parent/child/three.py | 0 Lib/test/namespace_pkgs/top_level_portion1.zip | 0 Lib/test/test_namespace_pkgs.py | 0 Misc/NEWS | 2 ++ 16 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/namespace_pkgs/both_portions/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/both_portions/foo/one.py rename from Lib/test/namespace_pkgs/both_portions/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/both_portions/foo/one.py diff --git a/Lib/test/namespace_pkgs/both_portions/foo/two.py b/Lib/test/test_importlib/namespace_pkgs/both_portions/foo/two.py rename from Lib/test/namespace_pkgs/both_portions/foo/two.py rename to Lib/test/test_importlib/namespace_pkgs/both_portions/foo/two.py diff --git a/Lib/test/namespace_pkgs/missing_directory.zip b/Lib/test/test_importlib/namespace_pkgs/missing_directory.zip rename from Lib/test/namespace_pkgs/missing_directory.zip rename to Lib/test/test_importlib/namespace_pkgs/missing_directory.zip diff --git a/Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py b/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test.py rename from Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py rename to Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test.py diff --git a/Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty b/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test/empty rename from Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty rename to Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test/empty diff --git a/Lib/test/namespace_pkgs/nested_portion1.zip b/Lib/test/test_importlib/namespace_pkgs/nested_portion1.zip rename from Lib/test/namespace_pkgs/nested_portion1.zip rename to Lib/test/test_importlib/namespace_pkgs/nested_portion1.zip diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py b/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py rename from Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py rename to Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/one.py rename from Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/one.py diff --git a/Lib/test/namespace_pkgs/portion1/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/portion1/foo/one.py rename from Lib/test/namespace_pkgs/portion1/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/portion1/foo/one.py diff --git a/Lib/test/namespace_pkgs/portion2/foo/two.py b/Lib/test/test_importlib/namespace_pkgs/portion2/foo/two.py rename from Lib/test/namespace_pkgs/portion2/foo/two.py rename to Lib/test/test_importlib/namespace_pkgs/portion2/foo/two.py diff --git a/Lib/test/namespace_pkgs/project1/parent/child/one.py b/Lib/test/test_importlib/namespace_pkgs/project1/parent/child/one.py rename from Lib/test/namespace_pkgs/project1/parent/child/one.py rename to Lib/test/test_importlib/namespace_pkgs/project1/parent/child/one.py diff --git a/Lib/test/namespace_pkgs/project2/parent/child/two.py b/Lib/test/test_importlib/namespace_pkgs/project2/parent/child/two.py rename from Lib/test/namespace_pkgs/project2/parent/child/two.py rename to Lib/test/test_importlib/namespace_pkgs/project2/parent/child/two.py diff --git a/Lib/test/namespace_pkgs/project3/parent/child/three.py b/Lib/test/test_importlib/namespace_pkgs/project3/parent/child/three.py rename from Lib/test/namespace_pkgs/project3/parent/child/three.py rename to Lib/test/test_importlib/namespace_pkgs/project3/parent/child/three.py diff --git a/Lib/test/namespace_pkgs/top_level_portion1.zip b/Lib/test/test_importlib/namespace_pkgs/top_level_portion1.zip rename from Lib/test/namespace_pkgs/top_level_portion1.zip rename to Lib/test/test_importlib/namespace_pkgs/top_level_portion1.zip diff --git a/Lib/test/test_namespace_pkgs.py b/Lib/test/test_importlib/test_namespace_pkgs.py rename from Lib/test/test_namespace_pkgs.py rename to Lib/test/test_importlib/test_namespace_pkgs.py diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -152,6 +152,8 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #21097: Move test_namespace_pkgs into test_importlib. + - Issue #20939: Avoid various network test failures due to new redirect of http://www.python.org/ to https://www.python.org: use http://www.example.com instead. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 23:15:23 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 23:15:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDk3?= =?utf-8?q?=3A_Move_test=5Fnamespace=5Fpkgs_into_test=5Fimportlib=2E?= Message-ID: <3fxDlH0GZtz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/48790b202a50 changeset: 90033:48790b202a50 branch: 3.4 parent: 90030:59f57d729d9a user: Eric Snow date: Sat Mar 29 15:57:44 2014 -0600 summary: Issue #21097: Move test_namespace_pkgs into test_importlib. files: Lib/test/namespace_pkgs/both_portions/foo/one.py | 0 Lib/test/namespace_pkgs/both_portions/foo/two.py | 0 Lib/test/namespace_pkgs/missing_directory.zip | 0 Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py | 0 Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty | 0 Lib/test/namespace_pkgs/nested_portion1.zip | 0 Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py | 0 Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py | 0 Lib/test/namespace_pkgs/portion1/foo/one.py | 0 Lib/test/namespace_pkgs/portion2/foo/two.py | 0 Lib/test/namespace_pkgs/project1/parent/child/one.py | 0 Lib/test/namespace_pkgs/project2/parent/child/two.py | 0 Lib/test/namespace_pkgs/project3/parent/child/three.py | 0 Lib/test/namespace_pkgs/top_level_portion1.zip | 0 Lib/test/test_namespace_pkgs.py | 0 Misc/NEWS | 2 ++ 16 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/namespace_pkgs/both_portions/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/both_portions/foo/one.py rename from Lib/test/namespace_pkgs/both_portions/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/both_portions/foo/one.py diff --git a/Lib/test/namespace_pkgs/both_portions/foo/two.py b/Lib/test/test_importlib/namespace_pkgs/both_portions/foo/two.py rename from Lib/test/namespace_pkgs/both_portions/foo/two.py rename to Lib/test/test_importlib/namespace_pkgs/both_portions/foo/two.py diff --git a/Lib/test/namespace_pkgs/missing_directory.zip b/Lib/test/test_importlib/namespace_pkgs/missing_directory.zip rename from Lib/test/namespace_pkgs/missing_directory.zip rename to Lib/test/test_importlib/namespace_pkgs/missing_directory.zip diff --git a/Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py b/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test.py rename from Lib/test/namespace_pkgs/module_and_namespace_package/a_test.py rename to Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test.py diff --git a/Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty b/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test/empty rename from Lib/test/namespace_pkgs/module_and_namespace_package/a_test/empty rename to Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test/empty diff --git a/Lib/test/namespace_pkgs/nested_portion1.zip b/Lib/test/test_importlib/namespace_pkgs/nested_portion1.zip rename from Lib/test/namespace_pkgs/nested_portion1.zip rename to Lib/test/test_importlib/namespace_pkgs/nested_portion1.zip diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py b/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py rename from Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py rename to Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/one.py rename from Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/one.py diff --git a/Lib/test/namespace_pkgs/portion1/foo/one.py b/Lib/test/test_importlib/namespace_pkgs/portion1/foo/one.py rename from Lib/test/namespace_pkgs/portion1/foo/one.py rename to Lib/test/test_importlib/namespace_pkgs/portion1/foo/one.py diff --git a/Lib/test/namespace_pkgs/portion2/foo/two.py b/Lib/test/test_importlib/namespace_pkgs/portion2/foo/two.py rename from Lib/test/namespace_pkgs/portion2/foo/two.py rename to Lib/test/test_importlib/namespace_pkgs/portion2/foo/two.py diff --git a/Lib/test/namespace_pkgs/project1/parent/child/one.py b/Lib/test/test_importlib/namespace_pkgs/project1/parent/child/one.py rename from Lib/test/namespace_pkgs/project1/parent/child/one.py rename to Lib/test/test_importlib/namespace_pkgs/project1/parent/child/one.py diff --git a/Lib/test/namespace_pkgs/project2/parent/child/two.py b/Lib/test/test_importlib/namespace_pkgs/project2/parent/child/two.py rename from Lib/test/namespace_pkgs/project2/parent/child/two.py rename to Lib/test/test_importlib/namespace_pkgs/project2/parent/child/two.py diff --git a/Lib/test/namespace_pkgs/project3/parent/child/three.py b/Lib/test/test_importlib/namespace_pkgs/project3/parent/child/three.py rename from Lib/test/namespace_pkgs/project3/parent/child/three.py rename to Lib/test/test_importlib/namespace_pkgs/project3/parent/child/three.py diff --git a/Lib/test/namespace_pkgs/top_level_portion1.zip b/Lib/test/test_importlib/namespace_pkgs/top_level_portion1.zip rename from Lib/test/namespace_pkgs/top_level_portion1.zip rename to Lib/test/test_importlib/namespace_pkgs/top_level_portion1.zip diff --git a/Lib/test/test_namespace_pkgs.py b/Lib/test/test_importlib/test_namespace_pkgs.py rename from Lib/test/test_namespace_pkgs.py rename to Lib/test/test_importlib/test_namespace_pkgs.py diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -122,6 +122,8 @@ - Issue #20743: Fix a reference leak in test_tcl. +- Issue #21097: Move test_namespace_pkgs into test_importlib. + - Issue #20939: Avoid various network test failures due to new redirect of http://www.python.org/ to https://www.python.org: use http://www.example.com instead. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Mar 29 23:15:24 2014 From: python-checkins at python.org (eric.snow) Date: Sat, 29 Mar 2014 23:15:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_from_3=2E4=2E?= Message-ID: <3fxDlJ1hKFz7Lkm@mail.python.org> http://hg.python.org/cpython/rev/a8bb571b41ab changeset: 90034:a8bb571b41ab parent: 90032:6d44bd8066ee parent: 90033:48790b202a50 user: Eric Snow date: Sat Mar 29 16:14:52 2014 -0600 summary: Null merge from 3.4. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 03:28:44 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 30 Mar 2014 03:28:44 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_467=3A_bytes/bytearray_AP?= =?utf-8?q?I_=26_docs_improvements?= Message-ID: <3fxK2N3Ljmz7LjW@mail.python.org> http://hg.python.org/peps/rev/d7bcb861dcff changeset: 5447:d7bcb861dcff user: Nick Coghlan date: Sun Mar 30 11:28:34 2014 +1000 summary: PEP 467: bytes/bytearray API & docs improvements files: pep-0467.txt | 280 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 280 insertions(+), 0 deletions(-) diff --git a/pep-0467.txt b/pep-0467.txt new file mode 100644 --- /dev/null +++ b/pep-0467.txt @@ -0,0 +1,280 @@ +PEP: 467 +Title: Improved API consistency for bytes and bytearray +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 2014-03-30 +Python-Version: 3.5 +Post-History: 2014-03-30 + + +Abstract +======== + +During the initial development of the Python 3 language specification, the +core ``bytes`` type for arbitrary binary data started as the mutable type +that is now referred to as ``bytearray``. Other aspects of operating in +the binary domain in Python have also evolved over the course of the Python +3 series. + +This PEP proposes a number of small adjustments to the APIs of the ``bytes`` +and ``bytearray`` types to make their behaviour more internally consistent +and to make it easier to operate entirely in the binary domain for use cases +that actually involve manipulating binary data directly, rather than +converting it to a more structured form with additional modelling +semantics (such as ``str``) and then converting back to binary format after +processing. + + +Background +========== + +Over the course of Python 3's evolution, a number of adjustments have been +made to the core ``bytes`` and ``bytearray`` types as additional practical +experience was gained with using them in code beyond the Python 3 standard +library and test suite. However, to date, these changes have been made +on a relatively ad hoc tactical basis as specific issues were identified, +rather than as part of a systematic review of the APIs of these types. This +approach has allowed inconsistencies to creep into the API design as to which +input types are accepted by different methods. Additional inconsistencies +linger from an earlier pre-release design where there was *no* separate +``bytearray`` type, and instead the core ``bytes`` type was mutable (with +no immutable counterpart), as well as from the origins of these types in +the text-like behaviour of the Python 2 ``str`` type. + +This PEP aims to provide the missing systematic review, with the goal of +ensuring that wherever feasible (given backwards compatibility constraints) +these current inconsistencies are addressed for the Python 3.5 release. + + +Proposals +========= + +As a "consistency improvement" proposal, this PEP is actually about a number +of smaller micro-proposals, each aimed at improving the self-consistency of +the binary data model in Python 3. Proposals are motivated by one of three +factors: + +* removing remnants of the original design of ``bytes`` as a mutable type +* more consistently accepting length 1 ``bytes`` objects as input where an + integer between ``0`` and ``255`` inclusive is expected, and vice-versa +* allowing users to easily convert integer output to a length 1 ``bytes`` + object + + +Alternate Constructors +---------------------- + +The ``bytes`` and ``bytearray`` constructors currently accept an integer +argument, but interpret it to mean a zero-filled object of the given length. +This is a legacy of the original design of ``bytes`` as a mutable type, +rather than a particularly intuitive behaviour for users. It has become +especially confusing now that other ``bytes`` interfaces treat integers +and the corresponding length 1 bytes instances as equivalent input. +Compare:: + + >>> b"\x03" in bytes([1, 2, 3]) + True + >>> 3 in bytes([1, 2, 3]) + True + + >>> bytes(b"\x03") + b'\x03' + >>> bytes(3) + b'\x00\x00\x00' + +This PEP proposes that the current handling of integers in the bytes and +bytearray constructors by deprecated in Python 3.5 and removed in Python +3.6, being replaced by two more type appropriate alternate constructors +provided as class methods. The initial python-ideas thread [ideas-thread1]_ +that spawned this PEP was specifically aimed at deprecating this constructor +behaviour. + +For ``bytes``, a ``byte`` constructor is proposed that converts integers +(as indicated by ``operator.index``) in the appropriate range to a ``bytes`` +object, converts objects that support the buffer API to bytes, and also +passes through length 1 byte strings unchanged:: + + >>> bytes.byte(3) + b'\x03' + >>> bytes.byte(bytearray(bytes([3]))) + b'\x03' + >>> bytes.byte(memoryview(bytes([3]))) + b'\x03' + >>> bytes.byte(bytes([3])) + b'\x03' + >>> bytes.byte(512) + Traceback (most recent call last): + File "", line 1, in + ValueError: bytes must be in range(0, 256) + >>> bytes.byte(b"ab") + Traceback (most recent call last): + File "", line 1, in + TypeError: bytes.byte() expected a byte, but buffer of length 2 found + +One specific use case for this alternate constructor is to easily convert +the result of indexing operations on ``bytes`` and other binary sequences +from an integer to a ``bytes`` object. The documentation for this API +should note that its counterpart for the reverse conversion is ``ord()``. + +For ``bytearray``, a ``from_len`` constructor is proposed that preallocates +the buffer filled with a particular value (default to ``0``) as a direct +replacement for the current constructor behaviour, rather than having to use +sequence repetition to achieve the same effect in a less intuitive way:: + + >>> bytearray.from_len(3) + bytearray(b'\x00\x00\x00') + >>> bytearray.from_len(3, 6) + bytearray(b'\x06\x06\x06') + +This part of the proposal was covered by an existing issue +[empty-buffer-issue]_ and a variety of names have been proposed +(``empty_buffer``, ``zeros``, ``zeroes``, ``allnull``, ``fill``). The +specific name currently proposed was chosen by analogy with +``dict.fromkeys()`` and ``itertools.chain.from_iter()`` to be completely +explicit that it is an alternate constructor rather than an in-place +mutation, as well as how it differs from the standard constructor. + +Open questions +^^^^^^^^^^^^^^ + +* Should ``bytearray.byte()`` also be added? Or is + ``bytearray(bytes.byte(x))`` sufficient for that case? +* Should ``bytes.from_len()`` also be added? Or is sequence repetition + sufficient for that case? +* Should ``bytearry.from_len()`` use a different name? +* Should ``bytes.byte()`` raise ``TypeError`` or ``ValueError`` for binary + sequences with more than one element? The ``TypeError`` currently proposed + is copied from the behaviour of ``ord()`` with strings containing more + than one code point, while ``ValueError`` would be more consistent with + the existing handling of out-of-range integer values. +* ``bytes.byte()`` is defined above as accepting length 1 binary sequences + as individual bytes, but this is currently inconsistent with the main + ``bytes`` constructor:: + + >>> bytes([b"a", b"b", b"c"]) + Traceback (most recent call last): + File "", line 1, in + TypeError: 'bytes' object cannot be interpreted as an integer + + Should the ``bytes`` constructor be changed to accept iterables of length 1 + bytes objects in addition to iterables of integers? If so, should it + allow a mixture of the two in a single iterable? + +Iteration +--------- + +Iteration over ``bytes`` objects and other binary sequences produces +integers. Rather than proposing a new method that would need to be added +not only to ``bytes``, ``bytearray`` and ``memoryview``, but potentially +to third party types as well, this PEP proposes that iteration to produce +length 1 ``bytes`` objects instead be handled by combining ``map`` with +the new ``bytes.byte()`` alternate constructor proposed above:: + + for x in map(bytes.byte, data): + # x is a length 1 ``bytes`` object, rather than an integer + # This works with *any* container of integers in the range + # 0 to 255 inclusive + + +Consistently accepting integer inputs to methods +------------------------------------------------ + +In Python 3.3, the binary search operations (``in``, ``count()``, +``find()``, ``index()``, ``rfind()`` and ``rindex()``) were updated to +accept integers in the range 0 to 255 (inclusive) as their first argument +(in addition to the existing support for binary sequences). + +This PEP proposes extending that behaviour of accepting integers as being +equivalent to the corresponding length 1 binary sequence to several other +``bytes`` and ``bytearray`` methods that currently expect a ``bytes`` +object for certain parameters. + +* ``startswith()`` prefix(es) +* ``endswith()`` suffix(es) + +* ``center()`` fill character +* ``ljust()`` fill character +* ``rjust()`` fill character + +* ``strip()`` characters to strip +* ``lstrip()`` characters to strip +* ``rstrip()`` characters to strip + +* ``partition()`` separator argument +* ``rpartition()`` separator argument + +* ``split()`` separator argument +* ``rsplit()`` separator argument + +* ``replace()`` old value and new value + +In addition to the consistency motive, this approach also makes it easier +to work with the indexing behaviour , as the result of an indexing operation +can more easily be fed back in to other methods. + + +Acknowledgement of surprising behaviour of some ``bytearray`` methods +--------------------------------------------------------------------- + +Several of the ``bytes`` and ``bytearray`` methods have their origins in the +Python 2 ``str`` API. As ``str`` is an immutable type, all of these +operations are defined as returning a *new* instances, rather than operating +in place. This contrasts with methods on other mutable types like ``list``, +where ``list.sort()`` and ``list.reverse()`` operate in-place and return +``None``, rather than creating a new object. + +Backwards compatibility constraints make it impractical to change this +behaviour at this point, but it may be appropriate to explicitly call out +this quirk in the documentation for the ``bytearray`` type. It affects the +following methods that could reasonably be expected to operate in-place on +a mutable type: + +* ``center()`` +* ``ljust()`` +* ``rjust()`` +* ``strip()`` +* ``lstrip()`` +* ``rstrip()`` +* ``replace()`` +* ``lower()`` +* ``upper()`` +* ``swapcase()`` +* ``title()`` +* ``capitalize()`` +* ``translate()`` +* ``expandtabs()`` +* ``zfill()`` + +Note that the following ``bytearray`` operations *do* operate in place, as +they're part of the mutable sequence API in ``bytearray``, rather than being +inspired by the immutable Python 2 ``str`` API: + +* ``reverse()`` +* ``remove()`` +* ``pop()`` + +References +========== + +.. [ideas-thread1] https://mail.python.org/pipermail/python-ideas/2014-March/027295.html +.. [empty-buffer-issue] http://bugs.python.org/issue20895 + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 30 03:55:05 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 30 Mar 2014 03:55:05 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_467=3A_Additional_tweaks_?= =?utf-8?q?after_re-reading?= Message-ID: <3fxKcn3lkVz7LjN@mail.python.org> http://hg.python.org/peps/rev/7d095b0d3109 changeset: 5448:7d095b0d3109 user: Nick Coghlan date: Sun Mar 30 11:54:55 2014 +1000 summary: PEP 467: Additional tweaks after re-reading files: pep-0467.txt | 40 +++++++++++++++++++++++++++++----------- 1 files changed, 29 insertions(+), 11 deletions(-) diff --git a/pep-0467.txt b/pep-0467.txt --- a/pep-0467.txt +++ b/pep-0467.txt @@ -138,6 +138,7 @@ explicit that it is an alternate constructor rather than an in-place mutation, as well as how it differs from the standard constructor. + Open questions ^^^^^^^^^^^^^^ @@ -145,12 +146,13 @@ ``bytearray(bytes.byte(x))`` sufficient for that case? * Should ``bytes.from_len()`` also be added? Or is sequence repetition sufficient for that case? -* Should ``bytearry.from_len()`` use a different name? +* Should ``bytearray.from_len()`` use a different name? * Should ``bytes.byte()`` raise ``TypeError`` or ``ValueError`` for binary sequences with more than one element? The ``TypeError`` currently proposed - is copied from the behaviour of ``ord()`` with strings containing more - than one code point, while ``ValueError`` would be more consistent with - the existing handling of out-of-range integer values. + is copied (with slightly improved wording) from the behaviour of ``ord()`` + with sequences containing more than one code point, while ``ValueError`` + would be more consistent with the existing handling of out-of-range + integer values. * ``bytes.byte()`` is defined above as accepting length 1 binary sequences as individual bytes, but this is currently inconsistent with the main ``bytes`` constructor:: @@ -164,6 +166,7 @@ bytes objects in addition to iterables of integers? If so, should it allow a mixture of the two in a single iterable? + Iteration --------- @@ -180,8 +183,8 @@ # 0 to 255 inclusive -Consistently accepting integer inputs to methods ------------------------------------------------- +Consistent support for different input types +-------------------------------------------- In Python 3.3, the binary search operations (``in``, ``count()``, ``find()``, ``index()``, ``rfind()`` and ``rindex()``) were updated to @@ -191,7 +194,9 @@ This PEP proposes extending that behaviour of accepting integers as being equivalent to the corresponding length 1 binary sequence to several other ``bytes`` and ``bytearray`` methods that currently expect a ``bytes`` -object for certain parameters. +object for certain parameters. In essence, if a value is an acceptable +input to the new ``bytes.byte`` constructor defined above, then it would +be acceptable in the roles defined here: * ``startswith()`` prefix(es) * ``endswith()`` suffix(es) @@ -200,9 +205,9 @@ * ``ljust()`` fill character * ``rjust()`` fill character -* ``strip()`` characters to strip -* ``lstrip()`` characters to strip -* ``rstrip()`` characters to strip +* ``strip()`` character to strip +* ``lstrip()`` character to strip +* ``rstrip()`` character to strip * ``partition()`` separator argument * ``rpartition()`` separator argument @@ -216,13 +221,22 @@ to work with the indexing behaviour , as the result of an indexing operation can more easily be fed back in to other methods. +For ``bytearray``, some additional changes are proposed to the current +integer based operations to ensure they remain consistent with the proposed +constructor changes:: + +* ``append()``: updated to be consistent with ``bytes.byte()`` +* ``remove()``: updated to be consistent with ``bytes.byte()`` +* ``+=``: updated to be consistent with ``bytes()`` changes (if any) +* ``extend()``: updated to be consistent with ``bytes()`` changes (if any) + Acknowledgement of surprising behaviour of some ``bytearray`` methods --------------------------------------------------------------------- Several of the ``bytes`` and ``bytearray`` methods have their origins in the Python 2 ``str`` API. As ``str`` is an immutable type, all of these -operations are defined as returning a *new* instances, rather than operating +operations are defined as returning a *new* instance, rather than operating in place. This contrasts with methods on other mutable types like ``list``, where ``list.sort()`` and ``list.reverse()`` operate in-place and return ``None``, rather than creating a new object. @@ -253,10 +267,14 @@ they're part of the mutable sequence API in ``bytearray``, rather than being inspired by the immutable Python 2 ``str`` API: +* ``+=`` +* ``append()`` +* ``extend()`` * ``reverse()`` * ``remove()`` * ``pop()`` + References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 30 04:04:29 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 30 Mar 2014 04:04:29 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_467=3A_be_clear_no_input_?= =?utf-8?q?type_support_will_be_removed?= Message-ID: <3fxKqd1DMRz7LjV@mail.python.org> http://hg.python.org/peps/rev/4eb137668084 changeset: 5449:4eb137668084 user: Nick Coghlan date: Sun Mar 30 12:04:18 2014 +1000 summary: PEP 467: be clear no input type support will be removed files: pep-0467.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0467.txt b/pep-0467.txt --- a/pep-0467.txt +++ b/pep-0467.txt @@ -196,7 +196,8 @@ ``bytes`` and ``bytearray`` methods that currently expect a ``bytes`` object for certain parameters. In essence, if a value is an acceptable input to the new ``bytes.byte`` constructor defined above, then it would -be acceptable in the roles defined here: +be acceptable in the roles defined here (in addition to any other already +supported inputs): * ``startswith()`` prefix(es) * ``endswith()`` suffix(es) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 30 06:01:58 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sun, 30 Mar 2014 06:01:58 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDI5?= =?utf-8?q?=3A_IDLE_now_colors_print_consistently_as_a_keyword=2E?= Message-ID: <3fxNRB0gRnz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/6f87f50ecab7 changeset: 90035:6f87f50ecab7 branch: 2.7 parent: 90029:3c577e68859e user: Raymond Hettinger date: Sat Mar 29 21:01:50 2014 -0700 summary: Issue #21029: IDLE now colors print consistently as a keyword. files: Lib/idlelib/ColorDelegator.py | 3 +++ Misc/NEWS | 4 ++++ 2 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/ColorDelegator.py b/Lib/idlelib/ColorDelegator.py --- a/Lib/idlelib/ColorDelegator.py +++ b/Lib/idlelib/ColorDelegator.py @@ -16,6 +16,9 @@ kw = r"\b" + any("KEYWORD", keyword.kwlist) + r"\b" builtinlist = [str(name) for name in dir(__builtin__) if not name.startswith('_')] + # We don't know whether "print" is a function or a keyword, + # so we always treat is as a keyword (the most common case). + builtinlist.remove('print') # self.file = file("file") : # 1st 'file' colorized normal, 2nd as builtin, 3rd as string builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -265,6 +265,10 @@ - Issue #20406: Use Python application icons for Idle window title bars. Patch mostly by Serhiy Storchaka. +- Issue #21029: Occurrences of "print" are now consistently colored as + being a keyword (the colorizer doesn't know if print functions are + enabled in the source). + - Issue #17721: Remove non-functional configuration dialog help button until we make it actually gives some help when clicked. Patch by Guilherme Sim?es. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 06:14:05 2014 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 30 Mar 2014 06:14:05 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_doc_typo=2E?= Message-ID: <3fxNj95VbBz7LjV@mail.python.org> http://hg.python.org/cpython/rev/c5f94552bcfc changeset: 90036:c5f94552bcfc parent: 90034:a8bb571b41ab user: Guido van Rossum date: Sat Mar 29 21:14:04 2014 -0700 summary: Fix doc typo. files: Doc/library/asyncio-protocol.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -9,7 +9,7 @@ Transports ========== -Transports are classed provided by :mod:`asyncio` in order to abstract +Transports are classes provided by :mod:`asyncio` in order to abstract various kinds of communication channels. You generally won't instantiate a transport yourself; instead, you will call a :class:`BaseEventLoop` method which will create the transport and try to initiate the underlying -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 06:26:23 2014 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 30 Mar 2014 06:26:23 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?peps=3A_Truncate_an_interminable_sent?= =?utf-8?b?ZW5jZSAoc29ycnkgTmljayA6LSku?= Message-ID: <3fxNzM2Cqsz7LjV@mail.python.org> http://hg.python.org/peps/rev/4c788239955c changeset: 5450:4c788239955c user: Guido van Rossum date: Sat Mar 29 21:26:23 2014 -0700 summary: Truncate an interminable sentence (sorry Nick :-). files: pep-0467.txt | 6 +----- 1 files changed, 1 insertions(+), 5 deletions(-) diff --git a/pep-0467.txt b/pep-0467.txt --- a/pep-0467.txt +++ b/pep-0467.txt @@ -22,11 +22,7 @@ This PEP proposes a number of small adjustments to the APIs of the ``bytes`` and ``bytearray`` types to make their behaviour more internally consistent -and to make it easier to operate entirely in the binary domain for use cases -that actually involve manipulating binary data directly, rather than -converting it to a more structured form with additional modelling -semantics (such as ``str``) and then converting back to binary format after -processing. +and to make it easier to operate entirely in the binary domain. Background -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Mar 30 08:55:11 2014 From: python-checkins at python.org (ned.deily) Date: Sun, 30 Mar 2014 08:55:11 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320022=3A_Eliminat?= =?utf-8?q?e_use_of_deprecated_bundlebuilder_in_OS_X_builds=2E?= Message-ID: <3fxSH34XVQz7Llr@mail.python.org> http://hg.python.org/cpython/rev/3cf72994d5ae changeset: 90037:3cf72994d5ae user: Ned Deily date: Sat Mar 29 23:54:15 2014 -0700 summary: Issue #20022: Eliminate use of deprecated bundlebuilder in OS X builds. files: Mac/PythonLauncher/Info.plist.in | 4 +- Mac/PythonLauncher/Makefile.in | 30 +- Mac/Tools/bundlebuilder.py | 934 ------------------- Misc/NEWS | 1 + 4 files changed, 16 insertions(+), 953 deletions(-) diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -1,5 +1,5 @@ - + CFBundleDevelopmentRegion @@ -38,7 +38,7 @@ CFBundleExecutable - PythonLauncher + Python Launcher CFBundleGetInfoString %VERSION%, ? 2001-2014 Python Software Foundation CFBundleIconFile diff --git a/Mac/PythonLauncher/Makefile.in b/Mac/PythonLauncher/Makefile.in --- a/Mac/PythonLauncher/Makefile.in +++ b/Mac/PythonLauncher/Makefile.in @@ -15,12 +15,10 @@ PYTHONFRAMEWORK=@PYTHONFRAMEWORK@ # Deployment target selected during configure, to be checked -# by distutils +# by distutils MACOSX_DEPLOYMENT_TARGET=@CONFIGURE_MACOSX_DEPLOYMENT_TARGET@ @EXPORT_MACOSX_DEPLOYMENT_TARGET at export MACOSX_DEPLOYMENT_TARGET -BUNDLEBULDER=$(srcdir)/../Tools/bundlebuilder.py - PYTHONAPPSDIR=@FRAMEWORKINSTALLAPPSPREFIX@/$(PYTHONFRAMEWORK) $(VERSION) OBJECTS=FileSettings.o MyAppDelegate.o MyDocument.o PreferencesWindowController.o doscript.o main.o @@ -30,10 +28,10 @@ /bin/cp -r "Python Launcher.app" "$(DESTDIR)$(PYTHONAPPSDIR)" touch "$(DESTDIR)$(PYTHONAPPSDIR)/Python Launcher.app" - clean: rm -f *.o "Python Launcher" rm -rf "Python Launcher.app" + rm -f Info.plist Python\ Launcher.app: Info.plist \ Python\ Launcher $(srcdir)/../Icons/PythonLauncher.icns \ @@ -41,20 +39,18 @@ $(srcdir)/../Icons/PythonCompiled.icns \ $(srcdir)/factorySettings.plist rm -fr "Python Launcher.app" - $(RUNSHARED) $(BUILDPYTHON) $(BUNDLEBULDER) \ - --builddir=. \ - --name="Python Launcher" \ - --executable="Python Launcher" \ - --iconfile=$(srcdir)/../Icons/PythonLauncher.icns \ - --bundle-id=org.python.PythonLauncher \ - --resource=$(srcdir)/../Icons/PythonSource.icns \ - --resource=$(srcdir)/../Icons/PythonCompiled.icns \ - --resource=$(srcdir)/English.lproj \ - --resource=$(srcdir)/factorySettings.plist \ - --plist Info.plist \ - build + mkdir "Python Launcher.app" + mkdir "Python Launcher.app/Contents" + mkdir "Python Launcher.app/Contents/MacOS" + mkdir "Python Launcher.app/Contents/Resources" + cp "Python Launcher" "Python Launcher.app/Contents/MacOS" + cp Info.plist "Python Launcher.app/Contents" + cp $(srcdir)/../Icons/PythonLauncher.icns "Python Launcher.app/Contents/Resources" + cp $(srcdir)/../Icons/PythonSource.icns "Python Launcher.app/Contents/Resources" + cp $(srcdir)/../Icons/PythonCompiled.icns "Python Launcher.app/Contents/Resources" + cp $(srcdir)/factorySettings.plist "Python Launcher.app/Contents/Resources" + cp -R $(srcdir)/English.lproj "Python Launcher.app/Contents/Resources" find "Python Launcher.app" -name '.svn' -print0 | xargs -0 rm -r - FileSettings.o: $(srcdir)/FileSettings.m $(CC) $(CFLAGS) -o $@ -c $(srcdir)/FileSettings.m diff --git a/Mac/Tools/bundlebuilder.py b/Mac/Tools/bundlebuilder.py deleted file mode 100755 --- a/Mac/Tools/bundlebuilder.py +++ /dev/null @@ -1,934 +0,0 @@ -#! /usr/bin/env python - -"""\ -bundlebuilder.py -- Tools to assemble MacOS X (application) bundles. - -This module contains two classes to build so called "bundles" for -MacOS X. BundleBuilder is a general tool, AppBuilder is a subclass -specialized in building application bundles. - -[Bundle|App]Builder objects are instantiated with a bunch of keyword -arguments, and have a build() method that will do all the work. See -the class doc strings for a description of the constructor arguments. - -The module contains a main program that can be used in two ways: - - % python bundlebuilder.py [options] build - % python buildapp.py [options] build - -Where "buildapp.py" is a user-supplied setup.py-like script following -this model: - - from bundlebuilder import buildapp - buildapp() - -""" - - -__all__ = ["BundleBuilder", "BundleBuilderError", "AppBuilder", "buildapp"] - - -import sys -import os, errno, shutil -import imp, marshal -import re -from copy import deepcopy -import getopt -from plistlib import Plist -from types import FunctionType as function - -class BundleBuilderError(Exception): pass - - -class Defaults: - - """Class attributes that don't start with an underscore and are - not functions or classmethods are (deep)copied to self.__dict__. - This allows for mutable default values. - """ - - def __init__(self, **kwargs): - defaults = self._getDefaults() - defaults.update(kwargs) - self.__dict__.update(defaults) - - def _getDefaults(cls): - defaults = {} - for base in cls.__bases__: - if hasattr(base, "_getDefaults"): - defaults.update(base._getDefaults()) - for name, value in list(cls.__dict__.items()): - if name[0] != "_" and not isinstance(value, - (function, classmethod)): - defaults[name] = deepcopy(value) - return defaults - _getDefaults = classmethod(_getDefaults) - - -class BundleBuilder(Defaults): - - """BundleBuilder is a barebones class for assembling bundles. It - knows nothing about executables or icons, it only copies files - and creates the PkgInfo and Info.plist files. - """ - - # (Note that Defaults.__init__ (deep)copies these values to - # instance variables. Mutable defaults are therefore safe.) - - # Name of the bundle, with or without extension. - name = None - - # The property list ("plist") - plist = Plist(CFBundleDevelopmentRegion = "English", - CFBundleInfoDictionaryVersion = "6.0") - - # The type of the bundle. - type = "BNDL" - # The creator code of the bundle. - creator = None - - # the CFBundleIdentifier (this is used for the preferences file name) - bundle_id = None - - # List of files that have to be copied to /Contents/Resources. - resources = [] - - # List of (src, dest) tuples; dest should be a path relative to the bundle - # (eg. "Contents/Resources/MyStuff/SomeFile.ext). - files = [] - - # List of shared libraries (dylibs, Frameworks) to bundle with the app - # will be placed in Contents/Frameworks - libs = [] - - # Directory where the bundle will be assembled. - builddir = "build" - - # Make symlinks instead copying files. This is handy during debugging, but - # makes the bundle non-distributable. - symlink = 0 - - # Verbosity level. - verbosity = 1 - - # Destination root directory - destroot = "" - - def setup(self): - # XXX rethink self.name munging, this is brittle. - self.name, ext = os.path.splitext(self.name) - if not ext: - ext = ".bundle" - bundleextension = ext - # misc (derived) attributes - self.bundlepath = pathjoin(self.builddir, self.name + bundleextension) - - plist = self.plist - plist.CFBundleName = self.name - plist.CFBundlePackageType = self.type - if self.creator is None: - if hasattr(plist, "CFBundleSignature"): - self.creator = plist.CFBundleSignature - else: - self.creator = "????" - plist.CFBundleSignature = self.creator - if self.bundle_id: - plist.CFBundleIdentifier = self.bundle_id - elif not hasattr(plist, "CFBundleIdentifier"): - plist.CFBundleIdentifier = self.name - - def build(self): - """Build the bundle.""" - builddir = self.builddir - if builddir and not os.path.exists(builddir): - os.mkdir(builddir) - self.message("Building %s" % repr(self.bundlepath), 1) - if os.path.exists(self.bundlepath): - shutil.rmtree(self.bundlepath) - if os.path.exists(self.bundlepath + '~'): - shutil.rmtree(self.bundlepath + '~') - bp = self.bundlepath - - # Create the app bundle in a temporary location and then - # rename the completed bundle. This way the Finder will - # never see an incomplete bundle (where it might pick up - # and cache the wrong meta data) - self.bundlepath = bp + '~' - try: - os.mkdir(self.bundlepath) - self.preProcess() - self._copyFiles() - self._addMetaFiles() - self.postProcess() - os.rename(self.bundlepath, bp) - finally: - self.bundlepath = bp - self.message("Done.", 1) - - def preProcess(self): - """Hook for subclasses.""" - pass - def postProcess(self): - """Hook for subclasses.""" - pass - - def _addMetaFiles(self): - contents = pathjoin(self.bundlepath, "Contents") - makedirs(contents) - # - # Write Contents/PkgInfo - assert len(self.type) == len(self.creator) == 4, \ - "type and creator must be 4-byte strings." - pkginfo = pathjoin(contents, "PkgInfo") - f = open(pkginfo, "wb") - f.write((self.type + self.creator).encode('latin1')) - f.close() - # - # Write Contents/Info.plist - infoplist = pathjoin(contents, "Info.plist") - self.plist.write(infoplist) - - def _copyFiles(self): - files = self.files[:] - for path in self.resources: - files.append((path, pathjoin("Contents", "Resources", - os.path.basename(path)))) - for path in self.libs: - files.append((path, pathjoin("Contents", "Frameworks", - os.path.basename(path)))) - if self.symlink: - self.message("Making symbolic links", 1) - msg = "Making symlink from" - else: - self.message("Copying files", 1) - msg = "Copying" - files.sort() - for src, dst in files: - if os.path.isdir(src): - self.message("%s %s/ to %s/" % (msg, src, dst), 2) - else: - self.message("%s %s to %s" % (msg, src, dst), 2) - dst = pathjoin(self.bundlepath, dst) - if self.symlink: - symlink(src, dst, mkdirs=1) - else: - copy(src, dst, mkdirs=1) - - def message(self, msg, level=0): - if level <= self.verbosity: - indent = "" - if level > 1: - indent = (level - 1) * " " - sys.stderr.write(indent + msg + "\n") - - def report(self): - # XXX something decent - pass - - -if __debug__: - PYC_EXT = ".pyc" -else: - PYC_EXT = ".pyo" - -MAGIC = imp.get_magic() -USE_ZIPIMPORT = "zipimport" in sys.builtin_module_names - -# For standalone apps, we have our own minimal site.py. We don't need -# all the cruft of the real site.py. -SITE_PY = """\ -import sys -if not %(semi_standalone)s: - del sys.path[1:] # sys.path[0] is Contents/Resources/ -""" - -if USE_ZIPIMPORT: - ZIP_ARCHIVE = "Modules.zip" - SITE_PY += "sys.path.append(sys.path[0] + '/%s')\n" % ZIP_ARCHIVE - def getPycData(fullname, code, ispkg): - if ispkg: - fullname += ".__init__" - path = fullname.replace(".", os.sep) + PYC_EXT - return path, MAGIC + '\0\0\0\0' + marshal.dumps(code) - -# -# Extension modules can't be in the modules zip archive, so a placeholder -# is added instead, that loads the extension from a specified location. -# -EXT_LOADER = """\ -def __load(): - import imp, sys, os - for p in sys.path: - path = os.path.join(p, "%(filename)s") - if os.path.exists(path): - break - else: - assert 0, "file not found: %(filename)s" - mod = imp.load_dynamic("%(name)s", path) - -__load() -del __load -""" - -MAYMISS_MODULES = ['mac', 'nt', 'ntpath', 'dos', 'dospath', - 'win32api', 'ce', '_winreg', 'nturl2path', 'sitecustomize', - 'org.python.core', 'riscos', 'riscosenviron', 'riscospath' -] - -STRIP_EXEC = "/usr/bin/strip" - -# -# We're using a stock interpreter to run the app, yet we need -# a way to pass the Python main program to the interpreter. The -# bootstrapping script fires up the interpreter with the right -# arguments. os.execve() is used as OSX doesn't like us to -# start a real new process. Also, the executable name must match -# the CFBundleExecutable value in the Info.plist, so we lie -# deliberately with argv[0]. The actual Python executable is -# passed in an environment variable so we can "repair" -# sys.executable later. -# -BOOTSTRAP_SCRIPT = """\ -#!%(hashbang)s - -import sys, os -execdir = os.path.dirname(sys.argv[0]) -executable = os.path.join(execdir, "%(executable)s") -resdir = os.path.join(os.path.dirname(execdir), "Resources") -libdir = os.path.join(os.path.dirname(execdir), "Frameworks") -mainprogram = os.path.join(resdir, "%(mainprogram)s") - -sys.argv.insert(1, mainprogram) -if %(standalone)s or %(semi_standalone)s: - os.environ["PYTHONPATH"] = resdir - if %(standalone)s: - os.environ["PYTHONHOME"] = resdir -else: - pypath = os.getenv("PYTHONPATH", "") - if pypath: - pypath = ":" + pypath - os.environ["PYTHONPATH"] = resdir + pypath -os.environ["PYTHONEXECUTABLE"] = executable -os.environ["DYLD_LIBRARY_PATH"] = libdir -os.environ["DYLD_FRAMEWORK_PATH"] = libdir -os.execve(executable, sys.argv, os.environ) -""" - - -# -# Optional wrapper that converts "dropped files" into sys.argv values. -# -ARGV_EMULATOR = """\ -import argvemulator, os - -argvemulator.ArgvCollector().mainloop() -execfile(os.path.join(os.path.split(__file__)[0], "%(realmainprogram)s")) -""" - -# -# When building a standalone app with Python.framework, we need to copy -# a subset from Python.framework to the bundle. The following list -# specifies exactly what items we'll copy. -# -PYTHONFRAMEWORKGOODIES = [ - "Python", # the Python core library - "Resources/English.lproj", - "Resources/Info.plist", - "Resources/version.plist", -] - -def isFramework(): - return sys.exec_prefix.find("Python.framework") > 0 - - -LIB = os.path.join(sys.prefix, "lib", "python" + sys.version[:3]) -SITE_PACKAGES = os.path.join(LIB, "site-packages") - - -class AppBuilder(BundleBuilder): - - # Override type of the bundle. - type = "APPL" - - # platform, name of the subfolder of Contents that contains the executable. - platform = "MacOS" - - # A Python main program. If this argument is given, the main - # executable in the bundle will be a small wrapper that invokes - # the main program. (XXX Discuss why.) - mainprogram = None - - # The main executable. If a Python main program is specified - # the executable will be copied to Resources and be invoked - # by the wrapper program mentioned above. Otherwise it will - # simply be used as the main executable. - executable = None - - # The name of the main nib, for Cocoa apps. *Must* be specified - # when building a Cocoa app. - nibname = None - - # The name of the icon file to be copied to Resources and used for - # the Finder icon. - iconfile = None - - # Symlink the executable instead of copying it. - symlink_exec = 0 - - # If True, build standalone app. - standalone = 0 - - # If True, build semi-standalone app (only includes third-party modules). - semi_standalone = 0 - - # If set, use this for #! lines in stead of sys.executable - python = None - - # If True, add a real main program that emulates sys.argv before calling - # mainprogram - argv_emulation = 0 - - # The following attributes are only used when building a standalone app. - - # Exclude these modules. - excludeModules = [] - - # Include these modules. - includeModules = [] - - # Include these packages. - includePackages = [] - - # Strip binaries from debug info. - strip = 0 - - # Found Python modules: [(name, codeobject, ispkg), ...] - pymodules = [] - - # Modules that modulefinder couldn't find: - missingModules = [] - maybeMissingModules = [] - - def setup(self): - if ((self.standalone or self.semi_standalone) - and self.mainprogram is None): - raise BundleBuilderError("must specify 'mainprogram' when " - "building a standalone application.") - if self.mainprogram is None and self.executable is None: - raise BundleBuilderError("must specify either or both of " - "'executable' and 'mainprogram'") - - self.execdir = pathjoin("Contents", self.platform) - - if self.name is not None: - pass - elif self.mainprogram is not None: - self.name = os.path.splitext(os.path.basename(self.mainprogram))[0] - elif executable is not None: - self.name = os.path.splitext(os.path.basename(self.executable))[0] - if self.name[-4:] != ".app": - self.name += ".app" - - if self.executable is None: - if not self.standalone and not isFramework(): - self.symlink_exec = 1 - if self.python: - self.executable = self.python - else: - self.executable = sys.executable - - if self.nibname: - self.plist.NSMainNibFile = self.nibname - if not hasattr(self.plist, "NSPrincipalClass"): - self.plist.NSPrincipalClass = "NSApplication" - - if self.standalone and isFramework(): - self.addPythonFramework() - - BundleBuilder.setup(self) - - self.plist.CFBundleExecutable = self.name - - if self.standalone or self.semi_standalone: - self.findDependencies() - - def preProcess(self): - resdir = "Contents/Resources" - if self.executable is not None: - if self.mainprogram is None: - execname = self.name - else: - execname = os.path.basename(self.executable) - execpath = pathjoin(self.execdir, execname) - if not self.symlink_exec: - self.files.append((self.destroot + self.executable, execpath)) - self.execpath = execpath - - if self.mainprogram is not None: - mainprogram = os.path.basename(self.mainprogram) - self.files.append((self.mainprogram, pathjoin(resdir, mainprogram))) - if self.argv_emulation: - # Change the main program, and create the helper main program (which - # does argv collection and then calls the real main). - # Also update the included modules (if we're creating a standalone - # program) and the plist - realmainprogram = mainprogram - mainprogram = '__argvemulator_' + mainprogram - resdirpath = pathjoin(self.bundlepath, resdir) - mainprogrampath = pathjoin(resdirpath, mainprogram) - makedirs(resdirpath) - open(mainprogrampath, "w").write(ARGV_EMULATOR % locals()) - if self.standalone or self.semi_standalone: - self.includeModules.append("argvemulator") - self.includeModules.append("os") - if "CFBundleDocumentTypes" not in self.plist: - self.plist["CFBundleDocumentTypes"] = [ - { "CFBundleTypeOSTypes" : [ - "****", - "fold", - "disk"], - "CFBundleTypeRole": "Viewer"}] - # Write bootstrap script - executable = os.path.basename(self.executable) - execdir = pathjoin(self.bundlepath, self.execdir) - bootstrappath = pathjoin(execdir, self.name) - makedirs(execdir) - if self.standalone or self.semi_standalone: - # XXX we're screwed when the end user has deleted - # /usr/bin/python - hashbang = "/usr/bin/python" - elif self.python: - hashbang = self.python - else: - hashbang = os.path.realpath(sys.executable) - standalone = self.standalone - semi_standalone = self.semi_standalone - open(bootstrappath, "w").write(BOOTSTRAP_SCRIPT % locals()) - os.chmod(bootstrappath, 0o775) - - if self.iconfile is not None: - iconbase = os.path.basename(self.iconfile) - self.plist.CFBundleIconFile = iconbase - self.files.append((self.iconfile, pathjoin(resdir, iconbase))) - - def postProcess(self): - if self.standalone or self.semi_standalone: - self.addPythonModules() - if self.strip and not self.symlink: - self.stripBinaries() - - if self.symlink_exec and self.executable: - self.message("Symlinking executable %s to %s" % (self.executable, - self.execpath), 2) - dst = pathjoin(self.bundlepath, self.execpath) - makedirs(os.path.dirname(dst)) - os.symlink(os.path.abspath(self.executable), dst) - - if self.missingModules or self.maybeMissingModules: - self.reportMissing() - - def addPythonFramework(self): - # If we're building a standalone app with Python.framework, - # include a minimal subset of Python.framework, *unless* - # Python.framework was specified manually in self.libs. - for lib in self.libs: - if os.path.basename(lib) == "Python.framework": - # a Python.framework was specified as a library - return - - frameworkpath = sys.exec_prefix[:sys.exec_prefix.find( - "Python.framework") + len("Python.framework")] - - version = sys.version[:3] - frameworkpath = pathjoin(frameworkpath, "Versions", version) - destbase = pathjoin("Contents", "Frameworks", "Python.framework", - "Versions", version) - for item in PYTHONFRAMEWORKGOODIES: - src = pathjoin(frameworkpath, item) - dst = pathjoin(destbase, item) - self.files.append((src, dst)) - - def _getSiteCode(self): - return compile(SITE_PY % {"semi_standalone": self.semi_standalone}, - "<-bundlebuilder.py->", "exec") - - def addPythonModules(self): - self.message("Adding Python modules", 1) - - if USE_ZIPIMPORT: - # Create a zip file containing all modules as pyc. - import zipfile - relpath = pathjoin("Contents", "Resources", ZIP_ARCHIVE) - abspath = pathjoin(self.bundlepath, relpath) - zf = zipfile.ZipFile(abspath, "w", zipfile.ZIP_DEFLATED) - for name, code, ispkg in self.pymodules: - self.message("Adding Python module %s" % name, 2) - path, pyc = getPycData(name, code, ispkg) - zf.writestr(path, pyc) - zf.close() - # add site.pyc - sitepath = pathjoin(self.bundlepath, "Contents", "Resources", - "site" + PYC_EXT) - writePyc(self._getSiteCode(), sitepath) - else: - # Create individual .pyc files. - for name, code, ispkg in self.pymodules: - if ispkg: - name += ".__init__" - path = name.split(".") - path = pathjoin("Contents", "Resources", *path) + PYC_EXT - - if ispkg: - self.message("Adding Python package %s" % path, 2) - else: - self.message("Adding Python module %s" % path, 2) - - abspath = pathjoin(self.bundlepath, path) - makedirs(os.path.dirname(abspath)) - writePyc(code, abspath) - - def stripBinaries(self): - if not os.path.exists(STRIP_EXEC): - self.message("Error: can't strip binaries: no strip program at " - "%s" % STRIP_EXEC, 0) - else: - import stat - self.message("Stripping binaries", 1) - def walk(top): - for name in os.listdir(top): - path = pathjoin(top, name) - if os.path.islink(path): - continue - if os.path.isdir(path): - walk(path) - else: - mod = os.stat(path)[stat.ST_MODE] - if not (mod & 0o100): - continue - relpath = path[len(self.bundlepath):] - self.message("Stripping %s" % relpath, 2) - inf, outf = os.popen4("%s -S \"%s\"" % - (STRIP_EXEC, path)) - output = outf.read().strip() - if output: - # usually not a real problem, like when we're - # trying to strip a script - self.message("Problem stripping %s:" % relpath, 3) - self.message(output, 3) - walk(self.bundlepath) - - def findDependencies(self): - self.message("Finding module dependencies", 1) - import modulefinder - mf = modulefinder.ModuleFinder(excludes=self.excludeModules) - if USE_ZIPIMPORT: - # zipimport imports zlib, must add it manually - mf.import_hook("zlib") - # manually add our own site.py - site = mf.add_module("site") - site.__code__ = self._getSiteCode() - mf.scan_code(site.__code__, site) - - # warnings.py gets imported implicitly from C - mf.import_hook("warnings") - - includeModules = self.includeModules[:] - for name in self.includePackages: - includeModules.extend(list(findPackageContents(name).keys())) - for name in includeModules: - try: - mf.import_hook(name) - except ImportError: - self.missingModules.append(name) - - mf.run_script(self.mainprogram) - modules = list(mf.modules.items()) - modules.sort() - for name, mod in modules: - path = mod.__file__ - if path and self.semi_standalone: - # skip the standard library - if path.startswith(LIB) and not path.startswith(SITE_PACKAGES): - continue - if path and mod.__code__ is None: - # C extension - filename = os.path.basename(path) - pathitems = name.split(".")[:-1] + [filename] - dstpath = pathjoin(*pathitems) - if USE_ZIPIMPORT: - if name != "zlib": - # neatly pack all extension modules in a subdirectory, - # except zlib, since it's necessary for bootstrapping. - dstpath = pathjoin("ExtensionModules", dstpath) - # Python modules are stored in a Zip archive, but put - # extensions in Contents/Resources/. Add a tiny "loader" - # program in the Zip archive. Due to Thomas Heller. - source = EXT_LOADER % {"name": name, "filename": dstpath} - code = compile(source, "" % name, "exec") - mod.__code__ = code - self.files.append((path, pathjoin("Contents", "Resources", dstpath))) - if mod.__code__ is not None: - ispkg = mod.__path__ is not None - if not USE_ZIPIMPORT or name != "site": - # Our site.py is doing the bootstrapping, so we must - # include a real .pyc file if USE_ZIPIMPORT is True. - self.pymodules.append((name, mod.__code__, ispkg)) - - if hasattr(mf, "any_missing_maybe"): - missing, maybe = mf.any_missing_maybe() - else: - missing = mf.any_missing() - maybe = [] - self.missingModules.extend(missing) - self.maybeMissingModules.extend(maybe) - - def reportMissing(self): - missing = [name for name in self.missingModules - if name not in MAYMISS_MODULES] - if self.maybeMissingModules: - maybe = self.maybeMissingModules - else: - maybe = [name for name in missing if "." in name] - missing = [name for name in missing if "." not in name] - missing.sort() - maybe.sort() - if maybe: - self.message("Warning: couldn't find the following submodules:", 1) - self.message(" (Note that these could be false alarms -- " - "it's not always", 1) - self.message(" possible to distinguish between \"from package " - "import submodule\" ", 1) - self.message(" and \"from package import name\")", 1) - for name in maybe: - self.message(" ? " + name, 1) - if missing: - self.message("Warning: couldn't find the following modules:", 1) - for name in missing: - self.message(" ? " + name, 1) - - def report(self): - # XXX something decent - import pprint - pprint.pprint(self.__dict__) - if self.standalone or self.semi_standalone: - self.reportMissing() - -# -# Utilities. -# - -SUFFIXES = [_suf for _suf, _mode, _tp in imp.get_suffixes()] -identifierRE = re.compile(r"[_a-zA-z][_a-zA-Z0-9]*$") - -def findPackageContents(name, searchpath=None): - head = name.split(".")[-1] - if identifierRE.match(head) is None: - return {} - try: - fp, path, (ext, mode, tp) = imp.find_module(head, searchpath) - except ImportError: - return {} - modules = {name: None} - if tp == imp.PKG_DIRECTORY and path: - files = os.listdir(path) - for sub in files: - sub, ext = os.path.splitext(sub) - fullname = name + "." + sub - if sub != "__init__" and fullname not in modules: - modules.update(findPackageContents(fullname, [path])) - return modules - -def writePyc(code, path): - f = open(path, "wb") - f.write(MAGIC) - f.write("\0" * 4) # don't bother about a time stamp - marshal.dump(code, f) - f.close() - -def copy(src, dst, mkdirs=0): - """Copy a file or a directory.""" - if mkdirs: - makedirs(os.path.dirname(dst)) - if os.path.isdir(src): - shutil.copytree(src, dst, symlinks=1) - else: - shutil.copy2(src, dst) - -def copytodir(src, dstdir): - """Copy a file or a directory to an existing directory.""" - dst = pathjoin(dstdir, os.path.basename(src)) - copy(src, dst) - -def makedirs(dir): - """Make all directories leading up to 'dir' including the leaf - directory. Don't moan if any path element already exists.""" - try: - os.makedirs(dir) - except OSError as why: - if why.errno != errno.EEXIST: - raise - -def symlink(src, dst, mkdirs=0): - """Copy a file or a directory.""" - if not os.path.exists(src): - raise IOError("No such file or directory: '%s'" % src) - if mkdirs: - makedirs(os.path.dirname(dst)) - os.symlink(os.path.abspath(src), dst) - -def pathjoin(*args): - """Safe wrapper for os.path.join: asserts that all but the first - argument are relative paths.""" - for seg in args[1:]: - assert seg[0] != "/" - return os.path.join(*args) - - -cmdline_doc = """\ -Usage: - python bundlebuilder.py [options] command - python mybuildscript.py [options] command - -Commands: - build build the application - report print a report - -Options: - -b, --builddir=DIR the build directory; defaults to "build" - -n, --name=NAME application name - -r, --resource=FILE extra file or folder to be copied to Resources - -f, --file=SRC:DST extra file or folder to be copied into the bundle; - DST must be a path relative to the bundle root - -e, --executable=FILE the executable to be used - -m, --mainprogram=FILE the Python main program - -a, --argv add a wrapper main program to create sys.argv - -p, --plist=FILE .plist file (default: generate one) - --nib=NAME main nib name - -c, --creator=CCCC 4-char creator code (default: '????') - --iconfile=FILE filename of the icon (an .icns file) to be used - as the Finder icon - --bundle-id=ID the CFBundleIdentifier, in reverse-dns format - (eg. org.python.BuildApplet; this is used for - the preferences file name) - -l, --link symlink files/folder instead of copying them - --link-exec symlink the executable instead of copying it - --standalone build a standalone application, which is fully - independent of a Python installation - --semi-standalone build a standalone application, which depends on - an installed Python, yet includes all third-party - modules. - --python=FILE Python to use in #! line in stead of current Python - --lib=FILE shared library or framework to be copied into - the bundle - -x, --exclude=MODULE exclude module (with --(semi-)standalone) - -i, --include=MODULE include module (with --(semi-)standalone) - --package=PACKAGE include a whole package (with --(semi-)standalone) - --strip strip binaries (remove debug info) - -v, --verbose increase verbosity level - -q, --quiet decrease verbosity level - -h, --help print this message -""" - -def usage(msg=None): - if msg: - print(msg) - print(cmdline_doc) - sys.exit(1) - -def main(builder=None): - if builder is None: - builder = AppBuilder(verbosity=1) - - shortopts = "b:n:r:f:e:m:c:p:lx:i:hvqa" - longopts = ("builddir=", "name=", "resource=", "file=", "executable=", - "mainprogram=", "creator=", "nib=", "plist=", "link", - "link-exec", "help", "verbose", "quiet", "argv", "standalone", - "exclude=", "include=", "package=", "strip", "iconfile=", - "lib=", "python=", "semi-standalone", "bundle-id=", "destroot=") - - try: - options, args = getopt.getopt(sys.argv[1:], shortopts, longopts) - except getopt.error: - usage() - - for opt, arg in options: - if opt in ('-b', '--builddir'): - builder.builddir = arg - elif opt in ('-n', '--name'): - builder.name = arg - elif opt in ('-r', '--resource'): - builder.resources.append(os.path.normpath(arg)) - elif opt in ('-f', '--file'): - srcdst = arg.split(':') - if len(srcdst) != 2: - usage("-f or --file argument must be two paths, " - "separated by a colon") - builder.files.append(srcdst) - elif opt in ('-e', '--executable'): - builder.executable = arg - elif opt in ('-m', '--mainprogram'): - builder.mainprogram = arg - elif opt in ('-a', '--argv'): - builder.argv_emulation = 1 - elif opt in ('-c', '--creator'): - builder.creator = arg - elif opt == '--bundle-id': - builder.bundle_id = arg - elif opt == '--iconfile': - builder.iconfile = arg - elif opt == "--lib": - builder.libs.append(os.path.normpath(arg)) - elif opt == "--nib": - builder.nibname = arg - elif opt in ('-p', '--plist'): - builder.plist = Plist.fromFile(arg) - elif opt in ('-l', '--link'): - builder.symlink = 1 - elif opt == '--link-exec': - builder.symlink_exec = 1 - elif opt in ('-h', '--help'): - usage() - elif opt in ('-v', '--verbose'): - builder.verbosity += 1 - elif opt in ('-q', '--quiet'): - builder.verbosity -= 1 - elif opt == '--standalone': - builder.standalone = 1 - elif opt == '--semi-standalone': - builder.semi_standalone = 1 - elif opt == '--python': - builder.python = arg - elif opt in ('-x', '--exclude'): - builder.excludeModules.append(arg) - elif opt in ('-i', '--include'): - builder.includeModules.append(arg) - elif opt == '--package': - builder.includePackages.append(arg) - elif opt == '--strip': - builder.strip = 1 - elif opt == '--destroot': - builder.destroot = arg - - if len(args) != 1: - usage("Must specify one command ('build', 'report' or 'help')") - command = args[0] - - if command == "build": - builder.setup() - builder.build() - elif command == "report": - builder.setup() - builder.report() - elif command == "help": - usage() - else: - usage("Unknown command '%s'" % command) - - -def buildapp(**kwargs): - builder = AppBuilder(**kwargs) - main(builder) - - -if __name__ == "__main__": - main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -176,6 +176,7 @@ - Issue #15968: Incorporated Tcl, Tk, and Tix builds into the Windows build solution. +- Issue #20022: Eliminate use of deprecated bundlebuilder in OS X builds. What's New in Python 3.4.0? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 09:03:55 2014 From: python-checkins at python.org (nick.coghlan) Date: Sun, 30 Mar 2014 09:03:55 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_467=3A_more_concrete_exam?= =?utf-8?q?ples_of_current_inconsistencies?= Message-ID: <3fxST76LmSz7LlP@mail.python.org> http://hg.python.org/peps/rev/9b80a489c6e4 changeset: 5451:9b80a489c6e4 user: Nick Coghlan date: Sun Mar 30 17:03:44 2014 +1000 summary: PEP 467: more concrete examples of current inconsistencies files: pep-0467.txt | 48 +++++++++++++++++++++++++++++++++++++-- 1 files changed, 45 insertions(+), 3 deletions(-) diff --git a/pep-0467.txt b/pep-0467.txt --- a/pep-0467.txt +++ b/pep-0467.txt @@ -182,12 +182,49 @@ Consistent support for different input types -------------------------------------------- +The ``bytes`` and ``bytearray`` methods inspired by the Python 2 ``str`` +type generally expect to operate on binary subsequences: other objects +implementing the buffer API. By contrast, the mutating APIs added to +the ``bytearray`` interface expect to operate on individual elements: +integer in the range 0 to 255 (inclusive). + In Python 3.3, the binary search operations (``in``, ``count()``, ``find()``, ``index()``, ``rfind()`` and ``rindex()``) were updated to -accept integers in the range 0 to 255 (inclusive) as their first argument -(in addition to the existing support for binary sequences). +accept integers in the range 0 to 255 (inclusive) as their first argument, +in addition to the existing support for binary subsequences. -This PEP proposes extending that behaviour of accepting integers as being +This results in behaviour like the following in Python 3.3+:: + + >>> data = bytes([1, 2, 3, 4]) + >>> 3 in data + True + >>> b"\x03" in data + True + >>> data.count(3) + 1 + >>> data.count(b"\x03") + 1 + + >>> data.replace(3, 4) + Traceback (most recent call last): + File "", line 1, in + TypeError: expected bytes, bytearray or buffer compatible object + >>> data.replace(b"\x03", b"\x04") + b'\x01\x02\x04\x04' + + >>> mutable = bytearray(data) + >>> mutable + bytearray(b'\x01\x02\x03\x04') + >>> mutable.append(b"\x05") + Traceback (most recent call last): + File "", line 1, in + TypeError: an integer is required + >>> mutable.append(5) + >>> mutable + bytearray(b'\x01\x02\x03\x04\x05') + + +This PEP proposes extending the behaviour of accepting integers as being equivalent to the corresponding length 1 binary sequence to several other ``bytes`` and ``bytearray`` methods that currently expect a ``bytes`` object for certain parameters. In essence, if a value is an acceptable @@ -227,6 +264,11 @@ * ``+=``: updated to be consistent with ``bytes()`` changes (if any) * ``extend()``: updated to be consistent with ``bytes()`` changes (if any) +The general principle behind these changes is to restore the flexible +"element-or-subsequence" behaviour seen in the ``str`` API, even though +Python 3 actually represents subsequences and individual elements as +distinct types in the binary domain. + Acknowledgement of surprising behaviour of some ``bytearray`` methods --------------------------------------------------------------------- -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Sun Mar 30 08:57:32 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 30 Mar 2014 08:57:32 +0200 Subject: [Python-checkins] Daily reference leaks (a8bb571b41ab): sum=4 Message-ID: results for a8bb571b41ab on branch "default" -------------------------------------------- test_site leaked [0, 0, 2] references, sum=2 test_site leaked [0, 0, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogXRWF3M', '-x'] From python-checkins at python.org Sun Mar 30 16:32:44 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 16:32:44 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_doc_typo?= =?utf-8?q?=2E?= Message-ID: <3fxfR00Wxxz7LkD@mail.python.org> http://hg.python.org/cpython/rev/d5681d94356a changeset: 90038:d5681d94356a branch: 3.4 parent: 90033:48790b202a50 user: Guido van Rossum date: Sat Mar 29 21:14:04 2014 -0700 summary: Fix doc typo. files: Doc/library/asyncio-protocol.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -9,7 +9,7 @@ Transports ========== -Transports are classed provided by :mod:`asyncio` in order to abstract +Transports are classes provided by :mod:`asyncio` in order to abstract various kinds of communication channels. You generally won't instantiate a transport yourself; instead, you will call a :class:`BaseEventLoop` method which will create the transport and try to initiate the underlying -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 16:32:45 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 16:32:45 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fxfR11wL0z7LkD@mail.python.org> http://hg.python.org/cpython/rev/2d8822ddedae changeset: 90039:2d8822ddedae parent: 90037:3cf72994d5ae parent: 90038:d5681d94356a user: Benjamin Peterson date: Sun Mar 30 10:32:37 2014 -0400 summary: merge 3.4 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 19:12:17 2014 From: python-checkins at python.org (raymond.hettinger) Date: Sun, 30 Mar 2014 19:12:17 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_21100=3A_Amazingly?= =?utf-8?q?=2C_tuple_lexicographic_ordering_was_untested=2E?= Message-ID: <3fxjz510j7z7LkB@mail.python.org> http://hg.python.org/cpython/rev/16fd3f4c9128 changeset: 90040:16fd3f4c9128 user: Raymond Hettinger date: Sun Mar 30 10:12:09 2014 -0700 summary: Issue 21100: Amazingly, tuple lexicographic ordering was untested. files: Lib/test/test_tuple.py | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tuple.py b/Lib/test/test_tuple.py --- a/Lib/test/test_tuple.py +++ b/Lib/test/test_tuple.py @@ -201,6 +201,14 @@ with self.assertRaises(TypeError): [3,] + T((1,2)) + def test_lexicographic_ordering(self): + # Issue 21100 + a = self.type2test([1, 2]) + b = self.type2test([1, 2, 0]) + c = self.type2test([1, 3]) + self.assertLess(a, b) + self.assertLess(b, c) + def test_main(): support.run_unittest(TupleTest) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 20:19:11 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 20:19:11 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_test_case_for_freeze?= =?utf-8?q?=2E?= Message-ID: <3fxlSH33CMz7Lk1@mail.python.org> http://hg.python.org/cpython/rev/023b119f20eb changeset: 90041:023b119f20eb user: Martin v. L?wis date: Sun Mar 30 20:18:58 2014 +0200 summary: Add test case for freeze. files: Misc/NEWS | 2 ++ Tools/freeze/test/Makefile | 10 ++++++++++ Tools/freeze/test/ok.py | 2 ++ 3 files changed, 14 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -150,6 +150,8 @@ Tests ----- +- Add test case for freeze. + - Issue #20743: Fix a reference leak in test_tcl. - Issue #21097: Move test_namespace_pkgs into test_importlib. diff --git a/Tools/freeze/test/Makefile b/Tools/freeze/test/Makefile new file mode 100644 --- /dev/null +++ b/Tools/freeze/test/Makefile @@ -0,0 +1,10 @@ +# Makefile to test freeze +# set PYTHON to path of Python interpreter to test +PYTHON=python +# set OUTDIR to the temp directory for freeze +OUTDIR=outdir + +test: + $(PYTHON) ../freeze.py -o $(OUTDIR) ok.py + make -C $(OUTDIR) + diff --git a/Tools/freeze/test/ok.py b/Tools/freeze/test/ok.py new file mode 100644 --- /dev/null +++ b/Tools/freeze/test/ok.py @@ -0,0 +1,2 @@ +import sys +sys.exit(0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 20:30:28 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 20:30:28 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzExODI0?= =?utf-8?q?=3A_Consider_ABI_tags_in_freeze=2E_Patch_by_Meador_Inge=2E?= Message-ID: <3fxljJ3h2Tz7LkB@mail.python.org> http://hg.python.org/cpython/rev/4e37a4a036c6 changeset: 90042:4e37a4a036c6 branch: 3.4 parent: 90038:d5681d94356a user: Martin v. L?wis date: Sun Mar 30 20:28:52 2014 +0200 summary: Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. files: Misc/NEWS | 2 ++ Tools/freeze/freeze.py | 10 ++++++---- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -137,6 +137,8 @@ Tools/Demos ----------- +- Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. + - Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. Patch by Arfrever Frehtes Taifersar Arahesis. diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -219,6 +219,7 @@ # locations derived from options version = sys.version[:3] + flagged_version = version + sys.abiflags if win: extensions_c = 'frozen_extensions.c' if ishome: @@ -233,10 +234,11 @@ frozendllmain_c = os.path.join(exec_prefix, 'Pc\\frozen_dllmain.c') else: binlib = os.path.join(exec_prefix, - 'lib', 'python%s' % version, 'config') - incldir = os.path.join(prefix, 'include', 'python%s' % version) + 'lib', 'python%s' % version, + 'config-%s' % flagged_version) + incldir = os.path.join(prefix, 'include', 'python%s' % flagged_version) config_h_dir = os.path.join(exec_prefix, 'include', - 'python%s' % version) + 'python%s' % flagged_version) config_c_in = os.path.join(binlib, 'config.c.in') frozenmain_c = os.path.join(binlib, 'frozenmain.c') makefile_in = os.path.join(binlib, 'Makefile') @@ -455,7 +457,7 @@ cflags = ['$(OPT)'] cppflags = defines + includes - libs = [os.path.join(binlib, 'libpython$(VERSION).a')] + libs = [os.path.join(binlib, '$(LDLIBRARY)')] somevars = {} if os.path.exists(makefile_in): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 20:30:29 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 20:30:29 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4=3A_Issue_=2311824=3A_Consider_ABI_tags_in_fr?= =?utf-8?q?eeze=2E_Patch_by_Meador_Inge=2E?= Message-ID: <3fxljK6Pfsz7LlC@mail.python.org> http://hg.python.org/cpython/rev/1b6fc88ae3f5 changeset: 90043:1b6fc88ae3f5 parent: 90041:023b119f20eb parent: 90042:4e37a4a036c6 user: Martin v. L?wis date: Sun Mar 30 20:29:56 2014 +0200 summary: Merge 3.4: Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. files: Misc/NEWS | 2 ++ Tools/freeze/freeze.py | 10 ++++++---- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -169,6 +169,8 @@ Tools/Demos ----------- +- Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. + - Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. Patch by Arfrever Frehtes Taifersar Arahesis. diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -219,6 +219,7 @@ # locations derived from options version = sys.version[:3] + flagged_version = version + sys.abiflags if win: extensions_c = 'frozen_extensions.c' if ishome: @@ -233,10 +234,11 @@ frozendllmain_c = os.path.join(exec_prefix, 'Pc\\frozen_dllmain.c') else: binlib = os.path.join(exec_prefix, - 'lib', 'python%s' % version, 'config') - incldir = os.path.join(prefix, 'include', 'python%s' % version) + 'lib', 'python%s' % version, + 'config-%s' % flagged_version) + incldir = os.path.join(prefix, 'include', 'python%s' % flagged_version) config_h_dir = os.path.join(exec_prefix, 'include', - 'python%s' % version) + 'python%s' % flagged_version) config_c_in = os.path.join(binlib, 'config.c.in') frozenmain_c = os.path.join(binlib, 'frozenmain.c') makefile_in = os.path.join(binlib, 'Makefile') @@ -455,7 +457,7 @@ cflags = ['$(OPT)'] cppflags = defines + includes - libs = [os.path.join(binlib, 'libpython$(VERSION).a')] + libs = [os.path.join(binlib, '$(LDLIBRARY)')] somevars = {} if os.path.exists(makefile_in): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:07:39 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 21:07:39 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_make_partialme?= =?utf-8?q?thod_example_work_=28closes_=2321105=29?= Message-ID: <3fxmXC6HYZz7LkD@mail.python.org> http://hg.python.org/cpython/rev/ed81acc970d9 changeset: 90044:ed81acc970d9 branch: 3.4 parent: 90042:4e37a4a036c6 user: Benjamin Peterson date: Sun Mar 30 15:07:09 2014 -0400 summary: make partialmethod example work (closes #21105) files: Doc/library/functools.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -218,6 +218,8 @@ Example:: >>> class Cell(object): + ... def __init__(self): + ... self._alive = False ... @property ... def alive(self): ... return self._alive -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:07:41 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 21:07:41 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjExMDUp?= Message-ID: <3fxmXF26F2z7Llq@mail.python.org> http://hg.python.org/cpython/rev/b8a76485b5ed changeset: 90045:b8a76485b5ed parent: 90043:1b6fc88ae3f5 parent: 90044:ed81acc970d9 user: Benjamin Peterson date: Sun Mar 30 15:07:25 2014 -0400 summary: merge 3.4 (#21105) files: Doc/library/functools.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -218,6 +218,8 @@ Example:: >>> class Cell(object): + ... def __init__(self): + ... self._alive = False ... @property ... def alive(self): ... return self._alive -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:15:41 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 21:15:41 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE2MDQ3?= =?utf-8?q?=3A_Fix_module_exception_list_and_=5F=5Ffile=5F=5F_handling_in_?= =?utf-8?q?freeze=2E?= Message-ID: <3fxmjT6Y4Bz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/001a6dc996e7 changeset: 90046:001a6dc996e7 branch: 3.4 parent: 90044:ed81acc970d9 user: Martin v. L?wis date: Sun Mar 30 21:07:25 2014 +0200 summary: Issue #16047: Fix module exception list and __file__ handling in freeze. Patch by Meador Inge. files: Lib/site.py | 11 ++++++++--- Misc/NEWS | 3 +++ Tools/freeze/freeze.py | 4 ++++ Tools/freeze/makeconfig.py | 2 +- 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Lib/site.py b/Lib/site.py --- a/Lib/site.py +++ b/Lib/site.py @@ -364,12 +364,17 @@ builtins.credits = _sitebuiltins._Printer("credits", """\ Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information.""") - here = os.path.dirname(os.__file__) + files, dirs = [], [] + # Not all modules are required to have a __file__ attribute. See + # PEP 420 for more details. + if hasattr(os, '__file__'): + here = os.path.dirname(os.__file__) + files.extend(["LICENSE.txt", "LICENSE"]) + dirs.extend([os.path.join(here, os.pardir), here, os.curdir]) builtins.license = _sitebuiltins._Printer( "license", "See http://www.python.org/download/releases/%.5s/license" % sys.version, - ["LICENSE.txt", "LICENSE"], - [os.path.join(here, os.pardir), here, os.curdir]) + files, dirs) def sethelper(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -137,6 +137,9 @@ Tools/Demos ----------- +- Issue #16047: Fix module exception list and __file__ handling in freeze. + Patch by Meador Inge. + - Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. - Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -365,6 +365,10 @@ else: mf.load_file(mod) + # Alias "importlib._bootstrap" to "_frozen_importlib" so that the + # import machinery can bootstrap. + mf.modules["_frozen_importlib"] = mf.modules["importlib._bootstrap"] + # Add the main script as either __main__, or the actual module name. if python_entry_is_main: mf.run_script(scriptfile) diff --git a/Tools/freeze/makeconfig.py b/Tools/freeze/makeconfig.py --- a/Tools/freeze/makeconfig.py +++ b/Tools/freeze/makeconfig.py @@ -3,7 +3,7 @@ # Write the config.c file -never = ['marshal', 'imp', '_ast', '__main__', 'builtins', +never = ['marshal', '_imp', '_ast', '__main__', 'builtins', 'sys', 'gc', '_warnings'] def makeconfig(infp, outfp, modules, with_ifdef=0): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:15:43 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 21:15:43 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40ICgjMTYwNDcp?= Message-ID: <3fxmjW1RNKz7Lm0@mail.python.org> http://hg.python.org/cpython/rev/87ded2fdac4b changeset: 90047:87ded2fdac4b parent: 90045:b8a76485b5ed parent: 90046:001a6dc996e7 user: Martin v. L?wis date: Sun Mar 30 21:15:26 2014 +0200 summary: Merge 3.4 (#16047) files: Lib/site.py | 11 ++++++++--- Misc/NEWS | 3 +++ Tools/freeze/freeze.py | 4 ++++ Tools/freeze/makeconfig.py | 2 +- 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Lib/site.py b/Lib/site.py --- a/Lib/site.py +++ b/Lib/site.py @@ -364,12 +364,17 @@ builtins.credits = _sitebuiltins._Printer("credits", """\ Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information.""") - here = os.path.dirname(os.__file__) + files, dirs = [], [] + # Not all modules are required to have a __file__ attribute. See + # PEP 420 for more details. + if hasattr(os, '__file__'): + here = os.path.dirname(os.__file__) + files.extend(["LICENSE.txt", "LICENSE"]) + dirs.extend([os.path.join(here, os.pardir), here, os.curdir]) builtins.license = _sitebuiltins._Printer( "license", "See http://www.python.org/download/releases/%.5s/license" % sys.version, - ["LICENSE.txt", "LICENSE"], - [os.path.join(here, os.pardir), here, os.curdir]) + files, dirs) def sethelper(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -169,6 +169,9 @@ Tools/Demos ----------- +- Issue #16047: Fix module exception list and __file__ handling in freeze. + Patch by Meador Inge. + - Issue #11824: Consider ABI tags in freeze. Patch by Meador Inge. - Issue #20535: PYTHONWARNING no longer affects the run_tests.py script. diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -365,6 +365,10 @@ else: mf.load_file(mod) + # Alias "importlib._bootstrap" to "_frozen_importlib" so that the + # import machinery can bootstrap. + mf.modules["_frozen_importlib"] = mf.modules["importlib._bootstrap"] + # Add the main script as either __main__, or the actual module name. if python_entry_is_main: mf.run_script(scriptfile) diff --git a/Tools/freeze/makeconfig.py b/Tools/freeze/makeconfig.py --- a/Tools/freeze/makeconfig.py +++ b/Tools/freeze/makeconfig.py @@ -3,7 +3,7 @@ # Write the config.c file -never = ['marshal', 'imp', '_ast', '__main__', 'builtins', +never = ['marshal', '_imp', '_ast', '__main__', 'builtins', 'sys', 'gc', '_warnings'] def makeconfig(infp, outfp, modules, with_ifdef=0): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:33:35 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 21:33:35 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Run_test_program=2E?= Message-ID: <3fxn675hSkz7Lk1@mail.python.org> http://hg.python.org/cpython/rev/6141448001b8 changeset: 90048:6141448001b8 user: Martin v. L?wis date: Sun Mar 30 21:27:17 2014 +0200 summary: Run test program. files: Tools/freeze/test/Makefile | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Tools/freeze/test/Makefile b/Tools/freeze/test/Makefile --- a/Tools/freeze/test/Makefile +++ b/Tools/freeze/test/Makefile @@ -7,4 +7,5 @@ test: $(PYTHON) ../freeze.py -o $(OUTDIR) ok.py make -C $(OUTDIR) + $(OUTDIR)/ok -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:33:37 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 21:33:37 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_test_case_?= =?utf-8?q?for_freeze=2E?= Message-ID: <3fxn690XcCz7Lk1@mail.python.org> http://hg.python.org/cpython/rev/8410ced476c2 changeset: 90049:8410ced476c2 branch: 3.4 parent: 90046:001a6dc996e7 user: Martin v. L?wis date: Sun Mar 30 21:32:03 2014 +0200 summary: Add test case for freeze. files: Misc/NEWS | 2 ++ Tools/freeze/test/Makefile | 11 +++++++++++ Tools/freeze/test/ok.py | 2 ++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -120,6 +120,8 @@ Tests ----- +- Add test case for freeze. + - Issue #20743: Fix a reference leak in test_tcl. - Issue #21097: Move test_namespace_pkgs into test_importlib. diff --git a/Tools/freeze/test/Makefile b/Tools/freeze/test/Makefile new file mode 100644 --- /dev/null +++ b/Tools/freeze/test/Makefile @@ -0,0 +1,11 @@ +# Makefile to test freeze +# set PYTHON to path of Python interpreter to test +PYTHON=python +# set OUTDIR to the temp directory for freeze +OUTDIR=outdir + +test: + $(PYTHON) ../freeze.py -o $(OUTDIR) ok.py + make -C $(OUTDIR) + $(OUTDIR)/ok + diff --git a/Tools/freeze/test/ok.py b/Tools/freeze/test/ok.py new file mode 100644 --- /dev/null +++ b/Tools/freeze/test/ok.py @@ -0,0 +1,2 @@ +import sys +sys.exit(0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 21:33:38 2014 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 30 Mar 2014 21:33:38 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <3fxn6B3Msyz7Llw@mail.python.org> http://hg.python.org/cpython/rev/6ab0f3af5bf2 changeset: 90050:6ab0f3af5bf2 parent: 90048:6141448001b8 parent: 90049:8410ced476c2 user: Martin v. L?wis date: Sun Mar 30 21:33:18 2014 +0200 summary: Merge 3.4. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 22:43:19 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 22:43:19 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_support_CDN_pu?= =?utf-8?q?rging?= Message-ID: <3fxpfb1VXfz7LkB@mail.python.org> http://hg.python.org/cpython/rev/a720f0a312de changeset: 90051:a720f0a312de branch: 2.7 parent: 90035:6f87f50ecab7 user: Benjamin Peterson date: Sun Mar 30 16:43:11 2014 -0400 summary: support CDN purging files: Doc/tools/dailybuild.py | 51 +++++++++++++++++++++++++++- 1 files changed, 48 insertions(+), 3 deletions(-) diff --git a/Doc/tools/dailybuild.py b/Doc/tools/dailybuild.py --- a/Doc/tools/dailybuild.py +++ b/Doc/tools/dailybuild.py @@ -34,12 +34,29 @@ BRANCHES = [ # checkout, target, isdev - (BUILDROOT + '/python33', WWWROOT + '/3.3', False), - (BUILDROOT + '/python34', WWWROOT + '/3.4', True), + (BUILDROOT + '/python34', WWWROOT + '/3.4', False), + (BUILDROOT + '/python35', WWWROOT + '/3.5', True), (BUILDROOT + '/python27', WWWROOT + '/2.7', False), ] +def _files_changed(old, new): + with open(old, 'rb') as fp1, open(new, 'rb') as fp2: + st1 = os.fstat(fp1.fileno()) + st2 = os.fstat(fp2.fileno()) + if st1.st_size != st2.st_size: + return False + if st1.st_mtime >= st2.st_mtime: + return True + while True: + one = fp1.read(4096) + two = fp2.read(4096) + if one != two: + return False + if one == '': + break + return True + def build_one(checkout, target, isdev, quick): print 'Doc autobuild started in %s' % checkout os.chdir(checkout) @@ -51,12 +68,40 @@ if os.WEXITSTATUS(os.system('cd Doc; make SPHINXBUILD=%s %s' % (SPHINXBUILD, maketarget))) == 2: print '*' * 80 return + print('Computing changed files') + changed = [] + for dirpath, dirnames, filenames in os.walk('Doc/build/html/'): + dir_rel = dirpath[len('Doc/build/html/'):] + for fn in filenames: + local_path = os.path.join(dirpath, fn) + rel_path = os.path.join(dir_rel, fn) + target_path = os.path.join(target, rel_path) + if (os.path.exists(target_path) and + not _files_changed(target_path, local_path)): + changed.append(rel_path) print 'Copying HTML files to %s' % target os.system('cp -a Doc/build/html/* %s' % target) if not quick: print 'Copying dist files' os.system('mkdir -p %s/archives' % target) os.system('cp -a Doc/dist/* %s/archives' % target) + changed.append('archives/') + for fn in os.listdir(os.path.join(target, 'archives')): + changed.append('archives/' + fn) + print '%s files changed' % len(changed) + if changed: + target_ino = os.stat(target).st_ino + targets_dir = os.path.dirname(target) + prefixes = [] + for fn in os.listdir(targets_dir): + if os.stat(os.path.join(targets_dir, fn)).st_ino == target_ino: + prefixes.append(fn) + to_purge = [] + for prefix in prefixes: + to_purge.extend(prefix + "/" + p for p in changed) + purge_cmd = 'curl -X PURGE "https://docs.python.org/{%s}"' % ','.join(to_purge) + print("Running CDN purge") + os.system(purge_cmd) print 'Finished' print '=' * 80 @@ -84,7 +129,7 @@ if args: if len(args) != 2: usage() - build_one(args[0], args[1], devel, quick) + build_one(os.path.abspath(args[0]), os.path.abspath(args[1]), devel, quick) else: for checkout, dest, devel in BRANCHES: build_one(checkout, dest, devel, quick) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 23:16:26 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 23:16:26 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_update_notes_t?= =?utf-8?q?o_talk_about_-X_showrefcount?= Message-ID: <3fxqNp1Sl8z7Lld@mail.python.org> http://hg.python.org/cpython/rev/0c20f9bea056 changeset: 90052:0c20f9bea056 branch: 3.4 parent: 90049:8410ced476c2 user: Benjamin Peterson date: Sun Mar 30 17:16:09 2014 -0400 summary: update notes to talk about -X showrefcount files: Misc/SpecialBuilds.txt | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Misc/SpecialBuilds.txt b/Misc/SpecialBuilds.txt --- a/Misc/SpecialBuilds.txt +++ b/Misc/SpecialBuilds.txt @@ -13,12 +13,14 @@ ------------ Turn on aggregate reference counting. This arranges that extern _Py_RefTotal -hold a count of all references, the sum of ob_refcnt across all objects. In a -debug-mode build, this is where the "8288" comes from in +hold a count of all references, the sum of ob_refcnt across all objects. +Passing ``-X showrefcount`` on the command line causes the interactive +interpreter to print the reference count total as well the number of memory +blocks allocated after each statement: >>> 23 23 - [8288 refs] + [8288 refs, 14332 blocks] >>> Note that if this count increases when you're not storing away new objects, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Mar 30 23:16:27 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 30 Mar 2014 23:16:27 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fxqNq3Vmfz7Lld@mail.python.org> http://hg.python.org/cpython/rev/c830436b25dc changeset: 90053:c830436b25dc parent: 90050:6ab0f3af5bf2 parent: 90052:0c20f9bea056 user: Benjamin Peterson date: Sun Mar 30 17:16:18 2014 -0400 summary: merge 3.4 files: Misc/SpecialBuilds.txt | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Misc/SpecialBuilds.txt b/Misc/SpecialBuilds.txt --- a/Misc/SpecialBuilds.txt +++ b/Misc/SpecialBuilds.txt @@ -13,12 +13,14 @@ ------------ Turn on aggregate reference counting. This arranges that extern _Py_RefTotal -hold a count of all references, the sum of ob_refcnt across all objects. In a -debug-mode build, this is where the "8288" comes from in +hold a count of all references, the sum of ob_refcnt across all objects. +Passing ``-X showrefcount`` on the command line causes the interactive +interpreter to print the reference count total as well the number of memory +blocks allocated after each statement: >>> 23 23 - [8288 refs] + [8288 refs, 14332 blocks] >>> Note that if this count increases when you're not storing away new objects, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:16:56 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:16:56 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_indentatio?= =?utf-8?q?n_and_add_braces?= Message-ID: <3fxt3r4cTkz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/79dad8ec1805 changeset: 90054:79dad8ec1805 branch: 2.7 parent: 90051:a720f0a312de user: Benjamin Peterson date: Sun Mar 30 19:16:44 2014 -0400 summary: fix indentation and add braces files: Objects/stringobject.c | 35 +++++++++++++++-------------- 1 files changed, 18 insertions(+), 17 deletions(-) diff --git a/Objects/stringobject.c b/Objects/stringobject.c --- a/Objects/stringobject.c +++ b/Objects/stringobject.c @@ -3091,24 +3091,25 @@ i = 0; /* chars up to and including most recent \n or \r */ j = 0; /* chars since most recent \n or \r (use in tab calculations) */ e = PyString_AS_STRING(self) + PyString_GET_SIZE(self); /* end of input */ - for (p = PyString_AS_STRING(self); p < e; p++) - if (*p == '\t') { - if (tabsize > 0) { - incr = tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX - incr) + for (p = PyString_AS_STRING(self); p < e; p++) { + if (*p == '\t') { + if (tabsize > 0) { + incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow1; + j += incr; + } + } + else { + if (j > PY_SSIZE_T_MAX - 1) goto overflow1; - j += incr; - } - } - else { - if (j > PY_SSIZE_T_MAX - 1) - goto overflow1; - j++; - if (*p == '\n' || *p == '\r') { - if (i > PY_SSIZE_T_MAX - j) - goto overflow1; - i += j; - j = 0; + j++; + if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow1; + i += j; + j = 0; + } } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:23:32 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:23:32 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_braces_and?= =?utf-8?q?_fix_indentation?= Message-ID: <3fxtCS0G5xzQk8@mail.python.org> http://hg.python.org/cpython/rev/4cd80c35d954 changeset: 90055:4cd80c35d954 branch: 2.7 user: Benjamin Peterson date: Sun Mar 30 19:23:24 2014 -0400 summary: add braces and fix indentation files: Objects/stringobject.c | 35 +++++++++++++++-------------- 1 files changed, 18 insertions(+), 17 deletions(-) diff --git a/Objects/stringobject.c b/Objects/stringobject.c --- a/Objects/stringobject.c +++ b/Objects/stringobject.c @@ -3125,25 +3125,26 @@ q = PyString_AS_STRING(u); /* next output char */ qe = PyString_AS_STRING(u) + PyString_GET_SIZE(u); /* end of output */ - for (p = PyString_AS_STRING(self); p < e; p++) - if (*p == '\t') { - if (tabsize > 0) { - i = tabsize - (j % tabsize); - j += i; - while (i--) { - if (q >= qe) - goto overflow2; - *q++ = ' '; + for (p = PyString_AS_STRING(self); p < e; p++) { + if (*p == '\t') { + if (tabsize > 0) { + i = tabsize - (j % tabsize); + j += i; + while (i--) { + if (q >= qe) + goto overflow2; + *q++ = ' '; + } } } - } - else { - if (q >= qe) - goto overflow2; - *q++ = *p; - j++; - if (*p == '\n' || *p == '\r') - j = 0; + else { + if (q >= qe) + goto overflow2; + *q++ = *p; + j++; + if (*p == '\n' || *p == '\r') + j = 0; + } } return u; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:54:06 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:54:06 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_expandtabs?= =?utf-8?q?_overflow_detection_to_be_consistent_and_not_rely_on_signed?= Message-ID: <3fxttk2whMz7Lkq@mail.python.org> http://hg.python.org/cpython/rev/a2254f4338e6 changeset: 90056:a2254f4338e6 branch: 2.7 user: Benjamin Peterson date: Sun Mar 30 19:47:57 2014 -0400 summary: fix expandtabs overflow detection to be consistent and not rely on signed overflow files: Objects/stringlib/transmogrify.h | 38 ++++++++++---------- 1 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h --- a/Objects/stringlib/transmogrify.h +++ b/Objects/stringlib/transmogrify.h @@ -15,7 +15,7 @@ { const char *e, *p; char *q; - size_t i, j; + Py_ssize_t i, j; PyObject *u; int tabsize = 8; @@ -25,34 +25,30 @@ /* First pass: determine size of output string */ i = j = 0; e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } + Py_ssize_t incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; - if (i > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } } } + } - if ((i + j) > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, "result is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ u = STRINGLIB_NEW(NULL, i + j); @@ -62,7 +58,7 @@ j = 0; q = STRINGLIB_STR(u); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { i = tabsize - (j % tabsize); @@ -77,8 +73,12 @@ if (*p == '\n' || *p == '\r') j = 0; } - + } + return u; + overflow: + PyErr_SetString(PyExc_OverflowError, "result too long"); + return NULL; } Py_LOCAL_INLINE(PyObject *) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:54:07 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:54:07 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_fix_expandtabs?= =?utf-8?q?_overflow_detection_to_be_consistent_and_not_rely_on_signed?= Message-ID: <3fxttl4JmLz7Lkq@mail.python.org> http://hg.python.org/cpython/rev/f4995c63a177 changeset: 90057:f4995c63a177 branch: 3.2 parent: 89609:789ca594960f user: Benjamin Peterson date: Sun Mar 30 19:47:57 2014 -0400 summary: fix expandtabs overflow detection to be consistent and not rely on signed overflow files: Objects/stringlib/transmogrify.h | 38 ++++++++++---------- 1 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h --- a/Objects/stringlib/transmogrify.h +++ b/Objects/stringlib/transmogrify.h @@ -15,7 +15,7 @@ { const char *e, *p; char *q; - size_t i, j; + Py_ssize_t i, j; PyObject *u; int tabsize = 8; @@ -25,34 +25,30 @@ /* First pass: determine size of output string */ i = j = 0; e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } + Py_ssize_t incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; - if (i > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } } } + } - if ((i + j) > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, "result is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ u = STRINGLIB_NEW(NULL, i + j); @@ -62,7 +58,7 @@ j = 0; q = STRINGLIB_STR(u); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { i = tabsize - (j % tabsize); @@ -77,8 +73,12 @@ if (*p == '\n' || *p == '\r') j = 0; } - + } + return u; + overflow: + PyErr_SetString(PyExc_OverflowError, "result too long"); + return NULL; } Py_LOCAL_INLINE(PyObject *) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:54:08 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:54:08 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2?= Message-ID: <3fxttm5kGrz7LmJ@mail.python.org> http://hg.python.org/cpython/rev/40dbf7055335 changeset: 90058:40dbf7055335 branch: 3.3 parent: 89927:0b2814fc53ae parent: 90057:f4995c63a177 user: Benjamin Peterson date: Sun Mar 30 19:52:22 2014 -0400 summary: merge 3.2 files: Objects/stringlib/transmogrify.h | 38 ++++++++++---------- 1 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h --- a/Objects/stringlib/transmogrify.h +++ b/Objects/stringlib/transmogrify.h @@ -15,7 +15,7 @@ { const char *e, *p; char *q; - size_t i, j; + Py_ssize_t i, j; PyObject *u; int tabsize = 8; @@ -25,34 +25,30 @@ /* First pass: determine size of output string */ i = j = 0; e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } + Py_ssize_t incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; - if (i > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } } } + } - if ((i + j) > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, "result is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ u = STRINGLIB_NEW(NULL, i + j); @@ -61,8 +57,8 @@ j = 0; q = STRINGLIB_STR(u); - - for (p = STRINGLIB_STR(self); p < e; p++) + + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { i = tabsize - (j % tabsize); @@ -77,8 +73,12 @@ if (*p == '\n' || *p == '\r') j = 0; } + } return u; + overflow: + PyErr_SetString(PyExc_OverflowError, "result too long"); + return NULL; } Py_LOCAL_INLINE(PyObject *) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:54:10 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:54:10 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3?= Message-ID: <3fxttp03h7z7LmP@mail.python.org> http://hg.python.org/cpython/rev/bd513e5a5047 changeset: 90059:bd513e5a5047 branch: 3.4 parent: 90052:0c20f9bea056 parent: 90058:40dbf7055335 user: Benjamin Peterson date: Sun Mar 30 19:52:39 2014 -0400 summary: merge 3.3 files: Objects/stringlib/transmogrify.h | 38 ++++++++++---------- 1 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h --- a/Objects/stringlib/transmogrify.h +++ b/Objects/stringlib/transmogrify.h @@ -15,7 +15,7 @@ { const char *e, *p; char *q; - size_t i, j; + Py_ssize_t i, j; PyObject *u; static char *kwlist[] = {"tabsize", 0}; int tabsize = 8; @@ -27,34 +27,30 @@ /* First pass: determine size of output string */ i = j = 0; e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } + Py_ssize_t incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; - if (i > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } } } + } - if ((i + j) > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, "result is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ u = STRINGLIB_NEW(NULL, i + j); @@ -63,8 +59,8 @@ j = 0; q = STRINGLIB_STR(u); - - for (p = STRINGLIB_STR(self); p < e; p++) + + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { i = tabsize - (j % tabsize); @@ -79,8 +75,12 @@ if (*p == '\n' || *p == '\r') j = 0; } + } return u; + overflow: + PyErr_SetString(PyExc_OverflowError, "result too long"); + return NULL; } Py_LOCAL_INLINE(PyObject *) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 01:54:11 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 01:54:11 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fxttq1PRZz7LmS@mail.python.org> http://hg.python.org/cpython/rev/cb66598b0bfb changeset: 90060:cb66598b0bfb parent: 90053:c830436b25dc parent: 90059:bd513e5a5047 user: Benjamin Peterson date: Sun Mar 30 19:52:50 2014 -0400 summary: merge 3.4 files: Objects/stringlib/transmogrify.h | 38 ++++++++++---------- 1 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h --- a/Objects/stringlib/transmogrify.h +++ b/Objects/stringlib/transmogrify.h @@ -15,7 +15,7 @@ { const char *e, *p; char *q; - size_t i, j; + Py_ssize_t i, j; PyObject *u; static char *kwlist[] = {"tabsize", 0}; int tabsize = 8; @@ -27,34 +27,30 @@ /* First pass: determine size of output string */ i = j = 0; e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); - for (p = STRINGLIB_STR(self); p < e; p++) + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (j > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } + Py_ssize_t incr = tabsize - (j % tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n' || *p == '\r') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; - if (i > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "result is too long"); - return NULL; - } } } + } - if ((i + j) > PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, "result is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ u = STRINGLIB_NEW(NULL, i + j); @@ -63,8 +59,8 @@ j = 0; q = STRINGLIB_STR(u); - - for (p = STRINGLIB_STR(self); p < e; p++) + + for (p = STRINGLIB_STR(self); p < e; p++) { if (*p == '\t') { if (tabsize > 0) { i = tabsize - (j % tabsize); @@ -79,8 +75,12 @@ if (*p == '\n' || *p == '\r') j = 0; } + } return u; + overflow: + PyErr_SetString(PyExc_OverflowError, "result too long"); + return NULL; } Py_LOCAL_INLINE(PyObject *) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:10:10 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:10:10 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_overflow_d?= =?utf-8?q?etection_of_strop=2Eexpandtabs?= Message-ID: <3fxvFG08JQz7Lld@mail.python.org> http://hg.python.org/cpython/rev/5dabc2d2f776 changeset: 90061:5dabc2d2f776 branch: 2.7 parent: 90056:a2254f4338e6 user: Benjamin Peterson date: Sun Mar 30 20:09:44 2014 -0400 summary: fix overflow detection of strop.expandtabs files: Lib/test/test_strop.py | 6 ++++++ Misc/NEWS | 3 +++ Modules/stropmodule.c | 28 +++++++++++++++------------- 3 files changed, 24 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_strop.py b/Lib/test/test_strop.py --- a/Lib/test/test_strop.py +++ b/Lib/test/test_strop.py @@ -4,6 +4,7 @@ r'test.test_strop|unittest') import strop import unittest +import sys from test import test_support @@ -115,6 +116,11 @@ strop.uppercase strop.whitespace + @unittest.skipUnless(sys.maxsize == 2147483647, "only for 32-bit") + def test_expandtabs_overflow(self): + s = '\t\n' * 0x10000 + 'A' * 0x1000000 + self.assertRaises(OverflowError, strop.expandtabs, s, 0x10001) + @test_support.precisionbigmemtest(size=test_support._2G - 1, memuse=5) def test_stropjoin_huge_list(self, size): a = "A" * size diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -40,6 +40,9 @@ Library ------- +- Fix possible overflow bug in strop.expandtabs. You shouldn't be using this + module! + - Issue #20145: `assertRaisesRegex` now raises a TypeError if the second argument is not a string or compiled regex. diff --git a/Modules/stropmodule.c b/Modules/stropmodule.c --- a/Modules/stropmodule.c +++ b/Modules/stropmodule.c @@ -593,7 +593,7 @@ char* e; char* p; char* q; - Py_ssize_t i, j, old_j; + Py_ssize_t i, j; PyObject* out; char* string; Py_ssize_t stringlen; @@ -610,30 +610,29 @@ } /* First pass: determine size of output string */ - i = j = old_j = 0; /* j: current column; i: total of previous lines */ + i = j = 0; /* j: current column; i: total of previous lines */ e = string + stringlen; for (p = string; p < e; p++) { if (*p == '\t') { - j += tabsize - (j%tabsize); - if (old_j > j) { - PyErr_SetString(PyExc_OverflowError, - "new string is too long"); - return NULL; - } - old_j = j; + Py_ssize_t incr = tabsize - (j%tabsize); + if (j > PY_SSIZE_T_MAX - incr) + goto overflow; + j += incr; } else { + if (j > PY_SSIZE_T_MAX - 1) + goto overflow; j++; if (*p == '\n') { + if (i > PY_SSIZE_T_MAX - j) + goto overflow; i += j; j = 0; } } } - if ((i + j) < 0) { - PyErr_SetString(PyExc_OverflowError, "new string is too long"); - return NULL; - } + if (i > PY_SSIZE_T_MAX - j) + goto overflow; /* Second pass: create output string and fill it */ out = PyString_FromStringAndSize(NULL, i+j); @@ -658,6 +657,9 @@ } return out; + overflow: + PyErr_SetString(PyExc_OverflowError, "result is too long"); + return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:34:38 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:34:38 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_Ian_Beer?= Message-ID: <3fxvnV3WHGz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/b6a5d820d300 changeset: 90062:b6a5d820d300 branch: 2.7 user: Benjamin Peterson date: Sun Mar 30 20:33:47 2014 -0400 summary: add Ian Beer files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -96,6 +96,7 @@ Robin Becker Torsten Becker Bill Bedford +Ian Beer Stefan Behnel Reimer Behrends Ben Bell -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:34:39 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:34:39 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_add_Ian_Beer?= Message-ID: <3fxvnW5B6Vz7Lld@mail.python.org> http://hg.python.org/cpython/rev/9b1286d122a2 changeset: 90063:9b1286d122a2 branch: 3.2 parent: 90057:f4995c63a177 user: Benjamin Peterson date: Sun Mar 30 20:33:47 2014 -0400 summary: add Ian Beer files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -86,6 +86,7 @@ Neal Becker Torsten Becker Bill Bedford +Ian Beer Stefan Behnel Reimer Behrends Ben Bell -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:34:40 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:34:40 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2?= Message-ID: <3fxvnX6w2rz7Ll4@mail.python.org> http://hg.python.org/cpython/rev/b4cb431badac changeset: 90064:b4cb431badac branch: 3.3 parent: 90058:40dbf7055335 parent: 90063:9b1286d122a2 user: Benjamin Peterson date: Sun Mar 30 20:34:05 2014 -0400 summary: merge 3.2 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -95,6 +95,7 @@ Robin Becker Torsten Becker Bill Bedford +Ian Beer Stefan Behnel Reimer Behrends Ben Bell -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:34:42 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:34:42 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3?= Message-ID: <3fxvnZ16hnz7LmX@mail.python.org> http://hg.python.org/cpython/rev/a4672c0f1187 changeset: 90065:a4672c0f1187 branch: 3.4 parent: 90059:bd513e5a5047 parent: 90064:b4cb431badac user: Benjamin Peterson date: Sun Mar 30 20:34:20 2014 -0400 summary: merge 3.3 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -96,6 +96,7 @@ Robin Becker Torsten Becker Bill Bedford +Ian Beer Stefan Behnel Reimer Behrends Ben Bell -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 02:34:43 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 02:34:43 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <3fxvnb2Ykpz7LmJ@mail.python.org> http://hg.python.org/cpython/rev/8017e8df6a30 changeset: 90066:8017e8df6a30 parent: 90060:cb66598b0bfb parent: 90065:a4672c0f1187 user: Benjamin Peterson date: Sun Mar 30 20:34:30 2014 -0400 summary: merge 3.4 files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -96,6 +96,7 @@ Robin Becker Torsten Becker Bill Bedford +Ian Beer Stefan Behnel Reimer Behrends Ben Bell -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Mar 31 09:53:07 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 31 Mar 2014 09:53:07 +0200 Subject: [Python-checkins] Daily reference leaks (8017e8df6a30): sum=4 Message-ID: results for 8017e8df6a30 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 4] memory blocks, sum=4 test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKGzXNx', '-x'] From python-checkins at python.org Mon Mar 31 17:42:56 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 31 Mar 2014 17:42:56 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNjY4?= =?utf-8?q?=3A_Remove_tests=2Etxt_of_test=5Fasyncio?= Message-ID: <3fyHxX1QgFz7Lkd@mail.python.org> http://hg.python.org/cpython/rev/bf661f72aaab changeset: 90067:bf661f72aaab branch: 3.4 parent: 90065:a4672c0f1187 user: Victor Stinner date: Mon Mar 31 17:40:06 2014 +0200 summary: Issue #20668: Remove tests.txt of test_asyncio It's useless, tests are now discovered automatically. files: Lib/test/test_asyncio/tests.txt | 14 -------------- 1 files changed, 0 insertions(+), 14 deletions(-) diff --git a/Lib/test/test_asyncio/tests.txt b/Lib/test/test_asyncio/tests.txt deleted file mode 100644 --- a/Lib/test/test_asyncio/tests.txt +++ /dev/null @@ -1,14 +0,0 @@ -test_asyncio.test_base_events -test_asyncio.test_events -test_asyncio.test_futures -test_asyncio.test_locks -test_asyncio.test_proactor_events -test_asyncio.test_queues -test_asyncio.test_selector_events -test_asyncio.test_streams -test_asyncio.test_subprocess -test_asyncio.test_tasks -test_asyncio.test_transports -test_asyncio.test_unix_events -test_asyncio.test_windows_events -test_asyncio.test_windows_utils -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 17:42:57 2014 From: python-checkins at python.org (victor.stinner) Date: Mon, 31 Mar 2014 17:42:57 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2320668=3A_Remove_tests=2Etxt_o?= =?utf-8?q?f_test=5Fasyncio?= Message-ID: <3fyHxY37spz7Ln7@mail.python.org> http://hg.python.org/cpython/rev/a76e459e09be changeset: 90068:a76e459e09be parent: 90066:8017e8df6a30 parent: 90067:bf661f72aaab user: Victor Stinner date: Mon Mar 31 17:40:52 2014 +0200 summary: (Merge 3.4) Issue #20668: Remove tests.txt of test_asyncio It's useless, tests are now discovered automatically. files: Lib/test/test_asyncio/tests.txt | 14 -------------- 1 files changed, 0 insertions(+), 14 deletions(-) diff --git a/Lib/test/test_asyncio/tests.txt b/Lib/test/test_asyncio/tests.txt deleted file mode 100644 --- a/Lib/test/test_asyncio/tests.txt +++ /dev/null @@ -1,14 +0,0 @@ -test_asyncio.test_base_events -test_asyncio.test_events -test_asyncio.test_futures -test_asyncio.test_locks -test_asyncio.test_proactor_events -test_asyncio.test_queues -test_asyncio.test_selector_events -test_asyncio.test_streams -test_asyncio.test_subprocess -test_asyncio.test_tasks -test_asyncio.test_transports -test_asyncio.test_unix_events -test_asyncio.test_windows_events -test_asyncio.test_windows_utils -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:07 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:07 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_use_https_docs?= =?utf-8?q?_url_=28=2321115=29?= Message-ID: <3fyLnR6RDMz7Ln7@mail.python.org> http://hg.python.org/cpython/rev/8263d3c1cf34 changeset: 90069:8263d3c1cf34 branch: 2.7 parent: 90062:b6a5d820d300 user: Benjamin Peterson date: Mon Mar 31 13:44:53 2014 -0400 summary: use https docs url (#21115) files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -155,12 +155,12 @@ ## self._test_urls(urls, self._extra_handlers()+[bauth, dauth]) def test_urlwithfrag(self): - urlwith_frag = "http://docs.python.org/2/glossary.html#glossary" + urlwith_frag = "https://docs.python.org/2/glossary.html#glossary" with test_support.transient_internet(urlwith_frag): req = urllib2.Request(urlwith_frag) res = urllib2.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/2/glossary.html#glossary") + "https://docs.python.org/2/glossary.html#glossary") def test_fileno(self): req = urllib2.Request("http://www.example.com") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:09 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:09 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_use_https_docs?= =?utf-8?q?_url_=28=2321115=29?= Message-ID: <3fyLnT123Vz7LnS@mail.python.org> http://hg.python.org/cpython/rev/fc28ce0983de changeset: 90070:fc28ce0983de branch: 3.2 parent: 90063:9b1286d122a2 user: Benjamin Peterson date: Mon Mar 31 13:44:53 2014 -0400 summary: use https docs url (#21115) files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -156,12 +156,12 @@ ## self._test_urls(urls, self._extra_handlers()+[bauth, dauth]) def test_urlwithfrag(self): - urlwith_frag = "http://docs.python.org/2/glossary.html#glossary" + urlwith_frag = "https://docs.python.org/2/glossary.html#glossary" with support.transient_internet(urlwith_frag): req = urllib.request.Request(urlwith_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/2/glossary.html#glossary") + "https://docs.python.org/2/glossary.html#glossary") def test_custom_headers(self): url = "http://www.example.com" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:10 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:10 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2?= Message-ID: <3fyLnV2fl9z7Lkn@mail.python.org> http://hg.python.org/cpython/rev/cb3a8abc0870 changeset: 90071:cb3a8abc0870 branch: 3.3 parent: 90064:b4cb431badac parent: 90070:fc28ce0983de user: Benjamin Peterson date: Mon Mar 31 13:46:26 2014 -0400 summary: merge 3.2 files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -155,12 +155,12 @@ ## self._test_urls(urls, self._extra_handlers()+[bauth, dauth]) def test_urlwithfrag(self): - urlwith_frag = "http://docs.python.org/2/glossary.html#glossary" + urlwith_frag = "https://docs.python.org/2/glossary.html#glossary" with support.transient_internet(urlwith_frag): req = urllib.request.Request(urlwith_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/2/glossary.html#glossary") + "https://docs.python.org/2/glossary.html#glossary") def test_custom_headers(self): url = "http://www.example.com" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:11 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:11 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3_=28=2321115=29?= Message-ID: <3fyLnW4Mb1z7LnZ@mail.python.org> http://hg.python.org/cpython/rev/7ac893834a5e changeset: 90072:7ac893834a5e branch: 3.4 parent: 90067:bf661f72aaab parent: 90071:cb3a8abc0870 user: Benjamin Peterson date: Mon Mar 31 13:46:45 2014 -0400 summary: merge 3.3 (#21115) files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -157,12 +157,12 @@ ## self._test_urls(urls, self._extra_handlers()+[bauth, dauth]) def test_urlwithfrag(self): - urlwith_frag = "http://docs.python.org/2/glossary.html#glossary" + urlwith_frag = "https://docs.python.org/2/glossary.html#glossary" with support.transient_internet(urlwith_frag): req = urllib.request.Request(urlwith_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/2/glossary.html#glossary") + "https://docs.python.org/2/glossary.html#glossary") def test_redirect_url_withfrag(self): redirect_url_with_frag = "http://bitly.com/urllibredirecttest" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:12 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:12 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjExMTUp?= Message-ID: <3fyLnX611nz7LnF@mail.python.org> http://hg.python.org/cpython/rev/c0d89375d4c8 changeset: 90073:c0d89375d4c8 parent: 90068:a76e459e09be parent: 90072:7ac893834a5e user: Benjamin Peterson date: Mon Mar 31 13:47:04 2014 -0400 summary: merge 3.4 (#21115) files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -157,12 +157,12 @@ ## self._test_urls(urls, self._extra_handlers()+[bauth, dauth]) def test_urlwithfrag(self): - urlwith_frag = "http://docs.python.org/2/glossary.html#glossary" + urlwith_frag = "https://docs.python.org/2/glossary.html#glossary" with support.transient_internet(urlwith_frag): req = urllib.request.Request(urlwith_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/2/glossary.html#glossary") + "https://docs.python.org/2/glossary.html#glossary") def test_redirect_url_withfrag(self): redirect_url_with_frag = "http://bitly.com/urllibredirecttest" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:14 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:14 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_use_https_docs?= =?utf-8?q?_url?= Message-ID: <3fyLnZ0RLtz7LnR@mail.python.org> http://hg.python.org/cpython/rev/d6fa1b30cdac changeset: 90074:d6fa1b30cdac branch: 3.4 parent: 90072:7ac893834a5e user: Benjamin Peterson date: Mon Mar 31 13:50:34 2014 -0400 summary: use https docs url files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -165,12 +165,12 @@ "https://docs.python.org/2/glossary.html#glossary") def test_redirect_url_withfrag(self): - redirect_url_with_frag = "http://bitly.com/urllibredirecttest" + redirect_url_with_frag = "http://bit.ly/1iSHToT" with support.transient_internet(redirect_url_with_frag): req = urllib.request.Request(redirect_url_with_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/3.4/glossary.html#term-global-interpreter-lock") + "https://docs.python.org/3.4/glossary.html#term-global-interpreter-lock") def test_custom_headers(self): url = "http://www.example.com" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 19:51:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 31 Mar 2014 19:51:15 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjExMTUp?= Message-ID: <3fyLnb1yhCz7LnT@mail.python.org> http://hg.python.org/cpython/rev/c8bef5664305 changeset: 90075:c8bef5664305 parent: 90073:c0d89375d4c8 parent: 90074:d6fa1b30cdac user: Benjamin Peterson date: Mon Mar 31 13:50:45 2014 -0400 summary: merge 3.4 (#21115) files: Lib/test/test_urllib2net.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -165,12 +165,12 @@ "https://docs.python.org/2/glossary.html#glossary") def test_redirect_url_withfrag(self): - redirect_url_with_frag = "http://bitly.com/urllibredirecttest" + redirect_url_with_frag = "http://bit.ly/1iSHToT" with support.transient_internet(redirect_url_with_frag): req = urllib.request.Request(redirect_url_with_frag) res = urllib.request.urlopen(req) self.assertEqual(res.geturl(), - "http://docs.python.org/3.4/glossary.html#term-global-interpreter-lock") + "https://docs.python.org/3.4/glossary.html#term-global-interpreter-lock") def test_custom_headers(self): url = "http://www.example.com" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 22:05:12 2014 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 31 Mar 2014 22:05:12 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDcz?= =?utf-8?q?=3A_explain_why_Py=5FReprEnter=28=29_allows_for_a_missing_threa?= =?utf-8?q?d_state=2E?= Message-ID: <3fyPm827rYzS0d@mail.python.org> http://hg.python.org/cpython/rev/c42cce290d50 changeset: 90076:c42cce290d50 branch: 3.4 parent: 90074:d6fa1b30cdac user: Antoine Pitrou date: Mon Mar 31 22:04:38 2014 +0200 summary: Issue #21073: explain why Py_ReprEnter() allows for a missing thread state. files: Objects/object.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1849,6 +1849,8 @@ Py_ssize_t i; dict = PyThreadState_GetDict(); + /* Ignore a missing thread-state, so that this function can be called + early on startup. */ if (dict == NULL) return 0; list = _PyDict_GetItemId(dict, &PyId_Py_Repr); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 22:05:13 2014 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 31 Mar 2014 22:05:13 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2321073=3A_explain_why_Py=5FReprEnter=28=29_allow?= =?utf-8?q?s_for_a_missing_thread_state=2E?= Message-ID: <3fyPm93WTsz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/fb217fc457ca changeset: 90077:fb217fc457ca parent: 90075:c8bef5664305 parent: 90076:c42cce290d50 user: Antoine Pitrou date: Mon Mar 31 22:05:02 2014 +0200 summary: Issue #21073: explain why Py_ReprEnter() allows for a missing thread state. files: Objects/object.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1849,6 +1849,8 @@ Py_ssize_t i; dict = PyThreadState_GetDict(); + /* Ignore a missing thread-state, so that this function can be called + early on startup. */ if (dict == NULL) return 0; list = _PyDict_GetItemId(dict, &PyId_Py_Repr); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 23:41:23 2014 From: python-checkins at python.org (andrew.svetlov) Date: Mon, 31 Mar 2014 23:41:23 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogRml4IHRoZSBkb2M6?= =?utf-8?q?_add_deprecation_warning_in_select_module=2E?= Message-ID: <3fyRv71LtXz7LkB@mail.python.org> http://hg.python.org/cpython/rev/f507727c5dd2 changeset: 90078:f507727c5dd2 branch: 3.4 parent: 90076:c42cce290d50 user: Andrew Svetlov date: Tue Apr 01 00:23:23 2014 +0300 summary: Fix the doc: add deprecation warning in select module. files: Doc/library/select.rst | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -377,9 +377,13 @@ Modifies an already registered fd. This has the same effect as ``register(fd, eventmask)``. Attempting to modify a file descriptor - that was never registered causes an :exc:`IOError` exception with errno + that was never registered causes an :exc:`OSError` exception with errno :const:`ENOENT` to be raised. + .. deprecated:: 3.3 + + Instead of :exc:`IOError` the :exc:`OSError` exception is used now. + .. method:: poll.unregister(fd) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 23:41:24 2014 From: python-checkins at python.org (andrew.svetlov) Date: Mon, 31 Mar 2014 23:41:24 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40ICgjMTY3MTYp?= Message-ID: <3fyRv82bWbz7Ln7@mail.python.org> http://hg.python.org/cpython/rev/5ad1d523b342 changeset: 90079:5ad1d523b342 parent: 90077:fb217fc457ca parent: 90078:f507727c5dd2 user: Andrew Svetlov date: Tue Apr 01 00:41:04 2014 +0300 summary: Merge 3.4 (#16716) files: Doc/library/select.rst | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -377,9 +377,13 @@ Modifies an already registered fd. This has the same effect as ``register(fd, eventmask)``. Attempting to modify a file descriptor - that was never registered causes an :exc:`IOError` exception with errno + that was never registered causes an :exc:`OSError` exception with errno :const:`ENOENT` to be raised. + .. deprecated:: 3.3 + + Instead of :exc:`IOError` the :exc:`OSError` exception is used now. + .. method:: poll.unregister(fd) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 23:45:24 2014 From: python-checkins at python.org (andrew.svetlov) Date: Mon, 31 Mar 2014 23:45:24 +0200 (CEST) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSU9FcnJvciAtPiBP?= =?utf-8?q?SError?= Message-ID: <3fyRzm2cnDzQk8@mail.python.org> http://hg.python.org/cpython/rev/de476c612548 changeset: 90080:de476c612548 branch: 3.4 parent: 90078:f507727c5dd2 user: Andrew Svetlov date: Tue Apr 01 00:44:13 2014 +0300 summary: IOError -> OSError files: Doc/library/exceptions.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -82,7 +82,7 @@ .. attribute:: args The tuple of arguments given to the exception constructor. Some built-in - exceptions (like :exc:`IOError`) expect a certain number of arguments and + exceptions (like :exc:`OSError`) expect a certain number of arguments and assign a special meaning to the elements of this tuple, while others are usually called only with a single string giving an error message. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Mar 31 23:45:25 2014 From: python-checkins at python.org (andrew.svetlov) Date: Mon, 31 Mar 2014 23:45:25 +0200 (CEST) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40ICgjMTY3MTYp?= Message-ID: <3fyRzn4Cxyz7Ln7@mail.python.org> http://hg.python.org/cpython/rev/c0c17718301f changeset: 90081:c0c17718301f parent: 90079:5ad1d523b342 parent: 90080:de476c612548 user: Andrew Svetlov date: Tue Apr 01 00:45:06 2014 +0300 summary: Merge 3.4 (#16716) files: Doc/library/exceptions.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -82,7 +82,7 @@ .. attribute:: args The tuple of arguments given to the exception constructor. Some built-in - exceptions (like :exc:`IOError`) expect a certain number of arguments and + exceptions (like :exc:`OSError`) expect a certain number of arguments and assign a special meaning to the elements of this tuple, while others are usually called only with a single string giving an error message. -- Repository URL: http://hg.python.org/cpython